• v1.24.0-alpha.3

  • Downloads for v1.24.0-alpha.3

    • Source Code
    • Client Binaries
    • Server Binaries
    • Node Binaries
    • Container Images

  • Changelog since v1.24.0-alpha.2

  • Urgent Upgrade Notes

    • (No, really, you MUST read this before you upgrade)

  • Changes by Kind

    • Deprecation
    • API Change
    • Feature
    • Bug or Regression
    • Other (Cleanup or Flake)

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.24.0-alpha.2

  • Downloads for v1.24.0-alpha.2

    • Source Code
    • Client Binaries
    • Server Binaries
    • Node Binaries
    • Container Images

  • Changelog since v1.24.0-alpha.1

  • Urgent Upgrade Notes

    • (No, really, you MUST read this before you upgrade)

  • Changes by Kind

    • Deprecation
    • API Change
    • Feature
    • Bug or Regression
    • Other (Cleanup or Flake)

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.24.0-alpha.1

  • Downloads for v1.24.0-alpha.1

    • Source Code
    • Client Binaries
    • Server Binaries
    • Node Binaries

  • Changelog since v1.23.0

  • Urgent Upgrade Notes

    • (No, really, you MUST read this before you upgrade)

  • Changes by Kind

    • Feature
    • Bug or Regression
    • Other (Cleanup or Flake)

  • Dependencies

    • Added
    • Changed
    • Removed

1. v1.24.0-alpha.3

1.1. Downloads for v1.24.0-alpha.3

1.1.1. Source Code

1.1.2. Client Binaries

1.1.3. Server Binaries

1.1.4. Node Binaries

1.1.5. Container Images

All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.

1.2. Changelog since v1.24.0-alpha.2

1.3. Urgent Upgrade Notes

1.3.1. (No, really, you MUST read this before you upgrade)

• Fixed bug with leads to Node goes "Not-ready" state when credentials for vCenter stored in a secret and Zones feature is in use. Zone labels setup moved to KCM component, kubelet skips this step during startup in such case. If credentials stored in cloud-provider config file as plaintext current behaviour does not change and no action required.

  For proper functioning "kube-system:vsphere-legacy-cloud-provider" should be allowed to update node object if vCenter credentials stored in secret and Zone feature used. (#101028, @lobziik) [SIG Cloud Provider]

1.4. Changes by Kind

1.4.1. Deprecation

• Cluster addon for dashboard was removed. To install dashboard, see here. (#107481, @shu-mutou) [SIG Cloud Provider and Testing]
• Kube-apiserver: the --master-count flag and --endpoint-reconciler-type=master-count reconciler are deprecated in favor of the lease reconciler (#108062, @aojea) [SIG API Machinery]
• Kubeadm: graduate the UnversionedKubeletConfigMap feature gate to Beta and enable the feature by default. This implies that 1) for new clusters kubeadm will start using the "kube-system/kubelet-config" naming scheme for the kubelet ConfigMap and RBAC rules, instead of the legacy "kubelet-config-x.yy" naming. 2) during upgrade, kubeadm will only write the new scheme ConfigMap and RBAC objects. To disable the feature you can pass "UnversionedKubeletConfigMap: false" in the kubeadm config for new clusters. For upgrade on existing clusters you can also override the behavior by patching the ClusterConfiguration object in "kube-system/kubeadm-config". More details in the associated KEP. (#108027, @neolit123) [SIG Cluster Lifecycle and Testing]
• Remove tolerate-unready-endpoints annotation in Service deprecated from 1.11, use Service.spec.publishNotReadyAddresses instead. (#108020, @tossmilestone) [SIG Apps and Network]
• The in-tree azure plugin has been deprecated. The https://github.com/Azure/kubelogin serves as an out-of-tree replacement via the kubectl/client-go credential plugin mechanism. Users will now see a warning in the logs regarding this deprecation. (#107904, @sabbey37) [SIG Auth]

1.4.2. API Change

• CRD deep copies should no longer contain shallow copies of JSONSchemaProps.XValidations. (#107956, @benluddy) [SIG API Machinery]
• Feature of NonPreemptingPriority is graduated to GA (#107432, @denkensk) [SIG Apps, Scheduling and Testing]
• Fix OpenAPI serialization of the x-kubernetes-validations field (#107970, @liggitt) [SIG API Machinery]
• Kube-apiserver: the metadata.selfLink field can no longer be populated by kube-apiserver; it was deprecated in 1.16 and has not been populated by default in 1.20+. (#107527, @wojtek-t) [SIG API Machinery, Apps, Auth, Autoscaling, CLI, Cloud Provider, Network, Scheduling, Storage and Testing]

1.4.3. Feature

• Kubernetes is now built with Golang 1.17.7 (#108091, @xmudrii) [SIG Release and Testing]
• Remove feature gate SetHostnameAsFQDN. (#108038, @mengjiao-liu) [SIG Node]
• The output of kubectl describe ingress now includes an IngressClass name if available (#107921, @mpuckett159) [SIG CLI]
• The scheduler prints info logs when the extender returned an error. (--v>5) (#107974, @sanposhiho) [SIG Scheduling]
• kubectl config delete-user now supports completion (#107142, @dimbleby) [SIG CLI]
• kubectl create token can now be used to request a service account token, and permission to request service account tokens is added to the edit and admin RBAC roles (#107880, @liggitt) [SIG Auth, CLI and Testing]

1.4.4. Bug or Regression

• A static pod that is rapidly updated was failing to start until the Kubelet was restarted. (#107900, @smarterclayton) [SIG Node and Testing]
• CRI-API: IPs returned by PodSandboxNetworkStatus are ignored by the kubelet for host-network pods. (#106715, @aojea) [SIG Node]
• Fixes bug in TopologyManager for ensuring aligned allocations on machines with more than 2 NUMA nodes (#108052, @klueska) [SIG Node]
• Kubeadm: fix a bug related to a warning printed if the KubeletConfiguration "resolvConf" field value does not match "/run/systemd/resolve/resolv.conf" (#107785, @chendave) [SIG Cluster Lifecycle]
• Kubeadm: fix the bug that 'kubeadm certs generate-csr' command does not remove duplicated SANs (#107982, @SataQiu) [SIG Cluster Lifecycle]
• RunCordonOrUncordon error if drainer has nil Ctx or Client (#105297, @jackfrancis) [SIG CLI]

1.4.5. Other (Cleanup or Flake)

• Added an e2e test to verify that the cluster is not vulnerable to CVE-2021-29923 when using Services with IPs with leading zeros, note that this test is a necessary but not sufficient condition, all the components in the clusters that consume IPs addresses from the APIs MUST interpret them as decimal or discard them. (#107552, @aojea) [SIG Network and Testing]
• Kubectl stack traces will now only print at -v=99 and not -v=6 (#108053, @eddiezane) [SIG CLI]
• Remove kubelet --non-masquerade-cidr deprecated CLI flag (#107096, @hakman) [SIG Cloud Provider and Node]
• [k8s.io/utils/clock]: IntervalClock is now deprecated in favour of SimpleIntervalClock (#108059, @RaghavRoy145) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Storage]

1.5. Dependencies

1.5.1. Added

Nothing has changed.

1.5.2. Changed

• k8s.io/utils: 7d6a63d → 3a6ce19

1.5.3. Removed

Nothing has changed.

2. v1.24.0-alpha.2

2.1. Downloads for v1.24.0-alpha.2

2.1.1. Source Code

2.1.2. Client Binaries

2.1.3. Server Binaries

2.1.4. Node Binaries

2.1.5. Container Images

All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.

2.2. Changelog since v1.24.0-alpha.1

2.3. Urgent Upgrade Notes

2.3.1. (No, really, you MUST read this before you upgrade)

• Docker runtime support using dockshim in the kubelet is now completely removed in 1.24. The kubelet used to have a a module called "dockershim" which implements CRI support for Docker and it has seen maintenance issues in the Kubernetes community. From 1.24 onwards, please move to a container runtime that is a full-fledged implementation of CRI (v1alpha1 or v1 compliant) as they become available. (#97252, @dims) [SIG Cloud Provider, Instrumentation, Network, Node and Testing]
  • The calculations for Pod topology spread skew now excludes nodes that don't match the node affinity/selector. This may lead to unschedulable pods if you previously had pods matching the spreading selector on those excluded nodes (not matching the node affinity/selector), especially when the topologyKey is not node-level. Revisit the node affinity and/or pod selector in the topology spread constraints to avoid this scenario. (#107009, @kerthcet) [SIG Scheduling]

2.4. Changes by Kind

2.4.1. Deprecation

• "kubeadm.k8s.io/v1beta2" has been deprecated and will be removed in a future release, possibly in 3 releases (one year). You should start using "kubeadm.k8s.io/v1beta3" for new clusters. To migrate your old configuration files on disk you can use the "kubeadm config migrate" command. (#107013, @pacoxu) [SIG Cluster Lifecycle]
• Deprecate Service.Spec.LoadBalancerIP. This field was under-specified and its meaning varies across implementations. As of Kubernetes v1.24, users are encouraged to use implementation-specific annotations when available. This field may be removed in a future API version. (#107235, @uablrek) [SIG Apps and Network]
• Kube-apiserver: the insecure address flags --address, --insecure-bind-address, --port and --insecure-port (inert since 1.20) are removed (#106859, @knight42) [SIG API Machinery, Cloud Provider and Cluster Lifecycle]
• The experimental dynamic log sanitization feature has been deprecated and removed in the 1.24 release. The feature is no longer available for use. (#107207, @ehashman) [SIG Instrumentation, Scheduling and Security]
• The insecure address flags --address and --port in kube-controller-manager have been no effect since v1.20 and is removed in v1.24. (#106860, @knight42) [SIG API Machinery, Node and Testing]

2.4.2. API Change

• Add a new metric webhook_fail_open_count to monitor webhooks that fail open (#107171, @ltagliamonte-dd) [SIG API Machinery and Instrumentation]
• Fix failed flushing logs in defer function when kubelet cmd exit 1. (#104774, @kerthcet) [SIG Node and Scheduling]
• Rename metrics evictions_number to evictions_total and mark it as stable. The original evictions_number metrics name is marked as "Deprecated" and will be removed in kubernetes 1.23 (#106366, @cyclinder) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Scheduling, Storage, Testing and Windows]
• The ServiceLBNodePortControl feature graduates to GA. The feature gate will be removed in 1.26. (#107027, @uablrek) [SIG Network and Testing]
• The feature DynamicKubeletConfig is removed from the kubelet. (#106932, @SergeyKanzhelev) [SIG Apps, Auth, Instrumentation, Node and Testing]
• Update default API priority-and-fairness config to avoid endpoint/configmaps operations from controller-manager to all match leader-election priority level. (#106725, @wojtek-t) [SIG API Machinery]

2.4.3. Feature

• A new Priority and Fairness metric 'apiserver_flowcontrol_work_estimate_seats_samples' has been added that tracks the estimated seats associated with a request (#106628, @tkashem) [SIG API Machinery and Instrumentation]
• Add completion for kubectl config set-context. (#106739, @kebe7jun) [SIG CLI]
• Add metric for measuring end-to-end volume mount timing (#107006, @gnufied) [SIG Node and Storage]
• Add more message for no PodSandbox container (#107116, @yxxhero) [SIG Node]
• Added field add_ambient_capabilities to the Capabilities message in the CRI-API. (#104620, @vinayakankugoyal) [SIG Node]
• Added label selector flag to all "kubectl rollout" commands (#99758, @aramperes) [SIG CLI]
• Added prune flag into diff command to simulate apply --prune (#105164, @ardaguclu) [SIG CLI and Testing]
• Adds SetTransform to SharedInformer to allow users to transform objects before they are stored. (#107507, @alexzielenski) [SIG API Machinery]
• Adds proxy-url flag into kubectl config set-cluster (#105566, @ardaguclu) [SIG CLI]
• Adds support for kubectl commands (kubectl exec and kubectl port-forward) via a SOCKS5 proxy. (#105632, @xens) [SIG API Machinery, Architecture, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Storage]
• Feature of PreferNominatedNode is graduated to GA (#106619, @chendave) [SIG Scheduling and Testing]
• In text format, log messages that previously used quoting to prevent multi-line output (for example, text="some \"quotation\", a\nline break") will now be printed with more readable multi-line output without the escape sequences. (#107103, @pohly) [SIG Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Storage]
• Kube-apiserver: when merging lists, Server Side Apply now prefers the order of the submitted request instead of the existing persisted object (#107565, @jiahuif) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Storage and Testing]
• Kube-scheduler remove insecure flags. You can use --bind-address and --secure-port instead. (#106865, @jonyhy96) [SIG Scheduling]
• Kubeadm: add support for dry running "kubeadm reset". The new flag "kubeadm reset --dry-run" is similar to the existing flag for "kubeadm init/join/upgrade" and allows you to see what changes would be applied. (#107512, @SataQiu) [SIG Cluster Lifecycle]

• Kubeadm: default the kubeadm configuration to the containerd socket (Unix: unix:///var/run/containerd/containerd.sock, Windows: "npipe:////./pipe/containerd-containerd") instead of the one for Docker. If the "Init|JoinConfiguration.nodeRegistration.criSocket" field is empty during cluster creation and multiple sockets are found on the host always throw an error and ask the user to specify which one to use by setting the value in the field. Make sure you update any kubeadm configuration files on disk, to not include the dockershim socket unless you are still using kubelet version < 1.24 with kubeadm >= 1.24.

  Remove the DockerValidor and ServiceCheck for the "docker" service from kubeadm preflight. Docker is no longer special cased during host validation and ideally this task should be done in the now external cri-dockerd project where the importance of the compatibility matters.

  Use crictl for all communication with CRI sockets for actions like pulling images and obtaining a list of running containers instead of using the docker CLI in the case of Docker. (#107317, @neolit123) [SIG Cluster Lifecycle]

• Kubectl logs will now warn and default to the first container in a pod. This new behavior brings it in line with kubectl exec. (#105964, @kidlj) [SIG CLI]

• Kubelet: following dockershim related flags are also removed along with dockershim --experimental-dockershim-root-directory, --docker-endpoint, --image-pull-progress-deadline, --network-plugin, --cni-conf-dir,--cni-bin-dir, --cni-cache-dir, --network-plugin-mtu (#106907, @cyclinder) [SIG Cloud Provider, Node and Testing]
• Kubernetes is now built with Golang 1.17.5 (#106956, @cpanato) [SIG API Machinery, Cloud Provider, Instrumentation, Release and Testing]
• Kubernetes is now built with Golang 1.17.6 (#107612, @palnabarun) [SIG Release and Testing]
• OpenStack Cinder CSI migration is now GA and switched on by default, Cinder CSI driver must be installed on clusters on OpenStack for Cinder volumes to work (has been since v1.21). (#107462, @dims) [SIG Scheduling and Storage]
• Remove feature gate ImmutableEphemeralVolumes. (#107152, @mengjiao-liu) [SIG Node and Storage]

• This adds a path /header?key= to agnhost netexec allowing one to view what the header value is of the incoming request.

  Ex:

  $ curl -H "X-Forwarded-For: something" 172.17.0.2:8080/header?key=X-Forwarded-For something (#107796, @alexanderConstantinescu) [SIG Testing]

• Update golang.org/x/net to v0.0.0-20211209124913-491a49abca63 (#106949, @cpanato) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node and Storage]

• We have added a new Priority and Fairness metric apiserver_flowcontrol_request_dispatch_no_accommodation_total' to track the number of times a request dispatch attempt results in a no-accommodation status due to lack of available seats (#106629, @tkashem) [SIG API Machinery and Instrumentation]

2.4.4. Bug or Regression

• A new label type has been added to apiserver_flowcontrol_request_execution_seconds metric - it has the following values:

  • 'regular': indicates that it is a non long running request
  • 'watch': indicates that it is a watch request (#105517, @tkashem) [SIG API Machinery and Instrumentation]

• Add a test to guarantee that conformance clusters require at least 2 untainted nodes (#106313, @aojea) [SIG Architecture and Testing]

• Allow attached volumes to be mounted quicker by skipping exp. backoff when checking for reported-in-use volumes (#106853, @gnufied) [SIG Apps, Node and Storage]
• An inefficient lock in EndpointSlice controller metrics cache has been reworked. Network programming latency may be significantly reduced in certain scenarios, especially in clusters with a large number of Services. (#107091, @robscott) [SIG Apps, Network and Scalability]
• Apiserver will now reject connection attempts to 0.0.0.0/:: when handling a proxy subresource request (#107402, @anguslees) [SIG Network]
• Apiserver, if configured to reconcile the kubernetes.default service endpoints, checks if the configured Service IP range matches the apiserver public address IP family, and fails to start if not. (#106721, @aojea) [SIG API Machinery and Testing]
• Change node staging path for csi driver to use a PV agnostic path. Nodes must be drained before updating the kubelet with this change. (#107065, @saikat-royc) [SIG Storage and Testing]
• Client-go: fix that paged list calls with ResourceVersionMatch set would fail once paging kicked in. (#107311, @fasaxc) [SIG API Machinery]
• Fix Azurefile volumeid collision issue in csi migration (#107575, @andyzhangx) [SIG Cloud Provider and Storage]
• Fix a panic when using invalid output format in kubectl create secret command (#107221, @rikatz) [SIG CLI]
• Fix libct/cg/fs2: fix GetStats for unsupported hugetlb error on Raspbian Bullseye (#106912, @Letme) [SIG Node]
• Fix performance regression in JSON logging caused by syncing stdout every time error was logged. (#107035, @serathius) [SIG Instrumentation and Scalability]
• Fix: azuredisk parameter lowercase translation issue (#107429, @andyzhangx) [SIG Cloud Provider and Storage]
• Fix: delete non existing Azure disk issue (#107406, @andyzhangx) [SIG Cloud Provider]
• Fix: remove outdated ipv4 route when the corresponding node is deleted (#106164, @nilo19) [SIG Cloud Provider]
• Fixed a bug that a pod's .status.nominatedNodeName is not cleared properly, and thus over-occupied system resources. (#106816, @Huang-Wei) [SIG Scheduling and Testing]
• Fixed a bug that could cause a panic when a /healthz request times out. (#107034, @benluddy) [SIG API Machinery]
• Fixed a bug where vSphere client connections where not being closed during testing. Leaked vSphere client sessions were causing resource exhaustion during automated testing. (#107337, @derek-pryor) [SIG Storage and Testing]
• Fixed detaching CSI volumes from nodes when a CSI driver name has prefix "csi-". (#107025, @jsafrane) [SIG Storage]
• Fixed duplicate port opening in kube-proxy when "--nodeport-addresses" is empty (#107413, @tnqn) [SIG Network]
• Fixed kubectl bug where bash completions don't work if --context flag is specified with a value that contains a colon (#107439, @brianpursley) [SIG CLI]
• Fixes a bug where unwanted fields were being returned from a create dry-run: uid and, if generateName was used, name. (#107088, @joejulian) [SIG API Machinery and Testing]
• Fixes a rare race condition handling requests that timeout (#107452, @liggitt) [SIG API Machinery]
• Fixes a regression in 1.23 that incorrectly pruned data from array items of a custom resource that set x-kubernetes-preserve-unknown-fields: true (#107688, @liggitt) [SIG API Machinery]
• Fixes a regression in 1.23 where update requests to previously persisted Service objects that have not been modified since 1.19 can be rejected with an incorrect spec.clusterIPs: Required value error (#107847, @thockin) [SIG API Machinery, Network and Testing]
• Fixes handling of objects with invalid selectors (#107559, @liggitt) [SIG API Machinery, Apps, Scheduling and Storage]
• Fixes regression in CPUManager that it will release exclusive CPUs in app containers inherited from init containers when the init containers were removed. (#104837, @eggiter) [SIG Node]
• Fixes static pod add and removes restarts in certain cases. (#107695, @rphillips) [SIG Node]
• Improve handling of unmount failures when device may be in-use by another container/process (#107789, @gnufied) [SIG Storage]
• Improve rounding of PodTopologySpread scores to offer better scoring when spreading a low number of pods. (#107384, @sanposhiho) [SIG Scheduling]
• Kubeadm: during execution of the "check expiration" command, treat the etcd CA as external if there is a missing etcd CA key file (etcd/ca.key) and perform the proper validation on certificates signed by the etcd CA. Additionally, make sure that the CA for all entries in the output table is included - for both certificates on disk and in kubeconfig files. (#106891, @neolit123) [SIG Cluster Lifecycle]
• No (#107769, @liurupeng) [SIG Cloud Provider and Windows]
• NodeRestriction admission: nodes are now allowed to update PersistentVolumeClaim status fields resizeStatus and allocatedResources when the RecoverVolumeExpansionFailure feature is enabled (#107686, @gnufied) [SIG Auth and Storage]
• Only extend token lifetimes when --service-account-extend-token-expiration is true and the requested token audiences are empty or exactly match all values for --api-audiences (#105954, @jyotimahapatra) [SIG Auth and Testing]
• Removed validation if AppArmor profiles are loaded on the local node. This should be handled by the container runtime. (#97966, @saschagrunert) [SIG Auth, Node and Security]
• Restore NumPDBViolations info of nodes, when HTTPExtender ProcessPreemption. This info will be used in subsequent filtering steps - pickOneNodeForPreemption (#105853, @caden2016) [SIG Scheduling]
• Reverts graceful node shutdown to match 1.21 behavior of setting pods that have not yet successfully completed to "Failed" phase if the GracefulNodeShutdown feature is enabled in kubelet. The GracefulNodeShutdown feature is beta and must be explicitly configured via kubelet config to be enabled in 1.21+. This changes 1.22 and 1.23 behavior on node shutdown to match 1.21. If you do not want pods to be marked terminated on node shutdown in 1.22 and 1.23, disable the GracefulNodeShutdown feature. (#106901, @bobbypage) [SIG Node and Testing]
• Some command line errors (for example, "kubectl list" -> "unknown command") were printed as log message with escaped line breaks instead of a multi-line plain text, which made the error harder to read. (#107044, @pohly) [SIG CLI and Testing]
• Some log messages were logged with "v":0 in JSON output although they are debug messages with a higher verbosity. (#106978, @pohly) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Scheduling and Storage]
• The Service field spec.internalTrafficPolicy is no longer defaulted for Services when the type is ExternalName. The field is also dropped on read when the Service type is ExternalName. (#104846, @andrewsykim) [SIG Apps and Network]

• The feature gate was mentioned as csiMigrationRBD where it should have been CSIMigrationRBD to be in parity with other migration plugins. This release correct the same and keep it as CSIMigrationRBD.

  users who have configured this feature gate as csiMigrationRBD has to reconfigure the same to CSIMigrationRBD from this release. (#107554, @humblec) [SIG Storage]

• When doing make test-integration, you can now usefully include -args $prog_args in KUBE_TEST_ARGS. (#107516, @MikeSpreitzer) [SIG Testing]

2.4.5. Other (Cleanup or Flake)

• --container-runtime kubelet flag is deprecated and will be removed in future releases (#107094, @adisky) [SIG Node]
• Add details about preemption in the event for scheduling failed (#107775, @denkensk) [SIG Scheduling]
• Build/dependencies.yaml: remove the dependency on Docker. With the dockershim removal, core Kubernetes no longer has to track the latest validated version of Docker. (#107607, @neolit123) [SIG Cluster Lifecycle and Node]
• Correct the error message to not use the "--max-resource-write-bytes" & "--json-patch-max-copy-bytes" string. (#106875, @warmchang) [SIG API Machinery]
• E2e tests wait for kube-root-ca.crt to be populated in namespaces for use with projected service account tokens, reducing delays starting those test pods and errors in the logs. (#107763, @smarterclayton) [SIG Testing]
• Fix documentation typo in cloud-provider (#106445, @majst01) [SIG Cloud Provider]
• Fix spelling of implemented in pkg/proxy/apis/config/types.go line 206 (#106453, @davidleitw) [SIG Network]
• Kubeadm: all warning messages are printed to stderr instead of stdout. (#107467, @SataQiu) [SIG Cluster Lifecycle]
• Kubeadm: handle the removal of dockershim related flags for new kubeadm clusters. If kubelet <1.24 is="" on="" the="" host,="" kubeadm="">=1.24 can continue using the built-in dockershim in the kubelet if the user passes the "{Init|Join}Configuration.nodeRegistration.criSocket" value in the kubeadm configuration to be equal to "unix:///var/run/dockershim.sock" on Unix or "npipe:////./pipe/dockershim" on Windows. If kubelet version >=1.24 is on the host, kubeadm >=1.24 will treat all container runtimes as "remote" using the kubelet flags "--container-runtime=remote --container-runtime-endpoint=scheme://some/path". The special management for kubelet <1.24 will be removed in kubeadm 1.25. (#106973, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: make sure that "kubeadm init/join" always use a URL scheme (unix:// on Linux and npipe:// on Windows) when passing a value to the "--container-runtime-endpoint" kubelet flag. This flag's value is taken from the kubeadm configuration "criSocket" field or the "--cri-socket" CLI flag. Automatically add a missing URL scheme to the user configuration in memory, but warn them that they should also update their configuration on disk manually. During "kubeadm upgrade apply/node" mutate the "/var/lib/kubelet/kubeadm-flags.env" file on disk and the "kubeadm.alpha.kubernetes.io/cri-socket" annotation Node object if needed. These automatic actions are temporary and will be removed in a future release. In the future the kubelet may not support CRI endpoints without an URL scheme. (#107295, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: remove the IPv6DualStack feature gate. The feature has been GA and locked to enabled since 1.23. (#106648, @calvin0327) [SIG Cluster Lifecycle and Testing]
• Kubeadm: remove the deprecated output/v1alpha1 API used for machine readable output by some kubeadm commands. In 1.23 kubeadm started using the newer version output/v1alpha2 for the same purpose. (#107468, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: remove the restriction that the ca.crt can only contain one certificate. If there is more than one certificate in the ca.crt file, kubeadm will pick the first one by default. (#107327, @SataQiu) [SIG Cluster Lifecycle]
• Kubectl: restores --dry-run, --dry-run=true, and --dry-run=false for compatibility with pre-1.23 invocations. (#107003, @julianvmodesto) [SIG CLI and Testing]
• Kubernetes e2e framework will use the url "invalid.registry.k8s.io/invalid" instead "invalid.com/invalid" for test that use an invalid registry. (#107455, @aojea) [SIG Testing]
• Mark kubelet --container-runtime-endpoint and --image-service-endpoint CLI flags as stable (#106954, @saschagrunert) [SIG Node]
• Migrate volume/csi/csi-client.go logs to structured logging (#99441, @CKchen0726) [SIG Storage]
• Please check your kubelet command line for enabling features and drop "RuntimeClass" if present. Note that this feature has been on by default since 1.14 and was GA'ed in 1.20. (#106882, @cyclinder) [SIG Node]
• The fluentd-elasticsearch addon is no longer included in the cluster directory. It is available from https://github.com/kubernetes-sigs/instrumentation-addons/tree/master/fluentd-elasticsearch (#107553, @liggitt) [SIG Cloud Provider and Instrumentation]
• This PR deprecates types in k8s.io/apimachinery/util/clock. Please use k8s.io/utils/clock instead. (#106850, @MadhavJivrajani) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Storage]
• Updated cri-tools to v1.23.0 (#107604, @saschagrunert) [SIG Cloud Provider and Release]

2.5. Dependencies

2.5.1. Added

• github.com/armon/go-socks5: e753329

2.5.2. Changed

• github.com/cespare/xxhash/v2: v2.1.1 → v2.1.2
• github.com/moby/term: 9d4ed18 → 3f7ff69
• github.com/opencontainers/runc: v1.0.2 → v1.0.3
• github.com/prometheus/client_golang: v1.11.0 → v1.12.0
• github.com/prometheus/common: v0.28.0 → v0.32.1
• github.com/prometheus/procfs: v0.6.0 → v0.7.3
• github.com/yuin/goldmark: v1.4.0 → v1.4.1
• golang.org/x/mod: v0.4.2 → v0.5.1
• golang.org/x/net: e898025 → 491a49a
• golang.org/x/sys: f4d4317 → da31bd3
• golang.org/x/tools: d4cc65f → v0.1.8
• k8s.io/gengo: 485abfe → c02415c
• k8s.io/klog/v2: v2.30.0 → v2.40.1
• k8s.io/utils: cb0fa31 → 7d6a63d
• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.25 → v0.0.27
• sigs.k8s.io/json: c049b76 → 9f7c6b3
• sigs.k8s.io/structured-merge-diff/v4: v4.1.2 → v4.2.1

2.5.3. Removed

Nothing has changed.

3. v1.24.0-alpha.1

3.1. Downloads for v1.24.0-alpha.1

3.1.1. Source Code

3.1.2. Client Binaries

3.1.3. Server Binaries

3.1.4. Node Binaries

3.2. Changelog since v1.23.0

3.3. Urgent Upgrade Notes

3.3.1. (No, really, you MUST read this before you upgrade)

• Docker runtime support using dockshim in the kubelet is now completely removed in 1.24. The kubelet used to have a a module called "dockershim" which implements CRI support for Docker and it has seen maintenance issues in the Kubernetes community. From 1.24 onwards, please move to a container runtime that is a full-fledged implementation of CRI (v1alpha1 or v1 compliant) as they become available. (#97252, @dims) [SIG Cloud Provider, Instrumentation, Network, Node and Testing]

3.4. Changes by Kind

3.4.1. Feature

• Kubernetes is now built with Golang 1.17.4 (#106833, @cpanato) [SIG API Machinery, Cloud Provider, Instrumentation, Release and Testing]
• The NamespaceDefaultLabelName feature gate, GA since v1.22, is now removed. (#106838, @mengjiao-liu) [SIG Apps and Node]

3.4.2. Bug or Regression

• Address a bug in rbd migration translation plugin (#106878, @humblec) [SIG Storage]
• Fix bug in error messaging for basic-auth and ssh secret validations. (#106179, @vivek-koppuru) [SIG Apps and Auth]
• Kubeadm: allow the "certs check-expiration" command to not require the existence of the cluster CA key (ca.key file) when checking the expiration of managed certificates in kubeconfig files. (#106854, @neolit123) [SIG Cluster Lifecycle]
• Publishing kube-proxy metrics for Windows kernel-mode (#106581, @knabben) [SIG Instrumentation, Network and Windows]
• The deprecated flag --really-crash-for-testing is removed. (#101719, @SergeyKanzhelev) [SIG API Machinery, Network, Node and Testing]
• [Metrics Server] Bump image to v0.5.2 (#106492, @serathius) [SIG Cloud Provider and Instrumentation]

3.4.3. Other (Cleanup or Flake)

• Added an example for the kubectl plugin list command. (#106600, @bergerhoffer) [SIG CLI]
• Kubelet config validation error messages are updated (#105360, @shuheiktgw) [SIG Node]

3.5. Dependencies

3.5.1. Added

Nothing has changed.

3.5.2. Changed

Nothing has changed.

3.5.3. Removed

• github.com/containernetworking/cni: v0.8.1