• v1.20.15

  • Downloads for v1.20.15

    • Source Code
    • Client Binaries
    • Server Binaries
    • Node Binaries

  • Changelog since v1.20.14

  • Changes by Kind

    • Bug or Regression

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.14

  • Downloads for v1.20.14

    • Source Code
    • Client Binaries
    • Server Binaries
    • Node Binaries

  • Changelog since v1.20.13

  • Changes by Kind

    • Bug or Regression

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.13

  • Downloads for v1.20.13

    • Source Code
    • Client Binaries
    • Server Binaries
    • Node Binaries

  • Changelog since v1.20.12

  • Changes by Kind

    • Feature
    • Failing Test
    • Bug or Regression

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.12

  • Downloads for v1.20.12

    • Source Code
    • Client Binaries
    • Server Binaries
    • Node Binaries

  • Changelog since v1.20.11

  • Changes by Kind

    • API Change
    • Bug or Regression
    • Other (Cleanup or Flake)

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.11

  • Downloads for v1.20.11

    • Source Code
    • Client Binaries
    • Server Binaries
    • Node Binaries

  • Changelog since v1.20.10

  • Important Security Information

    • CVE-2021-25741: Symlink Exchange Can Allow Host Filesystem Access

  • Changes by Kind

    • Bug or Regression
    • Other (Cleanup or Flake)

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.10

  • Downloads for v1.20.10

    • Source Code
    • Client Binaries
    • Server Binaries
    • Node Binaries

  • Changelog since v1.20.9

  • Changes by Kind

    • Feature
    • Bug or Regression

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.9

  • Downloads for v1.20.9

    • Source Code
    • Client Binaries
    • Server Binaries
    • Node Binaries

  • Changelog since v1.20.8

  • Changes by Kind

    • Feature
    • Bug or Regression

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.8

  • Downloads for v1.20.8

    • Source Code
    • Client Binaries
    • Server Binaries
    • Node Binaries

  • Changelog since v1.20.7

  • Changes by Kind

    • Feature
    • Failing Test
    • Bug or Regression
    • Other (Cleanup or Flake)

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.7

  • Downloads for v1.20.7

    • Source Code
    • Client Binaries
    • Server Binaries
    • Node Binaries

  • Changelog since v1.20.6

  • Changes by Kind

    • API Change
    • Feature
    • Bug or Regression

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.6

  • Downloads for v1.20.6

    • Source Code
    • Client binaries
    • Server binaries
    • Node binaries

  • Changelog since v1.20.5

  • Important Security Information

    • CVE-2021-25735: Validating Admission Webhook does not observe some previous fields

  • Changes by Kind

    • API Change
    • Feature
    • Bug or Regression
    • Uncategorized

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.5

  • Downloads for v1.20.5

    • Source Code
    • Client binaries
    • Server binaries
    • Node binaries

  • Changelog since v1.20.4

  • Changes by Kind

    • Failing Test
    • Bug or Regression

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.4

  • Downloads for v1.20.4

    • Source Code
    • Client binaries
    • Server binaries
    • Node binaries

  • Changelog since v1.20.3

  • Dependencies
    • Added
    • Changed
    • Removed

• v1.20.3

  • Downloads for v1.20.3

    • Source Code
    • Client binaries
    • Server binaries
    • Node binaries

  • Changelog since v1.20.2

  • Changes by Kind

    • API Change
    • Failing Test
    • Bug or Regression
    • Other (Cleanup or Flake)

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.2

  • Downloads for v1.20.2

    • Source Code
    • Client binaries
    • Server binaries
    • Node binaries

  • Changelog since v1.20.1

  • Changes by Kind

    • Bug or Regression

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.1

  • Downloads for v1.20.1

    • Source Code
    • Client binaries
    • Server binaries
    • Node binaries

  • Changelog since v1.20.0

  • Changes by Kind

    • Bug or Regression

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.0

  • Downloads for v1.20.0

    • Source Code
    • Client Binaries
    • Server Binaries
    • Node Binaries

  • Changelog since v1.19.0

  • What's New (Major Themes)

    • Dockershim deprecation
    • External credential provider for client-go
    • CronJob controller v2 is available through feature gate
    • PID Limits graduates to General Availability
    • API Priority and Fairness graduates to Beta
    • IPv4/IPv6 run
    • go1.15.5
    • CSI Volume Snapshot graduates to General Availability
    • Non-recursive Volume Ownership (FSGroup) graduates to Beta
    • CSIDriver policy for FSGroup graduates to Beta
    • Security Improvements for CSI Drivers (Alpha)
    • Introducing Graceful Node Shutdown (Alpha)
    • Runtime log sanitation
    • Pod resource metrics
    • Introducing RootCAConfigMap
    • kubectl debug graduates to Beta
    • Removing deprecated flags in kubeadm
    • Pod Hostname as FQDN graduates to Beta
    • TokenRequest / TokenRequestProjection graduates to General Availability
    • RuntimeClass feature graduates to General Availability.
    • Cloud Controller Manager now exclusively shipped by Cloud Provider

  • Known Issues

    • Summary API in kubelet doesn't have accelerator metrics

  • Urgent Upgrade Notes

    • (No, really, you MUST read this before you upgrade)

  • Changes by Kind

    • Deprecation
    • API Change
    • Feature
    • Documentation
    • Failing Test
    • Bug or Regression
    • Other (Cleanup or Flake)

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.0-rc.0

  • Downloads for v1.20.0-rc.0

    • Source Code
    • Client binaries
    • Server binaries
    • Node binaries

  • Changelog since v1.20.0-beta.2

  • Changes by Kind

    • Feature
    • Failing Test
    • Bug or Regression

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.0-beta.2

  • Downloads for v1.20.0-beta.2

    • Source Code
    • Client binaries
    • Server binaries
    • Node binaries

  • Changelog since v1.20.0-beta.1

  • Urgent Upgrade Notes

    • (No, really, you MUST read this before you upgrade)

  • Changes by Kind

    • Deprecation
    • API Change
    • Feature
    • Documentation
    • Bug or Regression
    • Other (Cleanup or Flake)

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.0-beta.1

  • Downloads for v1.20.0-beta.1

    • Source Code
    • Client binaries
    • Server binaries
    • Node binaries

  • Changelog since v1.20.0-beta.0

  • Changes by Kind

    • Deprecation
    • API Change
    • Feature
    • Documentation
    • Bug or Regression
    • Other (Cleanup or Flake)

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.0-beta.0

  • Downloads for v1.20.0-beta.0

    • Source Code
    • Client binaries
    • Server binaries
    • Node binaries

  • Changelog since v1.20.0-alpha.3

  • Urgent Upgrade Notes

    • (No, really, you MUST read this before you upgrade)

  • Changes by Kind

    • Deprecation
    • API Change
    • Feature
    • Documentation
    • Bug or Regression
    • Other (Cleanup or Flake)

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.0-alpha.3

  • Downloads for v1.20.0-alpha.3

    • Source Code
    • Client binaries
    • Server binaries
    • Node binaries

  • Changelog since v1.20.0-alpha.2

  • Changes by Kind

    • API Change
    • Feature
    • Bug or Regression
    • Other (Cleanup or Flake)

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.0-alpha.2

  • Downloads for v1.20.0-alpha.2

    • Source Code
    • Client binaries
    • Server binaries
    • Node binaries

  • Changelog since v1.20.0-alpha.1

  • Changes by Kind

    • Deprecation
    • API Change
    • Feature
    • Bug or Regression
    • Other (Cleanup or Flake)

  • Dependencies

    • Added
    • Changed
    • Removed

• v1.20.0-alpha.1

  • Downloads for v1.20.0-alpha.1

    • Source Code
    • Client binaries
    • Server binaries
    • Node binaries

  • Changelog since v1.20.0-alpha.0

  • Urgent Upgrade Notes

    • (No, really, you MUST read this before you upgrade)

  • Changes by Kind

    • Deprecation
    • API Change
    • Feature
    • Documentation
    • Failing Test
    • Bug or Regression
    • Other (Cleanup or Flake)

  • Dependencies

    • Added
    • Changed
    • Removed

1. v1.20.15

1.1. Downloads for v1.20.15

1.1.1. Source Code

1.1.2. Client Binaries

1.1.3. Server Binaries

1.1.4. Node Binaries

1.2. Changelog since v1.20.14

1.3. Changes by Kind

1.3.1. Bug or Regression

• Fixes a panic in kube-scheduler handling pods with invalid selectors (#107558, @Zheaoli) [SIG Scheduling]
• Fixes a rare race condition handling requests that timeout (#107461, @liggitt) [SIG API Machinery]

1.4. Dependencies

1.4.1. Added

Nothing has changed.

1.4.2. Changed

Nothing has changed.

1.4.3. Removed

Nothing has changed.

2. v1.20.14

2.1. Downloads for v1.20.14

2.1.1. Source Code

2.1.2. Client Binaries

2.1.3. Server Binaries

2.1.4. Node Binaries

2.2. Changelog since v1.20.13

2.3. Changes by Kind

2.3.1. Bug or Regression

• Ensure Pods are removed from the scheduler cache when the scheduler misses deletion events due to transient errors (#106695, @alculquicondor) [SIG Scheduling]
• Fix: skip instance not found when decoupling vmss from lb (#105834, @nilo19) [SIG Cloud Provider]
• Fixed SELinux relabeling of CSI volumes after CSI driver failure. (#106554, @jsafrane) [SIG Node and Storage]
• Kubeadm: allow the "certs check-expiration" command to not require the existence of the cluster CA key (ca.key file) when checking the expiration of managed certificates in kubeconfig files. (#106927, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: during execution of the "check expiration" command, treat the etcd CA as external if there is a missing etcd CA key file (etcd/ca.key) and perform the proper validation on certificates signed by the etcd CA. Additionally, make sure that the CA for all entries in the output table is included - for both certificates on disk and in kubeconfig files. (#106923, @neolit123) [SIG Cluster Lifecycle]
• Scheduler's assumed pods have 2min instead of 30s to receive nodeName pod updates (#106686, @ahg-g) [SIG Scheduling]

2.4. Dependencies

2.4.1. Added

Nothing has changed.

2.4.2. Changed

Nothing has changed.

2.4.3. Removed

Nothing has changed.

3. v1.20.13

3.1. Downloads for v1.20.13

3.1.1. Source Code

3.1.2. Client Binaries

3.1.3. Server Binaries

3.1.4. Node Binaries

3.2. Changelog since v1.20.12

3.3. Changes by Kind

3.3.1. Feature

• Update debian-base, debian-iptables, setcap images to pick up CVE fixes
  • Debian-base to v1.9.0
  • Debian-iptables to v1.6.7 (#106148, @cpanato) [SIG Release and Testing]

3.3.2. Failing Test

• Fixes hostpath storage e2e tests within SELinux enabled env (#105788, @Elbehery) [SIG Testing]

3.3.3. Bug or Regression

• EndpointSlice Mirroring controller now cleans up managed EndpointSlices when a Service selector is added (#106136, @robscott) [SIG Apps, Network and Testing]
• Fix concurrent map access causing panics when logging timed-out API calls. (#106124, @marseel) [SIG API Machinery]
• Support more than 100 disk mounts on Windows (#105673, @andyzhangx) [SIG Storage and Windows]

3.4. Dependencies

3.4.1. Added

Nothing has changed.

3.4.2. Changed

• k8s.io/kube-openapi: d219536 → 83f114c

3.4.3. Removed

Nothing has changed.

4. v1.20.12

4.1. Downloads for v1.20.12

4.1.1. Source Code

4.1.2. Client Binaries

4.1.3. Server Binaries

4.1.4. Node Binaries

4.2. Changelog since v1.20.11

4.3. Changes by Kind

4.3.1. API Change

• Kube-apiserver: Fixes handling of CRD schemas containing literal null values in enums (#104990, @liggitt) [SIG API Machinery, Apps and Network]

4.3.2. Bug or Regression

• Detach volumes from vSphere nodes not tracked by attach-detach controller (#104910, @gnufied) [SIG Cloud Provider and Storage]
• Fix: consolidate logs for instance not found error (#105364, @nilo19) [SIG Cloud Provider]
• Fix: ignore not a VMSS error for VMAS nodes in EnsureBackendPoolDeleted. (#105404, @ialidzhikov) [SIG Cloud Provider]
• Fix: ignore the case when updating Azure tags (#104687, @nilo19) [SIG Cloud Provider]
• Revert PR #102925 which introduced unexpected scheduling behavior based on balanced resource allocation (#105239, @damemi) [SIG Scheduling]
• Updates golang.org/x/text to v0.3.6 to fix CVE-2020-28852 (#102602, @jonesbr17) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Node]

4.3.3. Other (Cleanup or Flake)

• Allow CSI drivers to just run offline expansion tests (#102665, @gnufied) [SIG Storage and Testing]

4.4. Dependencies

4.4.1. Added

Nothing has changed.

4.4.2. Changed

• golang.org/x/text: v0.3.4 → v0.3.6

4.4.3. Removed

Nothing has changed.

5. v1.20.11

5.1. Downloads for v1.20.11

5.1.1. Source Code

5.1.2. Client Binaries

5.1.3. Server Binaries

5.1.4. Node Binaries

5.2. Changelog since v1.20.10

5.3. Important Security Information

This release contains changes that address the following vulnerabilities:

5.3.1. CVE-2021-25741: Symlink Exchange Can Allow Host Filesystem Access

A security issue was discovered in Kubernetes where a user may be able to create a container with subpath volume mounts to access files & directories outside of the volume, including on the host filesystem.

Affected Versions:

• kubelet v1.22.0 - v1.22.1
• kubelet v1.21.0 - v1.21.4
• kubelet v1.20.0 - v1.20.10
• kubelet <= v1.19.14

Fixed Versions:

• kubelet v1.22.2
• kubelet v1.21.5
• kubelet v1.20.11
• kubelet v1.19.15

This vulnerability was reported by Fabricio Voznika and Mark Wolters of Google.

CVSS Rating: High (8.8) CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H

5.4. Changes by Kind

5.4.1. Bug or Regression

• Fix: skip case sensitivity when checking Azure NSG rules fix: ensure InstanceShutdownByProviderID return false for creating Azure VMs (#104448, @feiskyer) [SIG Cloud Provider]
• Kube-proxy: delete stale conntrack UDP entries for loadbalancer ingress IP. (#104152, @aojea) [SIG Network]
• Metrics changes: Fix exposed buckets of scheduler_volume_scheduling_duration_seconds_bucket metric (#100720, @dntosas) [SIG Apps, Instrumentation, Scheduling and Storage]
• Pass additional flags to subpath mount to avoid flakes in certain conditions (#104348, @mauriciopoppe) [SIG Storage]
• When using kubectl replace (or the equivalent API call) on a Service, the caller no longer needs to do a read-modify-write cycle to fetch the allocated values for .spec.clusterIP and .spec.ports[].nodePort. Instead the API server will automatically carry these forward from the original object when the new object does not specify them. (#104674, @thockin) [SIG Network]

5.4.2. Other (Cleanup or Flake)

• Kube-apiserver: sets an upper-bound on the lifetime of idle keep-alive connections and time to read the headers of incoming requests (#103958, @liggitt) [SIG API Machinery and Node]

5.5. Dependencies

5.5.1. Added

Nothing has changed.

5.5.2. Changed

Nothing has changed.

5.5.3. Removed

Nothing has changed.

6. v1.20.10

6.1. Downloads for v1.20.10

6.1.1. Source Code

6.1.2. Client Binaries

6.1.3. Server Binaries

6.1.4. Node Binaries

6.2. Changelog since v1.20.9

6.3. Changes by Kind

6.3.1. Feature

• Kubernetes 1.20.x is now built using Go 1.15.15 (#104215, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]

6.3.2. Bug or Regression

• Disable aufs module for gce clusters (#103831, @lizhuqi) [SIG Cloud Provider]
• Fix kube-apiserver metric reporting for the deprecated watch path of /api//watch/... (#104191, @wojtek-t) [SIG API Machinery and Instrumentation]
• Fix: Provide IPv6 support for internal load balancer (#103794, @nilo19) [SIG Cloud Provider]
• Fix: ignore not a VMSS error for VMAS nodes in reconcileBackendPools (#103997, @nilo19) [SIG Cloud Provider]
• Fix: return empty VMAS name if using standalone VM (#103470, @nilo19) [SIG Cloud Provider]
• Fixed a bug that scheduler extenders are not called on preemptions (#103019, @ordovicia) [SIG Scheduling]
• Fixes an issue cleaning up CertificateSigningRequest objects with an unparseable status.certificate field (#103949, @liggitt) [SIG Apps and Auth]
• Fixes issue with websocket-based watches of Service objects not closing correctly on timeout (#102542, @liggitt) [SIG API Machinery and Testing]

6.4. Dependencies

6.4.1. Added

Nothing has changed.

6.4.2. Changed

• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.19 → v0.0.22

6.4.3. Removed

Nothing has changed.

7. v1.20.9

7.1. Downloads for v1.20.9

7.1.1. Source Code

7.1.2. Client Binaries

7.1.3. Server Binaries

7.1.4. Node Binaries

7.2. Changelog since v1.20.8

7.3. Changes by Kind

7.3.1. Feature

• Kubernetes 1.20.x is now built using Go 1.15.14 (#103677, @puerco) [SIG Cloud Provider, Instrumentation, Release and Testing]
• Updates the following images to pick up CVE fixes:
  • debian to v1.8.0
  • debian-iptables to v1.6.5
  • setcap to v2.0.3 (#103235, @thejoycekung) [SIG API Machinery, Release and Testing]

7.3.2. Bug or Regression

• Fix scoring for NodeResourcesMostAllocated and NodeResourcesBalancedAllocation plugins when nodes have containers with no requests. This was leaving to under-utilization of small nodes. (#102925, @alculquicondor) [SIG Scheduling]
• Switch scheduler to generate the merge patch on pod status instead of the full pod (#103133, @marwanad) [SIG Scheduling]
• VSphere: Fix regression during attach disk if datastore is within a storage folder or datastore cluster. (#102999, @gnufied) [SIG Cloud Provider]

7.4. Dependencies

7.4.1. Added

Nothing has changed.

7.4.2. Changed

• sigs.k8s.io/structured-merge-diff/v4: v4.0.3 → v4.1.2

7.4.3. Removed

Nothing has changed.

8. v1.20.8

8.1. Downloads for v1.20.8

8.1.1. Source Code

8.1.2. Client Binaries

8.1.3. Server Binaries

8.1.4. Node Binaries

8.2. Changelog since v1.20.7

8.3. Changes by Kind

8.3.1. Feature

• Kubernetes is now built using Go 1.15.13 (#102786, @thejoycekung) [SIG Cloud Provider, Instrumentation, Release and Testing]

8.3.2. Failing Test

• Fixes the should receive events on concurrent watches in same order conformance test to work properly on clusters that auto-create additional configmaps in namespaces (#101950, @liggitt) [SIG API Machinery and Testing]

8.3.3. Bug or Regression

• Added jitter factor to lease controller that better smears load on kube-apiserver over time. (#101652, @marseel) [SIG API Machinery and Scalability]
• Avoid caching the Azure VMSS instances whose network profile is nil (#100948, @feiskyer) [SIG Cloud Provider]
• Azure: avoid setting cached Sku when updating VMSS and VMSS instances (#102005, @feiskyer) [SIG Cloud Provider]
• Fix a bug on the endpoint slices mirroring controller where endpoint NotReadyAddresses were mirrored as Ready to the corresponding EndpointSlice (#102683, @aojea) [SIG Apps and Network]
• Fix a bug that a preemptor pod may exist as a phantom in the scheduler. (#102498, @Huang-Wei) [SIG Scheduling]
• Fix errors when accessing Windows container stats for Dockershim (#98510, @jsturtevant) [SIG Node and Windows]
• Fix removing pods from podTopologyHints mapping (#101896, @aheng-ch) [SIG Node]
• Fix: avoid nil-pointer panic when checking the frontend IP configuration (#101739, @nilo19) [SIG Cloud Provider]
• Fix: delete non existing disk issue (#102083, @andyzhangx) [SIG Cloud Provider]
• Fixed false-positive uncertain volume attachments, which led to unexpected detachment of CSI migrated volumes (#101737, @Jiawei0227) [SIG Apps and Storage]
• Fixed garbage collection of dangling VolumeAttachments for PersistentVolumes migrated to CSI on startup of kube-controller-manager. (#102176, @timebertt) [SIG Apps and Storage]
• Improve speed of vSphere PV provisioning and reduce number of API calls (#102350, @gnufied) [SIG Cloud Provider and Storage]
• Kubeadm: remove the "ephemeral_storage" request from the etcd static pod that kubeadm deploys on stacked etcd control plane nodes. This request has caused sporadic failures on some setups due to a problem in the kubelet with cadvisor and the LocalStorageCapacityIsolation feature gate. See this issue for more details: https://github.com/kubernetes/kubernetes/issues/99305 (#102673, @jackfrancis) [SIG Cluster Lifecycle]
• Register/Deregister Targets in chunks for AWS TargetGroup (#101592, @M00nF1sh) [SIG Cloud Provider]
• Respect annotation size limit for server-side apply updates to the client-side apply annotation. Also, fix opt-out of this behavior by setting the client-side apply annotation to the empty string. (#102105, @julianvmodesto) [SIG API Machinery]
• Reverted the previous fix for portforward cleanup because it introduced a kubelet regression which can lead into segmentation faults. (#102586, @saschagrunert) [SIG API Machinery and Node]
• ServiceOwnsFrontendIP shouldn't report error when the public IP doesn't match (#102516, @nilo19) [SIG Cloud Provider]

8.3.4. Other (Cleanup or Flake)

• Update the Debian images to pick up CVE fixes in the base images:
  • Update the debian-base image to v1.7.0
  • Update the debian-iptables image to v1.6.1 (#102341, @cpanato) [SIG API Machinery and Testing]

8.4. Dependencies

8.4.1. Added

Nothing has changed.

8.4.2. Changed

• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.15 → v0.0.19

8.4.3. Removed

Nothing has changed.

9. v1.20.7

9.1. Downloads for v1.20.7

9.1.1. Source Code

9.1.2. Client Binaries

9.1.3. Server Binaries

9.1.4. Node Binaries

9.2. Changelog since v1.20.6

9.3. Changes by Kind

9.3.1. API Change

• We have added a new Priority & Fairness rule that exempts all probes (/readyz, /healthz, /livez) to prevent restarting of "healthy" kube-apiserver instance(s) by kubelet. (#101112, @tkashem) [SIG API Machinery]

9.3.2. Feature

• Kubernetes is now built using go1.15.11 (#101192, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]
• Kubernetes is now built using go1.15.12 (#101845, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]

9.3.3. Bug or Regression

• Azurefile: Normalize share name to not include capital letters (#100731, @kassarl) [SIG Cloud Provider and Storage]
• EndpointSlice IP validation now matches Endpoints IP validation. (#101084, @robscott) [SIG Apps and Network]
• EndpointSlice controllers are less likely to create duplicate EndpointSlices. (#101763, @aojea) [SIG Apps and Network]
• Ensure service deleted when the Azure resource group has been deleted (#100944, @feiskyer) [SIG Cloud Provider]
• Fix panic in JSON logging format caused by missing Duration encoder (#101158, @serathius) [SIG API Machinery, Cluster Lifecycle and Instrumentation]
• Fix smb mount PermissionDenied issue on Windows (#99550, @andyzhangx) [SIG Cloud Provider, Storage and Windows]
• Fix: azure file inline volume namespace issue in csi migration translation (#101235, @andyzhangx) [SIG Apps, Cloud Provider, Node and Storage]
• Fix: not tagging static public IP (#101752, @nilo19) [SIG Cloud Provider]
• Fix: set "host is down" as corrupted mount (#101398, @andyzhangx) [SIG Cloud Provider and Storage]
• Fixed a bug where startupProbe stopped working after a container's first restart (#101093, @wzshiming) [SIG Node]
• Fixed port-forward memory leak for long-running and heavily used connections. (#99839, @saschagrunert) [SIG API Machinery and Node]
• Kubectl create service now respects namespace flag (#101005, @zxh326) [SIG CLI]
• Kubelet: improve the performance when waiting for a synchronization of the node list with the kube-apiserver (#99336, @neolit123) [SIG Node]
• No support endpointslice in linux userpace mode (#101503, @JornShen) [SIG Network]
• Renames the timeout field for the DelegatingAuthenticationOptions to TokenRequestTimeout and set the timeout only for the token review client. Previously the timeout was also applied to watches making them reconnecting every 10 seconds. (#101103, @p0lyn0mial) [SIG API Machinery, Auth and Cloud Provider]
• Respect ExecProbeTimeout=false for dockershim (#101126, @jackfrancis) [SIG Node and Testing]

9.4. Dependencies

9.4.1. Added

Nothing has changed.

9.4.2. Changed

Nothing has changed.

9.4.3. Removed

Nothing has changed.

10. v1.20.6

10.1. Downloads for v1.20.6

10.1.1. Source Code

10.1.2. Client binaries

10.1.3. Server binaries

10.1.4. Node binaries

10.2. Changelog since v1.20.5

10.3. Important Security Information

This release contains changes that address the following vulnerabilities:

10.3.1. CVE-2021-25735: Validating Admission Webhook does not observe some previous fields

A security issue was discovered in kube-apiserver that could allow node updates to bypass a Validating Admission Webhook. You are only affected by this vulnerability if you run a Validating Admission Webhook for Nodes that denies admission based at least partially on the old state of the Node object.

Note: This only impacts validating admission plugins that rely on old values in certain fields, and does not impact calls from kubelet that go through the built-in NodeRestriction admission plugin.

Affected Versions:

• kube-apiserver v1.20.0 - v1.20.5
• kube-apiserver v1.19.0 - v1.19.9
• kube-apiserver <= v1.18.17

Fixed Versions:

• kube-apiserver v1.21.0
• kube-apiserver v1.20.6
• kube-apiserver v1.19.10
• kube-apiserver v1.18.18

This vulnerability was reported by Rogerio Bastos & Ari Lima from RedHat

CVSS Rating: Medium (6.5) CVSS:3.0/AV:N/AC:L/PR:H/UI:N/S:U/C:N/I:H/A:H

10.4. Changes by Kind

10.4.1. API Change

• Fixes using server-side apply with APIService resources (#100714, @kevindelgado) [SIG API Machinery, Apps and Testing]
• Regenerate protobuf code to fix CVE-2021-3121 (#100501, @joelsmith) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node and Storage]

10.4.2. Feature

• AWS cloudprovider supports auto-discovering subnets without any kubernetes.io/cluster/ tags. It also supports additional service annotation service.beta.kubernetes.io/aws-load-balancer-subnets to manually configure the subnets. (#97431, @kishorj) [SIG Cloud Provider]
• Kubernetes is now built using go1.15.10 (#100375, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]

10.4.3. Bug or Regression

• 10.5. Changelog

  10.5.1. General

  • Fix priority expander falling back to a random choice even though there is a higher priority option to choose
  • Clone kubernetes/kubernetes in update-vendor.sh shallowly, instead of fetching all revisions
  • Speed up binpacking by reducing the number of PreFilter calls (call once per pod instead of #pods*#nodes times)
  • Speed up finding unneeded nodes by 5x+ in very large clusters by reducing the number of PreFilter calls
  • Expose --max-nodes-total as a metric
  • Errors in IncreaseSize changed from type apiError to cloudProviderError
  • Make build-in-docker and test-in-docker work on Linux systems with SELinux enabled
  • Fix an error where existing nodes were not considered as destinations while finding place for pods in scale-down simulations
  • Remove redundant log lines and reduce severity around parsing kubeEnv
  • Don't treat nodes created by virtual kubelet as nodes from non-autoscaled node groups
  • Remove redundant logging around calculating node utilization
  • Add configurable --network and --rm flags for docker in Makefile
  • Subtract DaemonSet pods' requests from node allocatable in the denominator while computing node utilization
  • Include taints by condition when determining if a node is unready/still starting
  • Fix update-vendor.sh to work on OSX and zsh
  • Add best-effort eviction for DaemonSet pods while scaling down non-empty nodes
  • Add build support for ARM64

  10.5.2. AliCloud

  • Add missing daemonsets and replicasets to ALI example cluster role

  10.5.3. Apache CloudStack

  • Add support for Apache CloudStack

  10.5.4. AWS

  • Regenerate list of EC2 instances
  • Fix pricing endpoint in AWS China Region

  10.5.5. Azure

  • Add optional jitter on initial VMSS VM cache refresh, keep the refreshes spread over time
  • Serve from cache for the whole period of ongoing throttling
  • Fix unwanted VMSS VMs cache invalidation
  • Enforce setting the number of retries if cloud provider backoff is enabled
  • Don't update capacity if VMSS provisioning state is updating
  • Support allocatable resources overrides via VMSS tags
  • Add missing stable labels in template nodes
  • Proactively set instance status to deleting on node deletions

  10.5.6. Cluster API

  • Migrate interaction with the API from using internal types to using Unstructured
  • Improve tests to work better with constrained resources
  • Add support for node auto-discovery
  • Add support for --cloud-config
  • Update group identifier to use for Cluster API annotations

  10.5.7. Exoscale

  • Add support for Exoscale

  10.5.8. GCE

  • Decrease the number of GCE Read Requests made while deleting nodes
  • Base pricing of custom instances on their instance family type
  • Add pricing information for missing machine types
  • Add pricing information for different GPU types
  • Ignore the new topology.gke.io/zone label when comparing groups
  • Add missing stable labels to template nodes

  10.5.9. HuaweiCloud

  • Add auto scaling group support
  • Implement node group by AS
  • Implement getting desired instance number of node group
  • Implement increasing node group size
  • Implement TemplateNodeInfo
  • Implement caching instances

  10.5.10. IONOS

  • Add support for IONOS

  10.5.11. Kubemark

  • Skip non-kubemark nodes while computing node information for node groups.

  10.5.12. Magnum

  • Add Magnum support in the Cluster Autoscaler helm chart

  10.5.13. Packet

  • Allow empty nodepools
  • Add support for multiple nodepools
  • Add pricing support

  10.6. Image

  Image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.20.0 (#97012, @towca) [SIG Cloud Provider]

• Fixed a bug where a high churn of events was causing master instability by reducing the maximum number of objects (events) attached to a single etcd lease. (#100084, @mborsz) [SIG API Machinery, Instrumentation and Scalability]

• Fixed a race condition on API server startup ensuring previously created webhook configurations are effective before the first write request is admitted. (#95783, @roycaihw) [SIG API Machinery]
• Fixes a data race issue in the priority and fairness API server filter (#100667, @tkashem) [SIG API Machinery]
• Kubectl: Fixed panic when describing an ingress backend without an API Group (#100541, @eddiezane) [SIG CLI]
• Reverts breaking change to inline AzureFile volumes in v1.20.2-v1.20.5; referenced secrets are now correctly searched for in the same namespace as the pod as in previous releases. (#100399, @andyzhangx) [SIG Cloud Provider and Storage]
• The endpointslice mirroring controller mirrors endpoints annotations and labels to the generated endpoint slices, it also ensures that updates on any of these fields on the endpoints are mirrored. The well-known annotation endpoints.kubernetes.io/last-change-trigger-time is skipped and not mirrored. (#100443, @aojea) [SIG Apps, Network and Testing]
• The maximum number of ports allowed in EndpointSlices has been increased from 100 to 20,000 (#99795, @robscott) [SIG Network]

10.6.1. Uncategorized

• GCE L4 Loadbalancers now handle > 5 ports in service spec correctly. (#99595, @prameshj) [SIG Cloud Provider]

10.7. Dependencies

10.7.1. Added

Nothing has changed.

10.7.2. Changed

• github.com/gogo/protobuf: v1.3.1 → v1.3.2
• github.com/kisielk/errcheck: v1.2.0 → v1.5.0
• github.com/yuin/goldmark: v1.1.27 → v1.2.1
• golang.org/x/sync: cd5d95a → 67f06af
• golang.org/x/tools: c1934b7 → 113979e
• sigs.k8s.io/structured-merge-diff/v4: v4.0.2 → v4.0.3

10.7.3. Removed

Nothing has changed.

11. v1.20.5

11.1. Downloads for v1.20.5

11.1.1. Source Code

11.1.2. Client binaries

11.1.3. Server binaries

11.1.4. Node binaries

11.2. Changelog since v1.20.4

11.3. Changes by Kind

11.3.1. Failing Test

• Fix handing special characters in the volume path on Windows (#99008, @yujuhong) [SIG Storage]
• Kube-proxy: fix a bug on UDP NodePort Services where stale conntrack entries may blackhole the traffic directed to the NodePort. (#98305, @aojea) [SIG Network]

11.3.2. Bug or Regression

• Avoid systemd-logind loading configuration warning (#97950, @wzshiming) [SIG Node]
• Count pod overhead against an entity's ResourceQuota (#99600, @gjkim42) [SIG API Machinery and Node]
• EndpointSlice controller is now less likely to emit FailedToUpdateEndpointSlices events. (#100113, @robscott) [SIG Apps and Network]
• EndpointSliceMirroring controller is now less likely to emit FailedToUpdateEndpointSlices events. (#100143, @robscott) [SIG Apps and Network]
• Ensure only one LoadBalancer rule is created when HA mode is enabled (#99825, @feiskyer) [SIG Cloud Provider]
• Fix kubelet from panic after getting the wrong signal (#98200, @wzshiming) [SIG Node]
• Fix repeatedly acquire the inhibit lock (#98088, @wzshiming) [SIG Node]
• Fixed bug that caused cAdvisor to incorrectly detect single-socket multi-NUMA topology. (#99207, @iwankgb) [SIG Node]
• Fixing a bug where a failed node may not have the NoExecute taint set correctly (#98168, @CKchen0726) [SIG Apps and Node]
• Kubelet now cleans up orphaned volume directories automatically (#95301, @lorenz) [SIG Node and Storage]
• Resolves spurious Failed to list *v1.Secret or Failed to list *v1.ConfigMap messages in kubelet logs. (#99538, @liggitt) [SIG Auth and Node]
• Sync node status during kubelet node shutdown. Adds an pod admission handler that rejects new pods when the node is in progress of shutting down. (#98005, @wzshiming) [SIG Node]
• We will no longer automatically delete all data when a failure is detected during creation of the volume data file on a CSI volume. Now we will only remove the data file and volume path. (#96021, @huffmanca) [SIG Storage]

11.4. Dependencies

11.4.1. Added

Nothing has changed.

11.4.2. Changed

• github.com/google/cadvisor: v0.38.7 → v0.38.8
• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.14 → v0.0.15

11.4.3. Removed

Nothing has changed.

12. v1.20.4

12.1. Downloads for v1.20.4

12.1.1. Source Code

12.1.2. Client binaries

12.1.3. Server binaries

12.1.4. Node binaries

12.2. Changelog since v1.20.3

12.3. Dependencies

12.3.1. Added

Nothing has changed.

12.3.2. Changed

Nothing has changed.

12.3.3. Removed

Nothing has changed.

13. v1.20.3

13.1. Downloads for v1.20.3

13.1.1. Source Code

13.1.2. Client binaries

13.1.3. Server binaries

13.1.4. Node binaries

13.2. Changelog since v1.20.2

13.3. Changes by Kind

13.3.1. API Change

• Kubernetes is now built using go1.15.8 (#98962, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]

13.3.2. Failing Test

• Kubelet: the HostPort implementation in dockershim was not taking into consideration the HostIP field, causing that the same HostPort can not be used with different IP addresses. This bug causes the conformance test "HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol" to fail. (#98838, @aojea) [SIG Network and Node]

13.3.3. Bug or Regression

• Aggregate errors when putting vmss (#98350, @nilo19) [SIG Cloud Provider]
• Avoid marking node as Ready until node has synced with API servers at least once (#97995, @ehashman) [SIG Node]
• Cleanup subnet in frontend IP configs to prevent huge subnet request bodies in some scenarios. (#98132, @nilo19) [SIG Cloud Provider]
• Fix CSI-migrated inline EBS volumes failing to mount if their volumeID is prefixed by aws:// (#96821, @wongma7) [SIG Storage]
• Fix azure file migration issue (#97877, @andyzhangx) [SIG Auth, Cloud Provider and Storage]
• Fix kubectl-convert import known versions (#97754, @wzshiming) [SIG CLI and Testing]
• Fix the description of command line flags that can override --config (#98786, @changshuchao) [SIG Scheduling]
• Fix the panic when kubelet registers if a node object already exists with no Status.Capacity or Status.Allocatable (#97803, @TeddyAndrieux) [SIG Node]
• Fix the regression with the slow pods termination. Before this fix pods may take an additional time to terminate - up to one minute. Reversing the change that ensured that CNI resources cleaned up when the pod is removed on API server. (#97980, @SergeyKanzhelev) [SIG Node]
• Fix to recover CSI volumes from certain dangling attachments (#96617, @yuga711) [SIG Apps and Storage]
• Fixed a bug that the kubelet cannot start on BtrfS. (#98014, @gjkim42) [SIG Node]
• Fixed an issue with garbage collection failing to clean up namespaced children of an object also referenced incorrectly by cluster-scoped children (#98068, @liggitt) [SIG API Machinery and Apps]
• Fixed provisioning of Cinder volumes migrated to CSI when StorageClass with AllowedTopologies was used. (#98311, @jsafrane) [SIG Storage]
• Fixes a panic in the disruption budget controller for PDB objects with invalid selectors (#98775, @ialidzhikov) [SIG Apps]
• Fixes connection errors when using --volume-host-cidr-denylist or --volume-host-allow-local-loopback (#98436, @liggitt) [SIG Network and Storage]
• Kubeadm: get k8s CI version markers from k8s infra bucket (#98836, @hasheddan) [SIG Cluster Lifecycle and Release]
• Kubelet should ignore cgroup driver check on Windows node. (#98383, @pacoxu) [SIG Node]
• Make podTopologyHints protected by lock (#95111, @choury) [SIG Node]
• Static pods will be deleted gracefully. (#98103, @gjkim42) [SIG Node]
• Truncates a message if it hits the NoteLengthLimit when the scheduler records an event for the pod that indicates the pod has failed to schedule. (#98715, @carlory) [SIG Scheduling]
• Warning about using a deprecated volume plugin is logged only once. (#96751, @jsafrane) [SIG Storage]

13.3.4. Other (Cleanup or Flake)

• Kubeadm: change the default image repository for CI images from 'gcr.io/kubernetes-ci-images' to 'gcr.io/k8s-staging-ci-images' (#97087, @SataQiu) [SIG Cluster Lifecycle]
• Resolves flakes in the Ingress conformance tests due to conflicts with controllers updating the Ingress object (#98430, @liggitt) [SIG Network and Testing]

13.4. Dependencies

13.4.1. Added

Nothing has changed.

13.4.2. Changed

• github.com/google/cadvisor: v0.38.6 → v0.38.7

13.4.3. Removed

Nothing has changed.

14. v1.20.2

14.1. Downloads for v1.20.2

14.1.1. Source Code

14.1.2. Client binaries

14.1.3. Server binaries

14.1.4. Node binaries

14.2. Changelog since v1.20.1

14.3. Changes by Kind

14.3.1. Bug or Regression

• Fix Azure file share not deleted issue when the namespace is deleted (#97417, @andyzhangx) [SIG Cloud Provider and Storage]
• Fix counting error in service/nodeport/loadbalancer quota check (#97826, @pacoxu) [SIG API Machinery and Network]
• Fix missing cadvisor machine metrics. (#97006, @lingsamuel) [SIG Node]
• Fix: azure file latency issue for metadata-heavy workloads (#97082, @andyzhangx) [SIG Cloud Provider and Storage]
• Fixed bug in CPUManager with race on container map access (#97427, @klueska) [SIG Node]
• GCE Internal LoadBalancer sync loop will now release the ILB IP address upon sync failure. An error in ILB forwarding rule creation will no longer leak IP addresses. (#97740, @prameshj) [SIG Cloud Provider and Network]
• Kubeadm: avoid detection of the container runtime for commands that do not need it (#97847, @pacoxu) [SIG Cluster Lifecycle]
• Performance regression #97685 has been fixed. (#97860, @MikeSpreitzer) [SIG API Machinery]
• Use network.Interface.VirtualMachine.ID to get the binded VM Skip standalone VM when reconciling LoadBalancer (#97639, @nilo19) [SIG Cloud Provider]

14.4. Dependencies

14.4.1. Added

Nothing has changed.

14.4.2. Changed

Nothing has changed.

14.4.3. Removed

Nothing has changed.

15. v1.20.1

15.1. Downloads for v1.20.1

15.1.1. Source Code

15.1.2. Client binaries

15.1.3. Server binaries

15.1.4. Node binaries

15.2. Changelog since v1.20.0

15.3. Changes by Kind

15.3.1. Bug or Regression

• AcceleratorStats will be available in the Summary API of kubelet when cri_stats_provider is used. (#97018, @ruiwen-zhao) [SIG Node]
• Fixed FibreChannel volume plugin corrupting filesystems on detach of multipath volumes. (#97013, @jsafrane) [SIG Storage]
• Fixed a bug in kubelet that will saturate CPU utilization after containerd got restarted. (#97175, @hanlins) [SIG Node]
• Kubeadm now installs version 3.4.13 of etcd when creating a cluster with v1.19 (#97284, @pacoxu) [SIG Cluster Lifecycle]
• Kubeadm: Fixes a kubeadm upgrade bug that could cause a custom CoreDNS configuration to be replaced with the default. (#97016, @rajansandeep) [SIG Cluster Lifecycle]

15.4. Dependencies

15.4.1. Added

Nothing has changed.

15.4.2. Changed

• github.com/google/cadvisor: v0.38.5 → v0.38.6

15.4.3. Removed

Nothing has changed.

16. v1.20.0

Documentation

16.1. Downloads for v1.20.0

16.1.1. Source Code

16.1.2. Client Binaries

16.1.3. Server Binaries

16.1.4. Node Binaries

16.2. Changelog since v1.19.0

16.3. What's New (Major Themes)

16.3.1. Dockershim deprecation

Docker as an underlying runtime is being deprecated. Docker-produced images will continue to work in your cluster with all runtimes, as they always have. The Kubernetes community has written a blog post about this in detail with a dedicated FAQ page for it.

16.3.2. External credential provider for client-go

The client-go credential plugins can now be passed in the current cluster information via the KUBERNETES_EXEC_INFO environment variable. Learn more about this on client-go credential plugins documentation.

16.3.3. CronJob controller v2 is available through feature gate

An alternative implementation of the CronJob controller is now available as an alpha feature in this release, which has experimental performance improvement by using informers instead of polling. While this will be the default behavior in the future, you can try them in this release through a feature gate.

16.3.4. PID Limits graduates to General Availability

PID Limits features are now generally available on both SupportNodePidsLimit (node-to-pod PID isolation) and SupportPodPidsLimit (ability to limit PIDs per pod), after being enabled-by-default in beta stage for a year.

16.3.5. API Priority and Fairness graduates to Beta

Initially introduced in 1.18, Kubernetes 1.20 now enables API Priority and Fairness (APF) by default. This allows kube-apiserver to categorize incoming requests by priority levels.

16.3.6. IPv4/IPv6 run

IPv4/IPv6 dual-stack has been reimplemented for 1.20 to support dual-stack Services, based on user and community feedback. If your cluster has dual-stack enabled, you can create Services which can use IPv4, IPv6, or both, and you can change this setting for existing Services. Details are available in updated IPv4/IPv6 dual-stack docs, which cover the nuanced array of options.

We expect this implementation to progress from alpha to beta and GA in coming releases, so we’re eager to have you comment about your dual-stack experiences in #k8s-dual-stack or in enhancements #563.

16.3.7. go1.15.5

go1.15.5 has been integrated into the Kubernetes project as of this release, including other infrastructure related updates on this effort.

16.3.8. CSI Volume Snapshot graduates to General Availability

CSI Volume Snapshot moves to GA in the 1.20 release. This feature provides a standard way to trigger volume snapshot operations in Kubernetes and allows Kubernetes users to incorporate snapshot operations in a portable manner on any Kubernetes environment regardless of supporting underlying storage providers. Additionally, these Kubernetes snapshot primitives act as basic building blocks that unlock the ability to develop advanced, enterprise-grade, storage administration features for Kubernetes: including application or cluster level backup solutions. Note that snapshot support will require Kubernetes distributors to bundle the Snapshot controller, Snapshot CRDs, and validation webhook. In addition, a CSI driver supporting the snapshot functionality must also be deployed on the cluster.

16.3.9. Non-recursive Volume Ownership (FSGroup) graduates to Beta

By default, the fsgroup setting, if specified, recursively updates permissions for every file in a volume on every mount. This can make mount, and pod startup, very slow if the volume has many files. This setting enables a pod to specify a PodFSGroupChangePolicy that indicates that volume ownership and permissions will be changed only when permission and ownership of the root directory do not match with expected permissions on the volume.

16.3.10. CSIDriver policy for FSGroup graduates to Beta

The FSGroup's CSIDriver Policy is now beta in 1.20. This allows CSIDrivers to explicitly indicate if they want Kubernetes to manage permissions and ownership for their volumes via fsgroup.

16.3.11. Security Improvements for CSI Drivers (Alpha)

In 1.20, we introduce a new alpha feature CSIServiceAccountToken. This feature allows CSI drivers to impersonate the pods that they mount the volumes for. This improves the security posture in the mounting process where the volumes are ACL’ed on the pods’ service account without handing out unnecessary permissions to the CSI drivers’ service account. This feature is especially important for secret-handling CSI drivers, such as the secrets-store-csi-driver. Since these tokens can be rotated and short-lived, this feature also provides a knob for CSI drivers to receive NodePublishVolume RPC calls periodically with the new token. This knob is also useful when volumes are short-lived, e.g. certificates.

16.3.12. Introducing Graceful Node Shutdown (Alpha)

The GracefulNodeShutdown feature is now in Alpha. This allows kubelet to be aware of node system shutdowns, enabling graceful termination of pods during a system shutdown. This feature can be enabled through feature gate.

16.3.13. Runtime log sanitation

Logs can now be configured to use runtime protection from leaking sensitive data. Details for this experimental feature is available in documentation.

16.3.14. Pod resource metrics

On-demand metrics calculation is now available through /metrics/resources. When enabled, the endpoint will report the requested resources and the desired limits of all running pods.

16.3.15. Introducing RootCAConfigMap

RootCAConfigMap graduates to Beta, separating from BoundServiceAccountTokenVolume. The kube-root-ca.crt ConfigMap is now available to every namespace, by default. It contains the Certificate Authority bundle for verify kube-apiserver connections.

16.3.16. kubectl debug graduates to Beta

kubectl alpha debug graduates from alpha to beta in 1.20, becoming kubectl debug. kubectl debug provides support for common debugging workflows directly from kubectl. Troubleshooting scenarios supported in this release of kubectl include: Troubleshoot workloads that crash on startup by creating a copy of the pod that uses a different container image or command. Troubleshoot distroless containers by adding a new container with debugging tools, either in a new copy of the pod or using an ephemeral container. (Ephemeral containers are an alpha feature that are not enabled by default.) Troubleshoot on a node by creating a container running in the host namespaces and with access to the host’s filesystem. Note that as a new builtin command, kubectl debug takes priority over any kubectl plugin named “debug”. You will need to rename the affected plugin. Invocations using kubectl alpha debug are now deprecated and will be removed in a subsequent release. Update your scripts to use kubectl debug instead of kubectl alpha debug! For more information about kubectl debug, see Debugging Running Pods on the Kubernetes website, kubectl help debug, or reach out to SIG CLI by visiting #sig-cli or commenting on enhancement #1441.

16.3.17. Removing deprecated flags in kubeadm

kubeadm applies a number of deprecations and removals of deprecated features in this release. More details are available in the Urgent Upgrade Notes and Kind / Deprecation sections.

16.3.18. Pod Hostname as FQDN graduates to Beta

Previously introduced in 1.19 behind a feature gate, SetHostnameAsFQDN is now enabled by default. More details on this behavior are available in documentation for DNS for Services and Pods

16.3.19. TokenRequest / TokenRequestProjection graduates to General Availability

Service account tokens bound to a pod is now a stable feature. The feature gates will be removed in 1.21 release. For more information, refer to notes below on the changelogs.

16.3.20. RuntimeClass feature graduates to General Availability.

The node.k8s.io API groups are promoted from v1beta1 to v1. v1beta1 is now deprecated and will be removed in a future release, please start using v1. (#95718, @SergeyKanzhelev) [SIG Apps, Auth, Node, Scheduling and Testing]

16.3.21. Cloud Controller Manager now exclusively shipped by Cloud Provider

Kubernetes will no longer ship an instance of the Cloud Controller Manager binary. Each Cloud Provider is expected to ship their own instance of this binary. Details for a Cloud Provider to create an instance of such a binary can be found here. Anyone with questions on building a Cloud Controller Manager should reach out to SIG Cloud Provider. Questions about the Cloud Controller Manager on a Managed Kubernetes solution should go to the relevant Cloud Provider. Questions about the Cloud Controller Manager on a non managed solution can be brought up with SIG Cloud Provider.

16.4. Known Issues

16.4.1. Summary API in kubelet doesn't have accelerator metrics

Currently, cadvisor_stats_provider provides AcceleratorStats but cri_stats_provider does not. As a result, when using cri_stats_provider, kubelet's Summary API does not have accelerator metrics. #96873.

16.5. Urgent Upgrade Notes

16.5.1. (No, really, you MUST read this before you upgrade)

• A bug was fixed in kubelet where exec probe timeouts were not respected. This may result in unexpected behavior since the default timeout (if not specified) is 1s which may be too small for some exec probes. Ensure that pods relying on this behavior are updated to correctly handle probe timeouts. See configure probe section of the documentation for more details.

  • This change in behavior may be unexpected for some clusters and can be disabled by turning off the ExecProbeTimeout feature gate. This gate will be locked and removed in future releases so that exec probe timeouts are always respected. (#94115, @andrewsykim) [SIG Node and Testing]

• RuntimeClass feature graduates to General Availability. Promote node.k8s.io API groups from v1beta1 to v1. v1beta1 is now deprecated and will be removed in a future release, please start using v1. (#95718, @SergeyKanzhelev) [SIG Apps, Auth, Node, Scheduling and Testing]

• API priority and fairness graduated to beta. 1.19 servers with APF turned on should not be run in a multi-server cluster with 1.20+ servers. (#96527, @adtac) [SIG API Machinery and Testing]
• For CSI drivers, kubelet no longer creates the target_path for NodePublishVolume in accordance with the CSI spec. Kubelet also no longer checks if staging and target paths are mounts or corrupted. CSI drivers need to be idempotent and do any necessary mount verification. (#88759, @andyzhangx) [SIG Storage]

• Kubeadm: http://git.k8s.io/enhancements/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint/README.md (#95382, @neolit123) [SIG Cluster Lifecycle]

  • The label applied to control-plane nodes "node-role.kubernetes.io/master" is now deprecated and will be removed in a future release after a GA deprecation period.
  • Introduce a new label "node-role.kubernetes.io/control-plane" that will be applied in parallel to "node-role.kubernetes.io/master" until the removal of the "node-role.kubernetes.io/master" label.
  • Make "kubeadm upgrade apply" add the "node-role.kubernetes.io/control-plane" label on existing nodes that only have the "node-role.kubernetes.io/master" label during upgrade.
  • Please adapt your tooling built on top of kubeadm to use the "node-role.kubernetes.io/control-plane" label.
  • The taint applied to control-plane nodes "node-role.kubernetes.io/master:NoSchedule" is now deprecated and will be removed in a future release after a GA deprecation period.
  • Apply toleration for a new, future taint "node-role.kubernetes.io/control-plane:NoSchedule" to the kubeadm CoreDNS / kube-dns managed manifests. Note that this taint is not yet applied to kubeadm control-plane nodes.
  • Please adapt your workloads to tolerate the same future taint preemptively.

• Kubeadm: improve the validation of serviceSubnet and podSubnet. ServiceSubnet has to be limited in size, due to implementation details, and the mask can not allocate more than 20 bits. PodSubnet validates against the corresponding cluster "--node-cidr-mask-size" of the kube-controller-manager, it fail if the values are not compatible. kubeadm no longer sets the node-mask automatically on IPv6 deployments, you must check that your IPv6 service subnet mask is compatible with the default node mask /64 or set it accordingly. Previously, for IPv6, if the podSubnet had a mask lower than /112, kubeadm calculated a node-mask to be multiple of eight and splitting the available bits to maximise the number used for nodes. (#95723, @aojea) [SIG Cluster Lifecycle]

• The deprecated flag --experimental-kustomize is now removed from kubeadm commands. Use --experimental-patches instead, which was introduced in 1.19. Migration information available in --help description for --experimental-patches. (#94871, @neolit123)

• Windows hyper-v container feature gate is deprecated in 1.20 and will be removed in 1.21 (#95505, @wawa0210) [SIG Node and Windows]
• The kube-apiserver ability to serve on an insecure port, deprecated since v1.10, has been removed. The insecure address flags --address and --insecure-bind-address have no effect in kube-apiserver and will be removed in v1.24. The insecure port flags --port and --insecure-port may only be set to 0 and will be removed in v1.24. (#95856, @knight42, [SIG API Machinery, Node, Testing])
• Add dual-stack Services (alpha). This is a BREAKING CHANGE to an alpha API. It changes the dual-stack API wrt Service from a single ipFamily field to 3 fields: ipFamilyPolicy (SingleStack, PreferDualStack, RequireDualStack), ipFamilies (a list of families assigned), and clusterIPs (inclusive of clusterIP). Most users do not need to set anything at all, defaulting will handle it for them. Services are single-stack unless the user asks for dual-stack. This is all gated by the "IPv6DualStack" feature gate. (#91824, @khenidak) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing]

• TokenRequest and TokenRequestProjection are now GA features. The following flags are required by the API server:

  • --service-account-issuer, should be set to a URL identifying the API server that will be stable over the cluster lifetime.
  • --service-account-key-file, set to one or more files containing one or more public keys used to verify tokens.
  • --service-account-signing-key-file, set to a file containing a private key to use to sign service account tokens. Can be the same file given to kube-controller-manager with --service-account-private-key-file. (#95896, @zshihang) [SIG API Machinery, Auth, Cluster Lifecycle]

• kubeadm: make the command "kubeadm alpha kubeconfig user" accept a "--config" flag and remove the following flags:

  • apiserver-advertise-address / apiserver-bind-port: use either localAPIEndpoint from InitConfiguration or controlPlaneEndpoint from ClusterConfiguration.
  • cluster-name: use clusterName from ClusterConfiguration
  • cert-dir: use certificatesDir from ClusterConfiguration (#94879, @knight42) [SIG Cluster Lifecycle]

• Resolves non-deterministic behavior of the garbage collection controller when ownerReferences with incorrect data are encountered. Events with a reason of OwnerRefInvalidNamespace are recorded when namespace mismatches between child and owner objects are detected. The kubectl-check-ownerreferences tool can be run prior to upgrading to locate existing objects with invalid ownerReferences.

  • A namespaced object with an ownerReference referencing a uid of a namespaced kind which does not exist in the same namespace is now consistently treated as though that owner does not exist, and the child object is deleted.
  • A cluster-scoped object with an ownerReference referencing a uid of a namespaced kind is now consistently treated as though that owner is not resolvable, and the child object is ignored by the garbage collector. (#92743, @liggitt) [SIG API Machinery, Apps and Testing]

16.6. Changes by Kind

16.6.1. Deprecation

• Docker support in the kubelet is now deprecated and will be removed in a future release. The kubelet uses a module called "dockershim" which implements CRI support for Docker and it has seen maintenance issues in the Kubernetes community. We encourage you to evaluate moving to a container runtime that is a full-fledged implementation of CRI (v1alpha1 or v1 compliant) as they become available. (#94624, @dims) [SIG Node]
• Kubeadm: deprecate self-hosting support. The experimental command "kubeadm alpha self-hosting" is now deprecated and will be removed in a future release. (#95125, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: graduate the "kubeadm alpha certs" command to a parent command "kubeadm certs". The command "kubeadm alpha certs" is deprecated and will be removed in a future release. Please migrate. (#94938, @yagonobre) [SIG Cluster Lifecycle]
• Kubeadm: remove the deprecated "kubeadm alpha kubelet config enable-dynamic" command. To continue using the feature please defer to the guide for "Dynamic Kubelet Configuration" at k8s.io. This change also removes the parent command "kubeadm alpha kubelet" as there are no more sub-commands under it for the time being. (#94668, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: remove the deprecated --kubelet-config flag for the command "kubeadm upgrade node" (#94869, @neolit123) [SIG Cluster Lifecycle]
• Kubectl: deprecate --delete-local-data (#95076, @dougsland) [SIG CLI, Cloud Provider and Scalability]
• Kubelet's deprecated endpoint metrics/resource/v1alpha1 has been removed, please adopt metrics/resource. (#94272, @RainbowMango) [SIG Instrumentation and Node]
• Removes deprecated scheduler metrics DeprecatedSchedulingDuration, DeprecatedSchedulingAlgorithmPredicateEvaluationSecondsDuration, DeprecatedSchedulingAlgorithmPriorityEvaluationSecondsDuration (#94884, @arghya88) [SIG Instrumentation and Scheduling]
• Scheduler alpha metrics binding_duration_seconds and scheduling_algorithm_preemption_evaluation_seconds are deprecated, Both of those metrics are now covered as part of framework_extension_point_duration_seconds, the former as a PostFilter the latter and a Bind plugin. The plan is to remove both in 1.21 (#95001, @arghya88) [SIG Instrumentation and Scheduling]
• Support controlplane as a valid EgressSelection type in the EgressSelectorConfiguration API. Master is deprecated and will be removed in v1.22. (#95235, @andrewsykim) [SIG API Machinery]
• The v1alpha1 PodPreset API and admission plugin has been removed with no built-in replacement. Admission webhooks can be used to modify pods on creation. (#94090, @deads2k) [SIG API Machinery, Apps, CLI, Cloud Provider, Scalability and Testing]

16.6.2. API Change

• TokenRequest and TokenRequestProjection features have been promoted to GA. This feature allows generating service account tokens that are not visible in Secret objects and are tied to the lifetime of a Pod object. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection for details on configuring and using this feature. The TokenRequest and TokenRequestProjection feature gates will be removed in v1.21.

  • kubeadm's kube-apiserver Pod manifest now includes the following flags by default "--service-account-key-file", "--service-account-signing-key-file", "--service-account-issuer". (#93258, @zshihang) [SIG API Machinery, Auth, Cluster Lifecycle, Storage and Testing]

• A new nofuzz go build tag now disables gofuzz support. Release binaries enable this. (#92491, @BenTheElder) [SIG API Machinery]

• Add WindowsContainerResources and Annotations to CRI-API UpdateContainerResourcesRequest (#95741, @katiewasnothere) [SIG Node]
• Add a serving and terminating condition to the EndpointSlice API. serving tracks the readiness of endpoints regardless of their terminating state. This is distinct from ready since ready is only true when pods are not terminating. terminating is true when an endpoint is terminating. For pods this is any endpoint with a deletion timestamp. (#92968, @andrewsykim) [SIG Apps and Network]
• Add dual-stack Services (alpha). This is a BREAKING CHANGE to an alpha API. It changes the dual-stack API wrt Service from a single ipFamily field to 3 fields: ipFamilyPolicy (SingleStack, PreferDualStack, RequireDualStack), ipFamilies (a list of families assigned), and clusterIPs (inclusive of clusterIP). Most users do not need to set anything at all, defaulting will handle it for them. Services are single-stack unless the user asks for dual-stack. This is all gated by the "IPv6DualStack" feature gate. (#91824, @khenidak) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing]
• Add support for hugepages to downward API (#86102, @derekwaynecarr) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing]
• Adds kubelet alpha feature, GracefulNodeShutdown which makes kubelet aware of node system shutdowns and result in graceful termination of pods during a system shutdown. (#96129, @bobbypage) [SIG Node]
• AppProtocol is now GA for Endpoints and Services. The ServiceAppProtocol feature gate will be deprecated in 1.21. (#96327, @robscott) [SIG Apps and Network]
• Automatic allocation of NodePorts for services with type LoadBalancer can now be disabled by setting the (new) parameter Service.spec.allocateLoadBalancerNodePorts=false. The default is to allocate NodePorts for services with type LoadBalancer which is the existing behavior. (#92744, @uablrek) [SIG Apps and Network]
• Certain fields on Service objects will be automatically cleared when changing the service's type to a mode that does not need those fields. For example, changing from type=LoadBalancer to type=ClusterIP will clear the NodePort assignments, rather than forcing the user to clear them. (#95196, @thockin) [SIG API Machinery, Apps, Network and Testing]
• Document that ServiceTopology feature is required to use service.spec.topologyKeys. (#96528, @andrewsykim) [SIG Apps]

• EndpointSlice has a new NodeName field guarded by the EndpointSliceNodeName feature gate.

  • EndpointSlice topology field will be deprecated in an upcoming release.
  • EndpointSlice "IP" address type is formally removed after being deprecated in Kubernetes 1.17.
  • The discovery.k8s.io/v1alpha1 API is deprecated and will be removed in Kubernetes 1.21. (#96440, @robscott) [SIG API Machinery, Apps and Network]

• External facing API podresources is now available under k8s.io/kubelet/pkg/apis/ (#92632, @RenaudWasTaken) [SIG Node and Testing]

• Fewer candidates are enumerated for preemption to improve performance in large clusters. (#94814, @adtac)
• Fix conversions for custom metrics. (#94481, @wojtek-t) [SIG API Machinery and Instrumentation]
• GPU metrics provided by kubelet are now disabled by default. (#95184, @RenaudWasTaken)
• If BoundServiceAccountTokenVolume is enabled, cluster admins can use metric serviceaccount_stale_tokens_total to monitor workloads that are depending on the extended tokens. If there are no such workloads, turn off extended tokens by starting kube-apiserver with flag --service-account-extend-token-expiration=false (#96273, @zshihang) [SIG API Machinery and Auth]
• Introduce alpha support for exec-based container registry credential provider plugins in the kubelet. (#94196, @andrewsykim) [SIG Node and Release]
• Introduces a metric source for HPAs which allows scaling based on container resource usage. (#90691, @arjunrn) [SIG API Machinery, Apps, Autoscaling and CLI]

• Kube-apiserver now deletes expired kube-apiserver Lease objects:

  • The feature is under feature gate APIServerIdentity.
  • A flag is added to kube-apiserver: identity-lease-garbage-collection-check-period-seconds (#95895, @roycaihw) [SIG API Machinery, Apps, Auth and Testing]

• Kube-controller-manager: volume plugins can be restricted from contacting local and loopback addresses by setting --volume-host-allow-local-loopback=false, or from contacting specific CIDR ranges by setting --volume-host-cidr-denylist (for example, --volume-host-cidr-denylist=127.0.0.1/28,feed::/16) (#91785, @mattcary) [SIG API Machinery, Apps, Auth, CLI, Network, Node, Storage and Testing]

• Migrate scheduler, controller-manager and cloud-controller-manager to use LeaseLock (#94603, @wojtek-t) [SIG API Machinery, Apps, Cloud Provider and Scheduling]
• Modify DNS-1123 error messages to indicate that RFC 1123 is not followed exactly (#94182, @mattfenwick) [SIG API Machinery, Apps, Auth, Network and Node]
• Move configurable fsgroup change policy for pods to beta (#96376, @gnufied) [SIG Apps and Storage]
• New flag is introduced, i.e. --topology-manager-scope=container|pod. The default value is the "container" scope. (#92967, @cezaryzukowski) [SIG Instrumentation, Node and Testing]
• New parameter defaultingType for PodTopologySpread plugin allows to use k8s defined or user provided default constraints (#95048, @alculquicondor) [SIG Scheduling]
• NodeAffinity plugin can be configured with AddedAffinity. (#96202, @alculquicondor) [SIG Node, Scheduling and Testing]
• Promote RuntimeClass feature to GA. Promote node.k8s.io API groups from v1beta1 to v1. (#95718, @SergeyKanzhelev) [SIG Apps, Auth, Node, Scheduling and Testing]
• Reminder: The labels "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" are deprecated in favor of "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" respectively. All users of the "failure-domain.beta..." labels should switch to the "topology..." equivalents. (#96033, @thockin) [SIG API Machinery, Apps, CLI, Cloud Provider, Network, Node, Scheduling, Storage and Testing]
• Server Side Apply now treats LabelSelector fields as atomic (meaning the entire selector is managed by a single writer and updated together), since they contain interrelated and inseparable fields that do not merge in intuitive ways. (#93901, @jpbetz) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Storage and Testing]
• Services will now have a clusterIPs field to go with clusterIP. clusterIPs[0] is a synonym for clusterIP and will be synchronized on create and update operations. (#95894, @thockin) [SIG Network]
• The ServiceAccountIssuerDiscovery feature gate is now Beta and enabled by default. (#91921, @mtaufen) [SIG Auth]
• The status of v1beta1 CRDs without "preserveUnknownFields:false" now shows a violation, "spec.preserveUnknownFields: Invalid value: true: must be false". (#93078, @vareti)
• The usage of mixed protocol values in the same LoadBalancer Service is possible if the new feature gate MixedProtocolLBService is enabled. The feature gate is disabled by default. The user has to enable it for the API Server. (#94028, @janosi) [SIG API Machinery and Apps]
• This PR will introduce a feature gate CSIServiceAccountToken with two additional fields in CSIDriverSpec. (#93130, @zshihang) [SIG API Machinery, Apps, Auth, CLI, Network, Node, Storage and Testing]
• Users can try the CronJob controller v2 using the feature gate. This will be the default controller in future releases. (#93370, @alaypatel07) [SIG API Machinery, Apps, Auth and Testing]
• VolumeSnapshotDataSource moves to GA in 1.20 release (#95282, @xing-yang) [SIG Apps]
• WinOverlay feature graduated to beta (#94807, @ksubrmnn) [SIG Windows]

16.6.3. Feature

• A new metric apiserver_request_filter_duration_seconds has been introduced that measures request filter latency in seconds. (#95207, @tkashem) [SIG API Machinery and Instrumentation]
• A new set of alpha metrics are reported by the Kubernetes scheduler under the /metrics/resources endpoint that allow administrators to easily see the resource consumption (requests and limits for all resources on the pods) and compare it to actual pod usage or node capacity. (#94866, @smarterclayton) [SIG API Machinery, Instrumentation, Node and Scheduling]
• Add --experimental-logging-sanitization flag enabling runtime protection from leaking sensitive data in logs (#96370, @serathius) [SIG API Machinery, Cluster Lifecycle and Instrumentation]
• Add a StorageVersionAPI feature gate that makes API server update storageversions before serving certain write requests. This feature allows the storage migrator to manage storage migration for built-in resources. Enabling internal.apiserver.k8s.io/v1alpha1 API and APIServerIdentity feature gate are required to use this feature. (#93873, @roycaihw) [SIG API Machinery, Auth and Testing]
• Add a metric for time taken to perform recursive permission change (#95866, @JornShen) [SIG Instrumentation and Storage]
• Add a new vSphere metric: cloudprovider_vsphere_vcenter_versions. Its content shows vCenter hostnames with the associated server version. (#94526, @Danil-Grigorev) [SIG Cloud Provider and Instrumentation]
• Add a new flag to set priority for the kubelet on Windows nodes so that workloads cannot overwhelm the node thereby disrupting kubelet process. (#96051, @ravisantoshgudimetla) [SIG Node and Windows]
• Add feature to size memory backed volumes (#94444, @derekwaynecarr) [SIG Storage and Testing]
• Add foreground cascading deletion to kubectl with the new kubectl delete foreground|background|orphan option. (#93384, @zhouya0)
• Add metrics for azure service operations (route and loadbalancer). (#94124, @nilo19) [SIG Cloud Provider and Instrumentation]
• Add network rule support in Azure account creation. (#94239, @andyzhangx)
• Add node_authorizer_actions_duration_seconds metric that can be used to estimate load to node authorizer. (#92466, @mborsz) [SIG API Machinery, Auth and Instrumentation]
• Add pod_ based CPU and memory metrics to Kubelet's /metrics/resource endpoint (#95839, @egernst) [SIG Instrumentation, Node and Testing]
• Added get-users and delete-user to the kubectl config subcommand (#89840, @eddiezane) [SIG CLI]
• Added counter metric "apiserver_request_self" to count API server self-requests with labels for verb, resource, and subresource. (#94288, @LogicalShark) [SIG API Machinery, Auth, Instrumentation and Scheduling]
• Added new k8s.io/component-helpers repository providing shared helper code for (core) components. (#92507, @ingvagabund) [SIG Apps, Node, Release and Scheduling]
• Adds create ingress command to kubectl (#78153, @amimof) [SIG CLI and Network]
• Adds a headless service on node-local-cache addon. (#88412, @stafot) [SIG Cloud Provider and Network]
• Allow cross-compilation of kubernetes on different platforms. (#94403, @bnrjee) [SIG Release]
• Azure: Support multiple services sharing one IP address (#94991, @nilo19) [SIG Cloud Provider]
• CRDs: For structural schemas, non-nullable null map fields will now be dropped and defaulted if a default is available. null items in the list will continue being preserved, and fail validation if not nullable. (#95423, @apelisse) [SIG API Machinery]
• Changed: default "Accept: /" header added to HTTP probes. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#http-probes (https://github.com/kubernetes/website/pull/24756) (#95641, @fonsecas72) [SIG Network and Node]
• Client-go credential plugins can now be passed in the current cluster information via the KUBERNETES_EXEC_INFO environment variable. (#95489, @ankeesler) [SIG API Machinery and Auth]
• Command to start network proxy changes from 'KUBE_ENABLE_EGRESS_VIA_KONNECTIVITY_SERVICE ./cluster/kube-up.sh' to 'KUBE_ENABLE_KONNECTIVITY_SERVICE=true ./hack/kube-up.sh' (#92669, @Jefftree) [SIG Cloud Provider]
• Configure AWS LoadBalancer health check protocol via service annotations. (#94546, @kishorj)
• DefaultPodTopologySpread graduated to Beta. The feature gate is enabled by default. (#95631, @alculquicondor) [SIG Scheduling and Testing]
• E2e test for PodFsGroupChangePolicy (#96247, @saikat-royc) [SIG Storage and Testing]
• Ephemeral containers now apply the same API defaults as initContainers and containers (#94896, @wawa0210) [SIG Apps and CLI]
• Graduate the Pod Resources API to G.A Introduces the pod_resources_endpoint_requests_total metric which tracks the total number of requests to the pod resources API (#92165, @RenaudWasTaken) [SIG Instrumentation, Node and Testing]

• In dual-stack bare-metal clusters, you can now pass dual-stack IPs to kubelet --node-ip. eg: kubelet --node-ip 10.1.0.5,fd01::0005. This is not yet supported for non-bare-metal clusters.

  In dual-stack clusters where nodes have dual-stack addresses, hostNetwork pods will now get dual-stack PodIPs. (#95239, @danwinship) [SIG Network and Node]

• Introduce api-extensions category which will return: mutating admission configs, validating admission configs, CRDs and APIServices when used in kubectl get, for example. (#95603, @soltysh) [SIG API Machinery]

• Introduces a new GCE specific cluster creation variable KUBE_PROXY_DISABLE. When set to true, this will skip over the creation of kube-proxy (whether the daemonset or static pod). This can be used to control the lifecycle of kube-proxy separately from the lifecycle of the nodes. (#91977, @varunmar) [SIG Cloud Provider]

• Kube-apiserver now maintains a Lease object to identify itself:

  • The feature is under feature gate APIServerIdentity.
  • Two flags are added to kube-apiserver: identity-lease-duration-seconds, identity-lease-renew-interval-seconds (#95533, @roycaihw) [SIG API Machinery]

• Kube-apiserver: The timeout used when making health check calls to etcd can now be configured with --etcd-healthcheck-timeout. The default timeout is 2 seconds, matching the previous behavior. (#93244, @Sh4d1) [SIG API Machinery]

• Kube-apiserver: added support for compressing rotated audit log files with --audit-log-compress (#94066, @lojies) [SIG API Machinery and Auth]
• Kubeadm now prints warnings instead of throwing errors if the current system time is outside of the NotBefore and NotAfter bounds of a loaded certificate. (#94504, @neolit123)
• Kubeadm: Add a preflight check that the control-plane node has at least 1700MB of RAM (#93275, @xlgao-zju) [SIG Cluster Lifecycle]
• Kubeadm: add the "--cluster-name" flag to the "kubeadm alpha kubeconfig user" to allow configuring the cluster name in the generated kubeconfig file (#93992, @prabhu43) [SIG Cluster Lifecycle]
• Kubeadm: add the "--kubeconfig" flag to the "kubeadm init phase upload-certs" command to allow users to pass a custom location for a kubeconfig file. (#94765, @zhanw15) [SIG Cluster Lifecycle]
• Kubeadm: make etcd pod request 100m CPU, 100Mi memory and 100Mi ephemeral_storage by default (#94479, @knight42) [SIG Cluster Lifecycle]

• Kubeadm: make the command "kubeadm alpha kubeconfig user" accept a "--config" flag and remove the following flags:

  • apiserver-advertise-address / apiserver-bind-port: use either localAPIEndpoint from InitConfiguration or controlPlaneEndpoint from ClusterConfiguration.
  • cluster-name: use clusterName from ClusterConfiguration
  • cert-dir: use certificatesDir from ClusterConfiguration (#94879, @knight42) [SIG Cluster Lifecycle]

• Kubectl create now supports creating ingress objects. (#94327, @rikatz) [SIG CLI and Network]

• Kubectl rollout history sts/sts-name --revision=some-revision will start showing the detailed view of the sts on that specified revision (#86506, @dineshba) [SIG CLI]
• Kubectl: Previously users cannot provide arguments to a external diff tool via KUBECTL_EXTERNAL_DIFF env. This release now allow users to specify args to KUBECTL_EXTERNAL_DIFF env. (#95292, @dougsland) [SIG CLI]
• Kubemark now supports both real and hollow nodes in a single cluster. (#93201, @ellistarn) [SIG Scalability]
• Kubernetes E2E test image manifest lists now contain Windows images. (#77398, @claudiubelu) [SIG Testing and Windows]

• Kubernetes is now built using go1.15.2

  • build: Update to k/repo-infra@v0.1.1 (supports go1.15.2)
  • build: Use go-runner:buster-v2.0.1 (built using go1.15.1)
  • bazel: Replace --features with Starlark build settings flag

  • hack/lib/util.sh: some bash cleanups

    • switched one spot to use kube::logging
    • make kube::util::find-binary return an error when it doesn't find anything so that hack scripts fail fast instead of with "binary not found errors".
    • this required deleting some genfeddoc stuff. the binary no longer exists in k/k repo since we removed federation/, and I don't see it in https://github.com/kubernetes-sigs/kubefed/ either. I'm assuming that it's gone for good now.

  • bazel: output go_binary rule directly from go_binary_conditional_pure

    From: @mikedanese: Instead of aliasing. Aliases are annoying in a number of ways. This is specifically bugging me now because they make the action graph harder to analyze programmatically. By using aliases here, we would need to handle potentially aliased go_binary targets and dereference to the effective target.

    The comment references an issue with pure = select(...) which appears to be resolved considering this now builds.

  • make kube::util::find-binary not dependent on bazel-out/ structure

    Implement an aspect that outputs go_build_mode metadata for go binaries, and use that during binary selection. (#94449, @justaugustus) [SIG Architecture, CLI, Cluster Lifecycle, Node, Release and Testing]

• Kubernetes is now built using go1.15.5

  • build: Update to k/repo-infra@v0.1.2 (supports go1.15.5) (#95776, @justaugustus) [SIG Cloud Provider, Instrumentation, Release and Testing]

• New default scheduling plugins order reduces scheduling and preemption latency when taints and node affinity are used (#95539, @soulxu) [SIG Scheduling]

• Only update Azure data disks when attach/detach (#94265, @andyzhangx) [SIG Cloud Provider]
• Promote SupportNodePidsLimit to GA to provide node-to-pod PID isolation. Promote SupportPodPidsLimit to GA to provide the ability to limit PIDs per pod. (#94140, @derekwaynecarr)
• SCTP support in API objects (Pod, Service, NetworkPolicy) is now GA. Note that this has no effect on whether SCTP is enabled on nodes at the kernel level, and note that some cloud platforms and network plugins do not support SCTP traffic. (#95566, @danwinship) [SIG Apps and Network]
• Scheduler now ignores Pod update events if the resourceVersion of old and new Pods are identical. (#96071, @Huang-Wei) [SIG Scheduling]
• Scheduling Framework: expose Run[Pre]ScorePlugins functions to PreemptionHandle which can be used in PostFilter extension point. (#93534, @everpeace) [SIG Scheduling and Testing]
• SelectorSpreadPriority maps to PodTopologySpread plugin when DefaultPodTopologySpread feature is enabled (#95448, @alculquicondor) [SIG Scheduling]
• Send GCE node startup scripts' logs to console and journal. (#95311, @karan)
• SetHostnameAsFQDN has been graduated to Beta and therefore it is enabled by default. (#95267, @javidiaz) [SIG Node]
• Support [service.beta.kubernetes.io/azure-pip-ip-tags] annotations to allow customers to specify ip-tags to influence public-ip creation in Azure [Tag1=Value1, Tag2=Value2, etc.] (#94114, @MarcPow) [SIG Cloud Provider]
• Support custom tags for cloud provider managed resources (#96450, @nilo19) [SIG Cloud Provider]
• Support customize load balancer health probe protocol and request path (#96338, @nilo19) [SIG Cloud Provider]
• Support for Windows container images (OS Versions: 1809, 1903, 1909, 2004) was added to the pause:3.4 image. (#91452, @claudiubelu) [SIG Node, Release and Windows]
• Support multiple standard load balancers in one cluster (#96111, @nilo19) [SIG Cloud Provider]
• The beta RootCAConfigMap feature gate is enabled by default and causes kube-controller-manager to publish a "kube-root-ca.crt" ConfigMap to every namespace. This ConfigMap contains a CA bundle used for verifying connections to the kube-apiserver. (#96197, @zshihang) [SIG API Machinery, Apps, Auth and Testing]
• The kubelet_runtime_operations_duration_seconds metric buckets were set to 0.005 0.0125 0.03125 0.078125 0.1953125 0.48828125 1.220703125 3.0517578125 7.62939453125 19.073486328125 47.6837158203125 119.20928955078125 298.0232238769531 and 745.0580596923828 seconds (#96054, @alvaroaleman) [SIG Instrumentation and Node]
• There is a new pv_collector_total_pv_count metric that counts persistent volumes by the volume plugin name and volume mode. (#95719, @tsmetana) [SIG Apps, Instrumentation, Storage and Testing]
• Volume snapshot e2e test to validate PVC and VolumeSnapshotContent finalizer (#95863, @RaunakShah) [SIG Cloud Provider, Storage and Testing]
• Warns user when executing kubectl apply/diff to a resource currently being deleted. (#95544, @SaiHarshaK) [SIG CLI]
• kubectl alpha debug has graduated to beta and is now kubectl debug. (#96138, @verb) [SIG CLI and Testing]
• kubectl debug gains support for changing container images when copying a pod for debugging, similar to how kubectl set image works. See kubectl help debug for more information. (#96058, @verb) [SIG CLI]

16.6.4. Documentation

• Fake dynamic client: document that List does not preserve TypeMeta in UnstructuredList (#95117, @andrewsykim) [SIG API Machinery]
• Kubelet: remove alpha warnings for CNI flags. (#94508, @andrewsykim) [SIG Network and Node]
• Updates docs and guidance on cloud provider InstancesV2 and Zones interface for external cloud providers:
  • removes experimental warning for InstancesV2
  • document that implementation of InstancesV2 will disable calls to Zones
  • deprecate Zones in favor of InstancesV2 (#96397, @andrewsykim) [SIG Cloud Provider]