LWKD: Week Ending November 13, 2022
Developer News
Two new security fixes last week: CVE-2022-3162 allowed custom resources in the same API group eavesdrop on each other; CVE-2022-3294 allowed users to proxy via the API server to addresses they shouldn’t have had access to. Both issues are fixed in the most recent patch releases and upgrading is recommended.
Release Schedule
Next Deadline: Test Freeze / Docs Ready for Review / Release Blog Ready, November 15th
We are in Code Freeze now, with the countdown of deadlines to final release on December 6th. This week is the trifecta of: have your feature docs ready for review, get the release blog ready for review, and freeze all test work until the final release (except bugfixes).
Patch releases for 1.25.4, 1.24.8, 1.23.14, and 1.22.16 came out last week, including fixes for both of the CVEs. This will be the last patch release for 1.22, so upgrade your cluster now.
Note that the next set of patch releases have been rescheduled to December 7th, with cherry-picks due on the 2nd. SIG-Release has proposed that we stop making release candidates of patch releases given the effort and image cost, so comment there if you have opinions on this change.
Featured PRs
#113599: Aggregated discovery client
The discovery APIs let clients query what APIs exist on the API server. This is used for everything from kubectl shell autocomplete to version negotiation. When Kubernetes was young and there were only a few dozen API types, querying each separately worked well enough, but the core APIs have grown and custom APIs (operators) have become widespread. Some single operators like Crossplane can register more API types than all of Kubernetes core. Suffice to say, this created scaling and performance challenges. We’ve used client-side caching to help mitigate things but this improved API (and a related server-side change), we have a more complete solution. Now kube-apiserver can cache discovery metadata itself and clients can request it all in one go. For the initial testing, this is being added to clients in a relatively drop-in fashion to mirror the existing caching client but in the future the plan is to remove the need for client-side caching entirely when aggregated discovery is in use. This should improve both client and server performance and simplify the code, an impressive win for the API team.
If you work on a non-Go client library, definitely check out the new data you’ve got available, it will likely speed up your tools as well.
#109189: Update redacting functionality to redact all sensitive info in config when printing with view
kubectl config view has long supported a redaction mechanism for a few hardwired config fields. This PR extends things to use our existing datapolicy struct tag to handle things more holistically. This especially helps with user plugin credentials, such as OIDC access tokens. As before, the --raw flag will disabled redaction. If you are using kubectl config view in scripts or automation, you should check if this change will impact you (or if you are currently leaking sensitive information).
#113629: Promote APIServerIdentity to Beta
The leases API was originally added to streamline leader elections but it has found many uses beyond that. A new one added recently, and now promoted to Beta, is that each kube-apiserver creates a lease and renews it as long as it is running. This allows real-time enumeration of what API servers are running. The likely first use for this new data will be automating object storage version updates. Currently storage-version-migrator requires external coordination to ensure it runs to completion between each step when doing a marathon upgrade. Plenty of other things can likely benefit in similar ways though, debugging tools, cluster configuration helpers, etc. A simple and fun extension of an existing API!
Other Merges
- Dynamic Resource Allocation, for adding and removing node-level resources at runtime, is complete enough for alpha
- StatefulSets support “slices”, allowing sets to start at a non-0 number, usually for sharding a database or storage resource
- Priority & Fairness “borrowing” is a complete feature, with a ton of metrics
- The
unhealthyPodEvictionPolicyallows non-ready pods to be evicted even if it exceeds PDB - OpenStack Cinder storage users, get ready to migrate to CSI
- CEL Admission Policy is renamed to ValidatingAdmissionPolicy; metrics change
- Add an API for cross-namespace Volumes, feature to come
- Prevent a race condition crash between podFailurePolicy and deleted Pods, which does mean that the policy might not execute
- Make sure that konnectivity connections don’t hang
- NUMA distances work even if nodes don’t start at 0
- Revert scheduler pre-filter skipper
- Credential provider metrics no longer named
kubelet_kubelet - Keep the startReflector from generating memory-eating nil pointers in the kubelet
- Kubelet keeps fast-updating until the node is ready
Test Cleanup: add disruption and jobs test, StorageVersion unit tests
Promotions
- Retriable pod failures for jobs to beta
kubectl eventsto beta (no morekubectl alpha events)- API Server Tracing was beta, but got reverted to alpha due to breakage