LWKD: Week Ending October 29, 2023
Developer News
As initially announced a few weeks ago, SIG K8s-infra is happy to finalize the changing of the guard! Aaron and Tim Hockin will be handing the reins over to Ben, Arnaud, and Mahamed.
Kubernetes Contributor Summit: You still have 2 days to register for the Summit in Chicago. We are also looking for volunteers to help staff it. While you are welcome to attend only the Social, you must still register to do so. And bring your best Kubernetes-themed poetry to the Social with you! Finally, don't forget to join your favorite SIG(s) for the Meet the Kubernetes Contributor Community event on Thursday.
Label your tests! We can now choose which E2E tests to run through Ginko v2 labels, but only if tests are labelled correctly.
There are several ingress-nginx security vulnerabilities, see below.
Release Schedule
Next Deadline: CODE FREEZE, November 1st
The Code Freeze milestone for Kubernetes 1.29 release cycle is here. Is everything checked in? This is followed by the regular release countdown: open docs PRs by Thursday, publish deprecations blog on Thursday, test freeze and finalize docs next week. As always, once we’re in Code Freeze, please respond to test failure issues with alacrity. You can reach out to us on Slack in the #sig-release channel if you have any questions.
Featured PRs
#116065: feature(scheduler): implement matchLabelKeys/mismatchLabelKeys in PodAffinity and PodAntiAffinity
The affinity and anti-affinity settings are main workhorses of controlling Pod distribution over a cluster. They allow configuring co-location and avoidance with simple rules rather than more elaborate manual resource allocation either directly or with labeled worker pools. This PR adds a simple shorthand for two common scenarios: label selectors that use the label value from the pod under consideration without restating them. This reduces the need for external webhooks or templating tools for cases like "affinity match Pods from this deployment's current rollout":
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- topologyKey: topology.kubernetes.io/zone
matchLabelKeys: [pod-template-hash]
This will be interpreted to:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- topologyKey: topology.kubernetes.io/zone
labelSelector:
matchExpressions:
- key: pod-template-hash
operator: In
values: [84cd68b55b] # Value based on the Pod's labels
The mismatchLabelKeys field similarly converts to a NotIn operator, allowing things like single-tenant nodes in a multi-tenant infrastructure.
#121104: [KEP-3751] introduce the VolumeAttributesClass API
While size is generally the most important value for most persistent volumes, many providers offer other tunable parameters such IO Operations/Second (IOPS) and overall available bandwidth. Befitting its importance, we've long supported dynamic adjustment of the size of a volume (assuming the underlying provider also allows it), this PR adds similar capabilities for these other values as well. Some providers have supported setting them as custom parameters in the underlying StorageClass object, however there's no handling in CSI or Kubernetes for runtime adjustments to a StorageClass resulting in an update of all the related volumes. Enter VolumeAttributesClass, a new API type similar to StorageClass in that it defines a named set of provider-specific values (e.g. iops, throughput) which can be applied to a PersistentVolumeClaim. But unlike a StorageClass, the volumeAttributesClassName on a PVC can be changed after creation. So like with dynamic volume resizing, you could create a new VolumeAttributesClass with higher (or lower) values than the previous one and update your PVCs to the new class, triggering an update call in the CSI driver to work its magic. This provides a structured way to evolve storage requirements over time without sacrificing the benefits of the CSI abstraction.
KEP of the Week
KEP 3983 - Add support for a drop-in kubelet configuration directory
This KEP adds support for drop-in configuration directory to the kubelet. The proposed way to configure the configuration directory is via a --config-dir flag. The flag will be empty by default and if nothing is specified, drop-in support will be disabled. If the drop-in directory is enabled, it will override the configuration for the kubelet located at /etc/kubernetes/kubelet.conf. Users would be able to configure individual kubelet config snippets in files, which would be formatted in the same way as the kubelet.conf file. The kubelet would process these files in alphanumeric order.
This KEP was in alpha stage in the v1.28 release, and is in track to graduate to beta in the upcoming v1.29 release.
Other Merges
- Admins can configure an age threshold to automatically garbage-collect unused images even without disk pressure
- In-place Pod resizing works on Windows
- Use CEL expressions for AuthorizationConfiguration webhooks
- KMSv2 supports tracing
- Switch to force Privileged containers to run as initContainers
- Add a bunch more info to bound ServiceAccount tokens
- Controllers have standardized metadata and with that, improved
--helpoutput - PodSecurityStandards can behave differently when working with User Namespaces
- Replace the PersistentVolumeLabel controller with a webhook
- Sidecars restart in the right order, and are terminated after other containers
- Only EnqueueExtensions from certain kinds of plugins
- Decoding time should be included in Etcd timeouts
- New Metrics: Pod startup latency, CRDValidationRatcheting, Job creation count
- AbortPlugin is the new FailedPlugin
- Skip unnecessary CRD cost limit checks
- CEL should ratchet errors from CEL expressions, handle schemas with no Properties
- Make sure that StatefulSets restart after an eviction
- Fix timing on Jobs getting rescheduled
- kubeadm: super-powered super-admin.conf file
- ResourceClaimParameter validation failures will throw an error
- Finish converting kube-scheduler to contextual logging
- Report container image filesystem to CRI
- Client-side apply will use OpenAPIv3
- Enlarge Pod start duration histogram metric
- Don't let other containers steal resources belonging to sidecar containers, especially CPU
- Kube-proxy supports nf_conntrack_tcp_be_liberal
- TaintManager and NodeLifecycleController go their separate ways
Testing Improvements: CEL cost estimation, remove duplicate job tests
Promotions
- Priority and Fairness to GA, with a v1 API
- Sidecar Containers to Beta
- PodReplacementPolicy to Beta
- KMSv2 to GA
- CRD Validation Rule to GA
- DevicePluginCDIDevices to Beta
- ReadWriteOncePod to GA
- ServiceAccount token cleanup to Beta
- PersistentVolumeLastPhaseTransitionTime to Beta
Deprecated
- Deprecate status.nodeinfo.kubeproxyversion, which has never been accurate
- GA'd features removing featuregates: CSIMigrationVSphere, OpenAPIv3
Version Updates
Subprojects and Dependency Updates
- ingress-nginx - CVE-2022-4886, CVE-2023-5043 and CVE-2023-5044 are high risk vulnerabilities identified last week in ingress-nginx. CVE-2023-5043 and CVE-2023-5044 are vulnerabilities where the
nginx.ingress.kubernetes.io/configuration-snippetandnginx.ingress.kubernetes.io/permanent-redirectannotations can be used to inject arbitrary commands and obtain the credentials of the ingress-nginx controller. This can be mitigated by setting the--enable-annotation-validationflag to enforce restrictions on the contents of ingress-nginx annotation fields.