March 14, 2023

LWKD: Week Ending March 12, 2023

Developer News

Lots of important and urgent news this week.

In one week, deprecated registry k8s.gcr.io will be automatically redirected to registry.k8s.io. This may break some existing Kubernetes setups, especially those running older versions in secure or gapped environments. Make sure that registry.k8s.io is on your allowlist for image pulls. This measure is only temporary; long-term, you still need to update your installations and components to use registry.k8s.io.

We've had to stop releasing Kubernetes for 32-bit Linux/ARM because Golang isn't consistently working on it. If 32-bit ARM as a platform matters to you, consider getting involved in the Go project to stabilize support.

The project is making a clear separation between Chair and Tech Lead roles for all SIGs. If you haven't had a separate TL role for your SIG yet, you'll need to figure out who is doing what soon.

Patrick Ohly and others have finished adding support for linter checks in Kubernetes pre-submit testing. This will get turned on for 1.28 development.

Registration for the EU Contributor Summit is open.

Release Schedule

Next Deadline: CODE FREEZE, March 14th

It's Deadline Week for 1.27! Pencils down, everyone. Tomorrow is both Code Freeze and the deadline for major themes submissions. Docs placeholder PRs are due on Thursday. Test freeze is next week. We currently have 78 listed Enhancements in 1.27; let's finish these up and get 1.27 out the door. CI Signal is, fortunately, green after dropping 32-bit ARM.

Patch releases for all supported versions are due out this Wednesday.

Featured PRs

#116293: Enable plugin resolution as subcommand for selected builtin commands

Kubectl plugins have allowed for a huge variety of workflow tools and other CLI improvements, all using a very simple system of converting kubectl foo to exec("kubectl-foo"). This has been a huge boon for the community but what if we took it ever farther? A limitation of the existing system is that plugins must use a top-level command that is currently unused. SIG-CLI has now added the next step, allowing arbitrary levels of commands to be provided by plugins. This follows the same pattern as before, kubectl create mything will try to run a command kubectl-create-mything. For now this is only supported for kubectl create as well as being behind a feature flag through the $KUBECTL_ENABLE_CMD_SHADOW environment variable. There are still some open questions about how to steer this towards GA, mostly around how to handle the case of a new built-in command taking priority over what used to be a plugin, however the SIG is hard at work hammering out those details and how to expand the feature for more of Kubectl.

#116161: Mutable pod scheduling directives

The new PodSchedulingReadiness feature, added in 1.26 as an alpha feature and graduating to beta in 1.27, allows for blocking a Pod from getting scheduled until some conditions are met. This works similarly to the finalizers system, with the conditions being entirely abstracted behind an array of tags, it being up to the controllers which manage them to know the meaning. Overall this allows for more flexible scheduling as a gate controller can do things like initialize hardware or check more complex system health conditions before scheduling is allowed to proceed. To better fulfill that mission, a limited mutation capability has been added for Pods in this in-between state where they have been accepted by the API but scheduling is disallowed. While in that limbo, you can now add extra terms to the node selector or affinity expressions. This only allows pure addition, existing values can't be removed or modified. But this could be used for things like a spot-instance-aware scheduling helper which can adjust bids based on Pod-level information, and then ensures things end up on the correct machines. If you've got any kind of complex scheduling needs, make sure to check out this new feature.

Other Merges

Lots of merges this week, it's Code Freeze Time!

• kubectl whoami is promoted from the alpha subcommand, it helps diagnose authentication issues
• Permit encrypting all resources with wildcard matching
• Gradually drain connections for terminating pods when downscaling nodes using Cluster-Autoscaler
• Allow API streaming from the reflector
• Provide a clear() function to wipe any generic set
• Implement contextual logging for: cluster role aggregation, StatefulSet, deployments, garbage collector, bootstrap, namespace controller
• Add messageExpression to ValidationRule for CEL, for custom error messages
• Detect and monitor runtime errors that prevent force-deleting pods
• ContainerStatus.AllocatedResources is the new ContainerStatus.ResourcesAllocated
• Make sure watch events are recognized if they come in during initialization
• Look up metrics less often in the Scheduler
• PodNominator and SchedulingQueue share a lock
• Add fallback for query param verifier
• The CSIMigrationvSphere feature gate is back, because of feature parity issues
• New Metrics: plugin_execution_duration_seconds, aggregated discovery
• DRA Manager checkpointing
• Run calculatePodResourceRequest during PreScore
• --output plaintext-openapiv2 allows seeing old OpenAPIv2 explain text
• QoS memoryThrottlingFactor defaults to 0.9 and reworked the formulas to support cGroupsv2 better
• kube-aggregator is statically linked
• Sync and re-configure LBs even less often

Reversions: * Device recovery flow improvements

Testing Improvements: sidecar lifecycle checks, SElinux, init containers, more init containers, dedup after-roundtrip fixtures, multi-protocol ports, terminating pod connections, parallelize node ipam test, implement AllAlpha and AllBeta, debug profiles, StatefulSet Ordinal, node expand volume, nodestatus, util.go, kuberuntime, Windows kubelet, winstats, kubelet types

Promotions

• SelfSubjectReview to beta
• OpenAPIv3 to GA
• Kubelet Topology Manager to GA
• API Server Tracing to beta
• Aggregated Discovery to beta
• Evented PLEG to beta
• StatefulSet Ordinal to beta

Deprecated

• Deprecated v1beta1 API of CSIStorageCapacity is gone
• AWS kubelet credential provider removed; use the cloud provider instead
• Azure disk in-tree storage removed; use CSI driver instead
• Deprecated kubelet flag --master-service-namespace is removed
• Deprecate SecurityContextDeny plugin with a warning
• Remove --enable-taint-manager from the controller manager
• Remove redundant APIserver metrics
• Remove deprecated --container-runtime kubelet flag

Version Updates

• Golang to 1.20.2 in 1.27
• Golang to 1.19.7 in 1.26, 1.25, 1.24, and 1.23
• Update the distroless-iptables image