March 21, 2023

LWKD: Week Ending March 19, 2023

Developer News

Deprecated registry k8s.gcr.io has been redirected to registry.k8s.io. This may cause issues with some builds and/or deployments, so check yours for image pull errors. Please update your infrastructure and code to use the new registry, as much as you can.

The resource.k8s.io/v1alpha1 API types for dynamic resource allocation (ResourceClaim, ResourceClaimTemplate, ResourceClass, PodScheduling) have been removed and replaced with a resource.k8s.io/v1alpha2. As this is still an alpha feature, no upgrade path is provided and any existing objects must be removed before upgrade or you risk database sadness. As part of this bump, some incompatible changes have been made. If you’ve been trying out this alpha feature, please make sure to check out the changes and take appropriate action in your development/testing environments.

Release Schedule

Next Deadline: Docs Due, March 21st; Test Freeze, March 22nd

We are in Code Freeze. Please fix any reported test failures or bugs (urgently). Your final enhancement documentation is due for editing Tuesday, and on Wednesday they’ll halt all changes to tests. The release team is also drafting the release notes and the release blog.

Patch releases 1.26.3, 1.25.8, 1.24.12 are available. Version 1.23 is now EOL, and 1.23.17 was the last patch release for that version.

Signing the images for this month’s patch releases did not go well, so you can’t yet enforce signing on all images via cosign. Next month, hopefully.

Featured PRs

#96120: KEP 2258: add node log query

While the venerable kubectl logs command has long provided quick access to log output from containers running in Kubernetes, this hasn’t extended to node-level logs. We’ve slowly made more node debugging tools available in-band both to provide a unified experience when debugging node troubles and to better support minimalist OSes, and with this PR we and now get to the underlying node logs remotely. As with container logging, this is part of the Kubelet API. On Linux it queries journald and for Windows, the Event Log. There isn’t yet a dedicated kubectl command for it but you can still try things out using a command like kubectl get --raw "/api/v1/nodes/$NODE_NAME/proxy/logs/?query=kubelet" (after enabling the feature gate, of course). Give it a try in your development clusters and report back to SIG-Node if you can!

#113218: Add certificates.k8s.io/v1alpha1 ClusterTrustBundle

There have been a lot of different solutions to distributing CA trust information in Kubernetes, from the inline embedding of admission webhooks to the magic ConfigMap used for in-cluster API access (and lots more from third-party projects). This new API type seeks to unify these. For now it’s mostly just a data-holder, the only behaviors so far are basic validations, however the goal is to grow it over time. If you work on any third-party tools that need (or already have) a mechanism to package up trust roots and use them with API objects, consider adding support for this new API when you can. Future extensions planned are reference support in webhooks and the ability to mount the PEM files into containers using the projected volume type.

#115075: IPAddress allocator

The existing IP allocator for Services has been with us for a long time and while battle-hardened it has some long term limitations that have frustrated many an admin. The biggest of these has been limiting the subnet size it can allocate from, a /12 for IPv4. A million IPs can seem like a lot until you’ve got heavy automation creating 10 Services on every commit and suddenly the bitmap allocator is having a very sad day. So we’ve created a new allocator that lets Etcd and Kube-apiserver handle more of the heavy lifting and thus removing most of the current limitations. This works by using an IPAddress API object as a mutual-exclusion lock and a new allocator in the Service controller. There’s still a lot of room for improvement of the actual allocation algorithms compared to the old system however this PR adds the basics so folks can kick the tires in 1.27 and see how it behaves.

Other Merges

• Topology Aware Hints get validated, as do ResourceClaims, just in time for the annotation to be replaced by Topology Mode
• Don’t check topology constraints if the pod selector is null, and don’t try to match null MatchLabelKeys
• Windows stats will only be updated once every 10 seconds to reduce overhead
• Handle nil response to discovery
• Add custom messages for Validating Admission Policies, as well as custom matching conditions and a type system
• Even stateless pods can have user namespaces
• Send a Cloud Provider a –node-ip directive
• kmsv2 supports DEK key rotation
• kubectl explain defaults to OpenAPIv3
• CloudControllerManager can host webhooks to better support vendor needs
• PodSchedulingContext is the new PodScheduling
• Eviction calls should handle preconditions and not actually evict in a dry-run
• Prevent the scheduler from mixing up Pod UIDs
• Rate-limit the podresource API to prevent DOS
• Don’t include APF wait times in SLO stats
• CRDs can now be ApplySet parent objects
• Change the AllocationResult resource field so that it can support DRA, which requires advancing resources to v1alpha2
• Some new HPA metrics, and more
• Don’t lose events when reinitializing the watch cache
• Name and default resize policies sensibly
• DRA values are in the PodResources API
• You can have an IngressClassName and and IngressClass Annotation
• The AppProtocol field has standard values
• Pods can use the net.ipv4.ip_local_reserved_ports sysctl setting
• Pods that aren’t restartable are marked terminal
• Pods created by a Job carry the Job Name
• Better Job controller backoff logic
• PodDNSPolicies apply to Windows
• The NodeResourceTopology API has been removed from k/k staging, it will be back in a new repo in the future

Contextual Logging: defaultbinder, kube-controller-manager, controller utils, daemonset, volumes

Testing Overhaul: resize policy defaults, fix kubemark deps, standalone test, more standalone, StatefulSet defaulting, e2e pluggability, snapshot resize

Promotions

• --subresource is beta, plus testing
• NodeExpandSecretRef to beta
• StatefulSet AutoDelete to beta
• SELinuxMountReadWriteOncePod to beta
• HPA container resource metrics to beta
• KubeletTracing to beta
• CSINodeExpandSecret to beta
• KMSv2 to beta

Deprecated

• DRA Kubelet plugin is v1alpha2, and v1alpha1 is disabled
• The old wait polling methods are deprecated and being replaced by PollUntilContextCancel

Version Updates

• kustomize to v5.0.1, and update its dependencies
• ginkgo to v2.9.1 and gomega to v1.27.3
• coredns to 1.10.1