Weekly GitHub Report for Kubernetes: December 01, 2025 - December 08, 2025 (12:03:35)
Weekly GitHub Report for Kubernetes
Thank you for subscribing to our weekly newsletter! Each week, we deliver a comprehensive summary of your GitHub project's latest activity right to your inbox, including an overview of your project's issues, pull requests, contributors, and commit activity.
Table of Contents
I. News
1.1 Recent Version Releases:
The current version of this repository is v1.32.3
1.2 Version Information:
The Kubernetes 1.32 release, announced on March 11, 2025, introduces several key updates and improvements detailed in the official CHANGELOG, with additional binary downloads available. This version continues to enhance Kubernetes' functionality and stability, reflecting ongoing development trends in the platform.
II. Issues
2.1 Top 5 Active Issues:
We consider active issues to be issues that that have been commented on most frequently within the last week. Bot comments are omitted.
-
kubelet TopologyManager Stalls on Large NUMA Systems (e.g. Nvidia GB200 Compute Tray with 34 NUMA nodes): This issue describes a problem where kubelet’s TopologyManager stalls indefinitely when scheduling pods that request PCI devices on very large NUMA systems, such as an Nvidia GB200 compute tray with 34 NUMA nodes. The exponential complexity of generating device topology hints causes kubelet to never reach the device Allocate call, leaving pods or VMs stuck in a Pending or Starting state despite available resources.
- The comments confirm the root cause as the TopologyManager’s exponential iteration over NUMA bitmasks, which blocks kubelet from calling Allocate and causes pods to hang; disabling TopologyManager or simplifying the workload immediately resolves the issue, and the discussion includes acknowledgments and triage labels indicating the problem’s priority and scope.
- Number of comments this week: 7
-
[Flake Test] k8s.io/kubernetes/pkg/controller: devicetainteviction - TestDeviceTaintRule: This issue reports a flaky test failure in the Kubernetes device taint eviction controller, specifically the TestDeviceTaintRule unit test, which has been intermittently failing since November 20, 2025. The flake appears to be caused by a race condition between processing pod deletion and updating the DeviceTaintRule status, leading to inconsistent test output due to concurrent worker execution.
- The comments discuss observed latency and detailed logs showing the race condition causing the test flake, with suggestions that limiting the number of concurrent workers to one could prevent the race in unit tests. There is also a proposal to relax the test’s strict equality check on status messages to reduce flakiness, and consensus that the race condition likely does not cause issues in real cluster operation but affects test reliability.
- Number of comments this week: 6
-
Kubelet fails to start on WSL2: "wrong number of fields" parsing /proc/mounts with Docker Desktop: This issue describes a problem where the kubelet fails to start on WSL2 when Docker Desktop is installed due to an error caused by improperly escaped spaces in mount entries within /proc/mounts. The root cause is that Docker Desktop’s 9p filesystem mounts include Windows paths with unescaped spaces, which leads the kubelet’s parser to misinterpret the number of fields, violating the expected format and causing startup failure.
- The comments confirm that the kubelet’s parsing logic is correct according to the fstab specification, which requires spaces in mount options to be escaped; the issue lies with Docker Desktop or WSL2 generating non-compliant mount entries. Contributors suggest filing a bug with Docker Desktop rather than modifying kubelet, noting that users have limited workarounds and that defensive parsing in kubelet is possible but unlikely to be accepted by maintainers.
- Number of comments this week: 5
-
PV Controller: workqueue accumulation when periodic resync overwhelms single-threaded volume and claim workers: This issue addresses the problem of workqueue accumulation in the PersistentVolume controller caused by periodic resyncs that enqueue all volumes and claims faster than the single-threaded volume and claim workers can process them. The single-threaded design is necessary due to non-thread-safe shared state accessed during volume and claim synchronization, which complicates efforts to improve throughput and reduce backlog.
- The comments acknowledge this as a known and recurring performance problem with references to related issues, discuss partial fixes made, and highlight the complexity of enabling multi-threaded processing due to the controller’s reliance on non-thread-safe shared state.
- Number of comments this week: 4
-
Make credential plugin policy/allowlist settable with
kubectl kuberc set: This issue requests enhancing thekubectl alpha kuberc setcommand to allow users to set thecredentialPluginPolicyandcredentialPluginAllowlistfields directly from the command line. This improvement aims to simplify security configuration for end-users by enabling easy activation of these security features without manually editing configuration files, which also benefits cloud providers by allowing them to offer straightforward CLI commands for plugin management.- The comments indicate the issue has been assigned to the appropriate maintainer and relevant SIGs, accepted for inclusion in milestone v1.36, and marked with an important-longterm priority, reflecting consensus on its significance and planned implementation timeline.
- Number of comments this week: 4
2.2 Top 5 Stale Issues:
We consider stale issues to be issues that has had no activity within the last 30 days. The team should work together to get these issues resolved and closed as soon as possible.
- Zone-aware down scaling behavior: This issue describes a problem with the horizontal pod autoscaler (HPA) scale-in behavior in a Kubernetes deployment that uses topology spread constraints to evenly distribute pods across multiple zones. Specifically, during scale-in events, the pods become unevenly distributed with one zone ending up with significantly fewer pods than allowed by the configured
maxSkew: 1, causing high CPU usage on the lone pod in that zone and violating the expected balanced pod distribution. - apimachinery's unstructured converter panics if the destination struct contains private fields: This issue describes a panic occurring in the apimachinery's DefaultUnstructuredConverter when it attempts to convert an unstructured object into a destination struct that contains private (non-exported) fields. The reporter highlights that the converter should ideally skip these private fields instead of panicking, especially since protobuf-generated gRPC structs often include private fields that cause this failure despite the unstructured data only containing public fields.
- Integration tests for kubelet image credential provider: This issue discusses the need to create integration tests for the kubelet image credential provider, similar to the existing tests for client-go credential plugins. It suggests that since there are already integration tests for pod certificate functionality, adding tests for the kubelet credential plugins would be a logical and beneficial extension.
- conversion-gen generates code that leads to panics when fields are accessed after conversion: This issue describes a bug in the conversion-gen tool where it generates incorrect conversion code for structs that have changed field types between API versions, specifically causing panics when accessing fields after conversion. The problem arises because the generated code uses unsafe pointer conversions instead of calling the appropriate conversion functions, leading to runtime errors when fields like
ExclusiveMaximumare accessed. - Failure cluster [ff7a6495...] TestProgressNotify fails when etcd in k/k upgraded to 3.6.2: This issue describes a failure in the TestProgressNotify test that occurs when the etcd component in the Kubernetes project is upgraded to version 3.6.2. The test times out after 30 seconds waiting on a result channel, with multiple errors indicating that the embedded etcd server fails to set up serving due to closed network connections and server shutdowns.
2.3 Open Issues
This section lists, groups, and then summarizes issues that were created within the last week in the repository.
Issues Opened This Week: 25
Summarized Issues:
- Test Failures and Flakiness in NFS and Integration Tests: Multiple issues report flaky or failing tests related to NFS volumes and integration tests in Kubernetes. These failures are caused by timeouts, missing binaries in test environments, or intermittent test behavior affecting CI jobs and e2e suites.
- issues/135523, issues/135540, issues/135556
- Scheduling and Pod Queue Management Issues: There are problems with pod scheduling including kubelet’s TopologyManager stalling on large NUMA systems and requests for requeue mechanisms to handle unschedulable pods. These issues cause pods and VMs to remain stuck or require enhancements to support gang scheduling.
- issues/135528, issues/135541, issues/135561
- Workqueue and Controller Performance Bottlenecks: The PersistentVolume controller experiences backlog due to single-threaded workqueues being overwhelmed by periodic resyncs, causing processing delays. This highlights thread safety constraints impacting volume and claim processing throughput.
- issues/135542
- Code and Interface Improvements in Kubernetes Components: Several issues call for cleaner interfaces and conditional logic improvements, such as respecting custom serializers in code-generator clientsets and defining clearer interfaces between kube-proxy and its backends. These changes aim to reduce complexity and improve extensibility.
- issues/135544, issues/135546
- Resource and Configuration Projection Bugs: There are bugs in resource projection such as the downwardAPI incorrectly returning default CPU limits and kubelet failing to start on WSL2 due to parsing errors in mount options. These issues cause incorrect resource reporting and startup failures.
- issues/135550, issues/135554
- Virtualization and Device Management Standardization: Requests to standardize attributes like mdevUUID in KubeVirt aim to simplify device virtualization integration. Additionally, device taint eviction controller tests suffer from race conditions causing flaky test results.
- issues/135552, issues/135588
- Debugging and CLI Enhancements: Proposals include adding flags to kubectl debug for sharing volumes and enabling configuration of credential plugin policies via kubectl commands. These enhancements seek to improve usability and security configuration workflows.
- issues/135558, issues/135574
- Documentation and User Experience Issues: Confusion arises from inaccurate kubectl apply --dry-run=client documentation, where the output does not match user expectations, leading to misunderstandings about command behavior.
- issues/135616
- Data Races and Concurrency Bugs: Multiple data race conditions have been detected in components like the garbage collector, resource quota monitor, and CEL openapi resolver, causing unsafe concurrent access and panics during testing and runtime.
- issues/135621, issues/135624, issues/135627
- SELinux and Volume Conflict Reporting Errors: The SELinux warning controller incorrectly reports conflicts for volumes used by Pods that have already terminated, despite volumes being unmounted, leading to false conflict alerts.
- issues/135628
- kube-proxy Performance and Stability Issues: Running kube-proxy in nftables mode on large clusters causes high CPU usage and kernel soft lockups, resulting in delayed rule syncing and impacting production stability.
- issues/135639
- Feature Gate and Resource Scaling Failures: Enabling InPlacePodVerticalScaling does not dynamically adjust Pod resources as expected because Kubernetes fails to update the parent cgroup slice, causing resource quota updates to be ignored and Pod crashes.
- issues/135640
- kubectl Apply and Custom Resource Diffing Bugs: Client-side kubectl apply on custom resources with map-typed lists causes kubectl diff to incorrectly mark new fields for deletion due to last-applied annotation inconsistencies, differing from server-side apply behavior.
- issues/135646
2.4 Closed Issues
This section lists, groups, and then summarizes issues that were closed within the last week in the repository. This section also links the associated pull requests if applicable.
Issues Closed This Week: 6
Summarized Issues:
- Environment Variable Handling Inconsistency: The Kubernetes EnvFiles feature preserves quotes literally in environment variable values, unlike ConfigMap which strips them, leading to unclear and inconsistent behavior. This discrepancy causes confusion about the expected syntax and behavior when using EnvFiles.
- issues/135519
- Security Vulnerability in kube-controller-manager: A medium-severity SSRF vulnerability in kube-controller-manager with the in-tree Portworx StorageClass allows authorized users to leak sensitive information by triggering unauthorized GET requests. The issue was mitigated by disabling the Portworx StorageClass by default in v1.31 and fixing it in later releases.
- issues/135525
- Metrics Registration Failure in kubelet: The use of sync.Once in metrics.go causes VolumeStatsCollector and LogMetricsCollector registration to fail after the first collector is registered. This results in missing important metrics data due to improper registration of these collectors.
- issues/135527
- Build Failure Due to Docker Network Template Parsing: Kubernetes builds prior to version 1.35 fail in the Dockerized build process because of a template parsing error related to missing Docker network IP address information. This causes image builds to fail and was resolved by subsequent patches.
- issues/135575
- Out-of-Memory Kill on GKE Pods Running Next.js: GKE pods running a Next.js server-side rendered application experience recurring OOM kills with Exit Code 137 after 5-11 hours due to steadily growing memory usage. This prompted investigations into memory management, resource limits, Node.js configuration, and monitoring to identify a potential memory leak.
- issues/135582
- Client-Server Compatibility Issue with client-go and EKS: Using client-go v0.35.0-rc.0 against an older Kubernetes v1.33 EKS cluster causes forbidden errors because the
sendInitialEventsfeature is enabled by default in the client but disabled on the server. Watch-list streams fail unless the WatchList feature gate is enabled on the server side. - issues/135583
2.5 Issue Discussion Insights
This section will analyze the tone and sentiment of discussions within this project's open and closed issues that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.
Based on our analysis, there are no instances of toxic discussions in the project's open or closed issues from the past week.
III. Pull Requests
3.1 Open Pull Requests
This section provides a summary of pull requests that were opened in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.
Pull Requests Opened This Week: 63
Key Open Pull Requests
1. Replace non-contextual API utilruntime.HandleError within the new context-aware version when possible: This pull request aims to replace existing calls to the non-contextual API utilruntime.HandleError with the new context-aware versions utilruntime.HandleErrorWithContext or utilruntime.HandleErrorWithLogger wherever possible, as part of the Kubernetes structured logging initiative to ensure log outputs are structured, include contextual fields, and properly handle context cancellation signals.
- URL: pull/135524
- Merged: No
2. selinux: Fix the controller to ignore finished pods: This pull request updates the SELinux warning controller to ignore completed Pods (both Succeeded and Failed) by treating them as if they were deleted, thereby preventing it from emitting conflict events for these finished Pods, and includes related refactoring, new unit tests, and end-to-end tests to validate this behavior.
- URL: pull/135629
- Merged: No
3. Promote workqueue metrics from ALPHA to BETA: This pull request promotes the workqueue metrics in the Kubernetes project from the ALPHA stage to the BETA stage, reflecting a stabilization and readiness for broader use as documented in the associated Kubernetes Enhancement Proposal (KEP) and linked issue.
- URL: pull/135522
- Merged: No
Other Open Pull Requests
- Codebase cleanup and dependency updates: Multiple pull requests focus on cleaning up deprecated APIs, removing archived dependencies, and updating linting tools. These changes include removing autoscaling/v2beta1 and v2beta2 APIs, replacing the archived grpc-prometheus dependency, moving httpcache to a forked directory, and updating golangci-lint to version 2.7.1 to fix linting issues and remove outdated build tags.
- Proxy and conntrack code reorganization: Several pull requests improve the structure and reuse of proxy-related code by moving conntrack setup from
cmd/kube-proxytopkg/proxyand splitting large proxier.go files into smaller, focused files. These changes enhance code maintainability without altering functionality.
- Testing improvements and flake fixes: Multiple pull requests address test reliability and correctness by fixing flakes, improving failure output, and modifying tests to avoid interference. This includes changes to TestDeviceTaintRule, DRA device taint tests, NFSv3 rpcbind race condition, and ReadWriteOncePod preemption tests.
- Bug fixes in resource parsing and kubelet image garbage collection: Fixes include correcting
resource.MustParseto handle large quantities nearmath.MaxInt64and decoupling MinAge and MaxAge validation in kubelet image garbage collection to prevent unnecessary image deletions. These changes improve correctness and efficiency.
- Workqueue and metrics enhancements: A pull request adds a "delayed" metric to the workqueue to better distinguish persistent backoff errors from transient ones and modernizes tests by replacing wait loops with synctest, improving alerting and test reliability.
- Mount-utils parsing fix for WSL2 Docker Desktop: The mount-utils parsing logic is modified to correctly handle
/proc/mountsentries with more than six fields, addressing issues caused by unescaped spaces in Windows path mount options. This ensures kubelet starts successfully on WSL2 environments.
- Endpoint handling for pod IP reuse: A fix removes an incorrect special case for Endpoint movement between EndpointSlices and implements logic to prefer the non-terminating pod’s Endpoint when pod IPs are reused, ensuring kube-proxy manages transient duplicate Endpoints correctly.
- Namespace creation warning for "kube-" prefix: A warning message is added to
kubectl create namespaceto alert users when creating namespaces with the "kube-" prefix, helping prevent conflicts with reserved Kubernetes system namespaces.
- Improved deployment controller performance: The deployment controller is optimized by adding a pod indexer and using owner references for pod queries, significantly reducing CPU usage, memory allocations, and latency.
- Declarative validation framework enabled for Event resource: The declarative validation framework is enabled for the Event resource, updating strategy code and adding test coverage as part of the ongoing migration effort.
- Fix for kubectl delete stuck on multiple StatefulSet pods: A bug causing
kubectlto hang when deleting multiple StatefulSet pods simultaneously is fixed, ensuring the command completes and exits normally.
- Linting enhancements for deprecation comments: A new godoclint check is introduced to detect incorrectly formatted deprecation comments and extra links in documentation, improving upon previous linting coverage.
- Fix for ConvertToVersion() GVK prioritization: The
ConvertToVersion()function is corrected to prioritize the object's own GroupVersionKind over the first registered kind when multiple kinds share the same Go type, fixing dynamic API server behavior.
- Addition of nodeManager interface to avoid import loops: An unexported
nodeManagerinterface is added topkg/proxy/healthcheckto prevent import loops and simplify unit testing in the healthcheck package.
3.2 Closed Pull Requests
This section provides a summary of pull requests that were closed in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.
Pull Requests Closed This Week: 30
Key Closed Pull Requests
1. Release v1.34.2 dd.1: This pull request is about preparing and managing the release process for Kubernetes version v1.34.2, including multiple release candidate commits, changelog updates, bug fixes, optimizations, and cherry-picked patches, although it was ultimately not merged.
- URL: pull/135632
- Merged: No
- Associated Commits: 40e8a, 180f5, 9710b, 26ec7, 10b10, a956e, 0bd03, e8bb3, b12b9, e5667, ed008, 41a1a, a46f1, e4096, f6713, ee314, c5d68, 7381c, 0bc56, a49ba, 4feac, 343b5, ed3cc, 78405, 91e47, 38a66, a99d6, f28b4, d1337, 3e75f, 98dca, 3ebbe, a7593, 08515, 94c0e, 2631a, 674cb, 96739, cb976, f779c, 48666, c45c5, 4fa47, 8c098, 7bd29, 04fd1, 76ab5, 45a88, b9739, 988bd, 93248, 16e3f, 4ae3b, 4a40d, 7d43d, 856ac, 35b87, 07ac8, 560cc, b74e8, b245b, 09ddc, 63a33, 3f170, b340e, 66366, e9f5e, 7ad36, 3b28d, 13b8d, 08013, f4f81, e438c, 1b4f5, a147a, 78003, 3ffc5, aea93, 2bfc9, 40899, 61c0f, ea75d, 74d58, d7aff, bc88c, dd524, d74bd, 5be5d, f306b, 5300f, ed6bf, f40e4, bddc1, 00b6a, 2674d, 8cd6b, 7191f, 55bc0, 6ebab, f3b63, 30fd2, b9369, 1d558, 03a5f, 60e3b, a42f8, e1e4e, 01133, 79ed9, 50096, 673a0, d5ffe, 7c5ec, aade7, b1373, 0a2c1, c48f8, 79a8f, fd812, f4fd1, 97650, 6c7b2, 5d5a4, 11e6c, 8b380, 732ea, 6f91c, 6c180, 77689, 11a8e, 3e31e, 8cc51, a7c0f, 27570, 8fe51, aaff3, 1b0b4, f5823, 5373a
2. add skew -2 version in SupportedEtcdVersion: This pull request adds support for a skew -2 version in the SupportedEtcdVersion list to address a failing test blocking issue #135553, helping to unblock progress on the publishing bot rules PR.
- URL: pull/135568
- Merged: Yes
3. Automated cherry pick of #135568: add skew -2 version in SupportedEtcdVersion: This pull request is an automated cherry pick of #135568 onto release-1.35 that adds support for a skew -2 version in SupportedEtcdVersion, includes fixes for etcd versions after rotation, and removes an obsolete test related to etcd version length.
- URL: pull/135569
- Merged: No
Other Closed Pull Requests
- Fixes for TransformingStore interface regression in client-go 1.34+: These pull requests fix a regression in Kubernetes client-go starting from version 1.34 by embedding the proper interface in TransformingStore to ensure that DeltaFIFO and RealFIFO implement it correctly. This enables informers to use configured Transformer functions without causing data inconsistency errors, as verified by added unit tests.
- Publishing rules updates and cleanup: These pull requests update the publishing rules by removing rules for the release-1.31 branch due to its end-of-life status and add publishing bot rules for the release-1.35 branch. They also include a cleanup effort addressing issue #4205 with coordination from release managers.
- Fixes for container IP detection during rsync build process: These pull requests fix the detection of the container's IP address during the build process involving rsync in versions 1.32, 1.33, and 1.34. Each addresses a bug specific to its version without backporting larger changes that remove rsync in later versions.
- Fixes for resource.MustParse parsing near math.MaxInt64: These pull requests address issues where resource.MustParse fails to correctly parse quantities near math.MaxInt64 by enhancing the AsInt64() function to properly convert inf.Dec values when representable. They refine the decimal parsing heuristic to allow a fast path for certain 19-digit values and add regression tests to ensure accurate handling of these edge cases.
- Go version and dependency upgrades: These pull requests update the Kubernetes project by bumping the Go programming language version to 1.25.5 and upgrading distroless iptables images. They also include updates to dependencies, container images, and versions to use Go 1.24.11, ensuring the build environment is current and secure.
- Test fixes and removals related to device tests and deprecated features: These pull requests fix failing device tests by correcting podresources API behavior and remove the flaking TestWatchStreamSeparation from storage/cacher tests due to feature deprecation. These changes aim to stabilize test results and maintain board health.
- Mutating Admission Policy bug fix for duplicate list items: This pull request fixes a bug in the Mutating Admission Policy where applying configurations to objects with duplicate list items failed due to a conversion error. The fix updates the ApplyStructuredMergeDiff function to allow duplicates during conversion while still enforcing strict validation on new patches.
- Dynamic Resource Allocation version skew handling: This pull request modifies the Dynamic Resource Allocation upgrade and downgrade logic to treat all versions up to and including x.y.z-alpha.0 as belonging to the previous minor version. This ensures version skew testing remains consistent during code freeze periods and early release cycles to prevent known failures.
- Reflector watchlist fallback log level adjustment: This pull request downgrades the log level of the reflector's watchlist fallback error message to V(4) to reduce noisy error logs. The original message is preserved and the error is attached as structured data for debugging during verbose logging.
- Security update for golang.org/x/crypto dependency: This pull request updates the golang.org/x/crypto dependency to version v0.45.0 to address two security vulnerabilities in the ssh and ssh/agent packages. This prevents unbounded memory consumption or program panics, ensuring a clean security state for the v1.35.0 release.
- Cloud-controller-manager manifest update: This pull request proposes updating the cloud-controller-manager manifest to use the latest available image version v34.2.0 from the Google Cloud Provider staging repository. This ensures the component runs the most recent release.
- Test and placeholder pull requests: These pull requests are test submissions or work-in-progress tests without merged changes or specific feature updates. They serve as placeholders or to trigger review features.
3.3 Pull Request Discussion Insights
This section will analyze the tone and sentiment of discussions within this project's open and closed pull requests that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.
Based on our analysis, there are no instances of toxic discussions in the project's open or closed pull requests from the past week.
IV. Contributors
4.1 Contributors
Active Contributors:
We consider an active contributor in this project to be any contributor who has made at least 1 commit, opened at least 1 issue, created at least 1 pull request, or made more than 2 comments in the last month.
If there are more than 10 active contributors, the list is truncated to the top 10 based on contribution metrics for better clarity.
| Contributor | Commits | Pull Requests | Issues | Comments |
|---|---|---|---|---|
| pohly | 18 | 9 | 9 | 38 |
| carlory | 27 | 14 | 1 | 25 |
| bwsalmon | 37 | 2 | 0 | 0 |
| aojea | 4 | 4 | 1 | 27 |
| jsafrane | 8 | 3 | 2 | 18 |
| danwinship | 10 | 5 | 1 | 14 |
| dims | 11 | 8 | 1 | 8 |
| harche | 7 | 6 | 4 | 9 |
| ania-borowiec | 2 | 2 | 0 | 20 |
| BenTheElder | 5 | 4 | 1 | 13 |