Weekly GitHub Report for Kubernetes: September 22, 2025 - September 29, 2025 (12:05:00)
Weekly GitHub Report for Kubernetes
Thank you for subscribing to our weekly newsletter! Each week, we deliver a comprehensive summary of your GitHub project's latest activity right to your inbox, including an overview of your project's issues, pull requests, contributors, and commit activity.
Table of Contents
I. News
1.1 Recent Version Releases:
The current version of this repository is v1.32.3
1.2 Version Information:
The Kubernetes 1.32 release, announced on March 11, 2025, introduces several key updates and improvements detailed in the official CHANGELOG, with additional binary downloads available. This version continues to enhance Kubernetes' functionality and stability, reflecting ongoing development trends in the platform.
II. Issues
2.1 Top 5 Active Issues:
We consider active issues to be issues that that have been commented on most frequently within the last week. Bot comments are omitted.
-
User Namespace: Allow simultaneous use of hostNetwork and userns: This issue discusses the possibility of allowing Kubernetes static pods to simultaneously use
hostNetworkand user namespaces (userns), which is currently restricted due to limitations in container runtimes like runc. The proposal aims to lift this restriction to enhance security by enabling userns for control plane components that require host networking, while acknowledging that only certain runtimes such as crun currently support this combination.- The comments include requests for testing the feasibility of combining hostNetwork and userns, clarifications that runc does not support this combination, and shared experiences disabling validation to run static pods with both features. Participants discuss runtime support limitations, the impact on kubeadm and kubelet behavior, and the potential for introducing a configurable option to enable this feature in the future, with consensus on the need for further runtime improvements before full adoption.
- Number of comments this week: 10
-
Preemption process does not leave a trait in the scheduler cache causing a race: This issue describes a race condition in the Kubernetes scheduler’s preemption process where the scheduler cache does not immediately reflect pod deletions, causing repeated attempts to preempt non-existent pods and generating excessive API server load. The problem is exacerbated during autoscaling events when pods are rapidly scheduled and preempted, and a recent change in version 1.34 introduced behavior that worsens this loop, prompting discussion about rolling back that change or applying a fix to update the cache more promptly.
- The comments acknowledge the race condition and its impact on API server load, discuss potential solutions for updating the scheduler cache either immediately after a delete call or upon delete failure, and consider rolling back a recent PR as a quick mitigation. Contributors share proposed fixes, emphasize the need for careful review and testing to avoid regressions, and highlight the importance of adding presubmit tests and increasing code coverage to prevent similar issues in future releases.
- Number of comments this week: 9
-
why net.core.somaxconn is unsafe in k8s?: This issue questions why the kernel parameter
net.core.somaxconnis considered unsafe in Kubernetes, despite its use in configuring network settings viasecurityContext.sysctls. The user seeks clarification on the rationale behind marking this parameter as unsafe, especially given changes in default values across kernel versions and the potential implications for container network namespaces.- The comments reference a past proposal discussing the safety of
net.core.somaxconn, noting it is a namespaced sysctl and may be safe, with all net/core sysctls generally considered safe. However, concerns remain about possible kernel memory exhaustion if the parameter is set too high, and further evidence is requested to confirm whether memory allocation for the accept backlog is managed within container cgroups to determine its safety. - Number of comments this week: 7
- The comments reference a past proposal discussing the safety of
-
CEL CIDR parser does not behave as documented: This issue addresses a discrepancy between the documented behavior and the actual implementation of the CIDR-related CEL extensions in Kubernetes, where the parser accepts strings that the documentation states it should reject. The discussion focuses on whether to fix the parser to match the documentation, potentially causing an API break, or to update the documentation to reflect the current behavior, with suggestions including adding a new function like
isCIDRMask()to clarify validation semantics.- The comments include suggestions for workarounds using CEL expressions, reflections on the original design decisions and documentation inconsistencies, proposals to introduce new functions to distinguish between CIDR masks and addresses, and requests for input from experts on CEL libraries and declarative validation, culminating in the issue being triaged and assigned for further action.
- Number of comments this week: 7
-
Asynchronous preemption can be initiated even before the previous one is finished for the same pod: This issue describes a problem in the Kubernetes scheduler where asynchronous preemption can be triggered multiple times for the same pod before the previous preemption finishes, leading to excessive and redundant Pod/Delete API calls. This occurs because existing safeguards designed to prevent such race conditions are bypassed due to recent feature changes, causing performance regressions especially under API throttling conditions.
- The comments discuss the root cause linked to recent features that remove safeguards, confirm the regression is reproducible under certain test conditions, and propose quick mitigations such as disabling AsyncPreemption, PopFromBackoffQ, or AsyncAPICalls. They also suggest longer-term fixes like marking pods as pending deletion to prevent duplicate preemptions and improving test coverage for slower API responses.
- Number of comments this week: 7
2.2 Top 5 Stale Issues:
We consider stale issues to be issues that has had no activity within the last 30 days. The team should work together to get these issues resolved and closed as soon as possible.
- Zone-aware down scaling behavior: This issue describes a problem with Kubernetes Horizontal Pod Autoscaler (HPA) workloads where, during scale-in events, the distribution of pods across availability zones becomes unbalanced despite using topology spread constraints with
maxSkew: 1. Specifically, the scale-in process can result in one zone having significantly fewer pods than others, causing high CPU usage on the lone pod in that zone and violating the expected even pod distribution across zones. - apimachinery's unstructured converter panics if the destination struct contains private fields: This issue describes a panic occurring in the apimachinery's DefaultUnstructuredConverter when it attempts to convert an unstructured object into a destination struct that contains private (non-exported) fields. The reporter expects the converter to safely ignore these private fields instead of panicking, as this problem arises particularly with protobuf-generated gRPC structs that include private fields for internal state, causing the conversion process to fail unexpectedly.
- Integration tests for kubelet image credential provider: This issue discusses the potential addition of integration tests specifically for the kubelet image credential provider, similar to the existing tests for client-go credential plugins. It suggests that since there are already integration tests for pod certificates, implementing similar tests for kubelet credential plugins would be a logical and beneficial next step.
- conversion-gen generates code that leads to panics when fields are accessed after conversion: This issue describes a bug in the conversion-gen tool where it generates incorrect conversion code for structs that have changed field types between API versions, specifically causing unsafe pointer conversions instead of properly calling the conversion functions. As a result, accessing certain fields like
ExclusiveMaximumafter conversion leads to runtime panics, highlighting the need for conversion-gen to produce safe and correct code to prevent such crashes. - Failure cluster [ff7a6495...] TestProgressNotify fails when etcd in k/k upgraded to 3.6.2: This issue describes a failure in the TestProgressNotify test that occurs when the etcd component in the Kubernetes project is upgraded to version 3.6.2. The test times out after 30 seconds waiting on a result channel, with multiple errors indicating that the embedded etcd server fails to set up serving due to closed network connections and server shutdowns.
2.3 Open Issues
This section lists, groups, and then summarizes issues that were created within the last week in the repository.
Issues Opened This Week: 26
Summarized Issues:
- Kernel parameter safety concerns: The kernel parameter
net.core.somaxconnis considered unsafe in Kubernetes because allowing it to be set arbitrarily large in container network namespaces can lead to kernel memory exhaustion. Although its use requires explicit authorization via--allowed-unsafe-sysctls, the potential risks remain unclear and need clarification.
- Priority and admission control inconsistencies: Multiple issues describe problems with PriorityLevelConfiguration and MutatingAdmissionPolicy where intended request rejections or default PriorityClass assignments fail, resulting in inconsistent enforcement or forbidden errors. These issues highlight challenges in correctly applying priority and admission policies in Kubernetes.
- Scheduler preemption race conditions and regressions: The Kubernetes scheduler suffers from race conditions and regressions in asynchronous preemption, causing repeated deletion attempts of non-existent pods and excessive API server load. These problems are exacerbated during autoscaling and due to removal of safeguards like backoff queues.
- Node resource and system stability issues: Some Kubernetes nodes experience intermittent failures such as "No space left on device" errors affecting cpuset.cpus settings, leading to node reboots, while others report kubelet metrics anomalies or failure to update ephemeral-storage capacity after disk expansion. These issues impact node stability and resource reporting accuracy.
- Client-go resource leaks: A critical resource leak in the client-go library occurs because HTTP response bodies are not closed properly, potentially causing memory and file descriptor exhaustion in long-running or high-traffic Kubernetes API clients.
- Parser and validation discrepancies: The CEL CIDR parser accepts inputs that contradict documentation, causing confusion about whether to fix the parser or update docs. Additionally, the ResourceClaim field "Effect" is incorrectly annotated as optional when it should be required, indicating validation inconsistencies.
- Server-side apply conflicts: Using server-side apply with server-side defaulting can cause conflict errors when applying the same manifest twice due to mismatched field management, particularly on ValidatingAdmissionPolicy resources.
- Pod lifecycle and status inconsistencies: Pods with multiple init containers may incorrectly transition to Running immediately after node reboot due to preserved ContainerStatus states, causing phase flip-flops instead of remaining Pending until all containers start.
- Test flakiness and instability: Various tests, including CPU Manager Metrics on specific kernel/containerd versions, AWS kops cluster TLS handshake, Kubernetes e2e cluster startup, and pod autoscaler controller tests, exhibit flaky failures causing instability and blocking PR updates.
- Scheduler performance benchmarking: There is a request to add workloads to the scheduler_perf benchmark that simulate lowered kube-apiserver performance to better evaluate scheduler behavior under preemption and detect regressions.
- User namespace and hostNetwork usage restriction: Kubernetes currently restricts simultaneous use of
hostNetworkand user namespaces, limiting static control plane pods from using user namespaces for enhanced security. The proposal aims to lift this restriction contingent on container runtime support.
- Declarative validation migration: The Kubernetes project is progressing on migrating to declarative validation for all unversioned API kinds and subresources by adding validation generation tags, updating strategies, and enabling tests, tracked via a detailed checklist.
- Host networking lifecycle hook failures: Pods configured with host networking fail to execute
httpGetpreStop lifecycle hooks because the pod’s network status lacks IP information, causing hook failures that do not occur with pod networking.
- Log size configuration discrepancies: Despite configuring kubelet with a containerLogMaxSize of 1536Ki, log files in
/var/log/podsexceed this size significantly, with some logs reaching 7.8MB, indicating a mismatch between configuration and actual log file sizes.
- Broken documentation links: A link on the kubernetes.io documentation for PersistentVolume Provisioner examples points to a non-existent README.md file, causing a broken link and potential confusion for users.
- Local state mutation before server confirmation: The kubectl drain package mutates the local node object to reflect desired state before the server patch or update request succeeds, risking local state inconsistency if the request fails.
- Resource claim deallocation failures: Resource claims created by the scheduler for extended resources are not properly de-allocated after pod completion, causing resource claims to remain allocated and potentially leading to resource leaks.
2.4 Closed Issues
This section lists, groups, and then summarizes issues that were closed within the last week in the repository. This section also links the associated pull requests if applicable.
Issues Closed This Week: 3
Summarized Issues:
- Code generation failures due to missing dependencies: The code generators produce blank output files when the 'gofmt' executable is missing from the system path, as the code returns nil in such cases instead of handling the error properly. This causes silent failures in the generation process, impacting build reliability.
- issues/134207
- CI test failures caused by system environment issues: The end-to-end node test suite failed due to missing required cgroups (cpuset) in cgroups v1, which caused system validation errors affecting multiple CI jobs. The problem was resolved by reverting the change that introduced the failure, restoring CI stability.
- issues/134234
- IPv6 cluster startup failures in CI: Kubeadm's IPv6 functionality was broken on the master branch, causing IPv6 clusters to fail to start healthily in continuous integration tests. Fixes were applied to restore successful operation of related CI jobs and maintain IPv6 support.
- issues/134272
2.5 Issue Discussion Insights
This section will analyze the tone and sentiment of discussions within this project's open and closed issues that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.
Based on our analysis, there are no instances of toxic discussions in the project's open or closed issues from the past week.
III. Pull Requests
3.1 Open Pull Requests
This section provides a summary of pull requests that were opened in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.
Pull Requests Opened This Week: 72
Key Open Pull Requests
1. [WIP] fix unit tests for watchlist : This pull request is focused on fixing and improving the unit tests related to the watchlist functionality in the Kubernetes client-go library, including enabling the WatchListClient by default and updating various informer and cache components to ensure test reliability and correctness.
- URL: pull/134292
- Merged: No
- Associated Commits: f68d6, 70c2b, 72204, b9033, 507fa, 40171, 6433b, 9d67d, 08ed4, 7a564, bd9b2, cf56b, ecf05, c3620, 581dd, 5513b
2. [WIP] Updates to DRA Partitionable Devices feature: This pull request is a work-in-progress feature update that enhances the DRA Partitionable Devices functionality in Kubernetes by refining types, validation, allocation logic, and associated tests, as detailed in the related Kubernetes Enhancement Proposal (KEP) 4815.
- URL: pull/134189
- Merged: No
3. Refactor: Centralize declarative validation logic: This pull request refactors and centralizes the declarative validation and migration logic for resource strategies in Kubernetes by introducing a new helper function, rest.ValidateDeclarativelyWithMigrationChecks, which reduces code duplication and complexity, while also simplifying scale subresource testing, adding unit tests for metric identifier generation, and removing obsolete validation functions and options.
- URL: pull/134240
- Merged: No
Other Open Pull Requests
- Declarative validation migration: Multiple pull requests migrate validation logic from hand-written code to declarative validation using +k8s tags, improving consistency and maintainability. These include validation for ResourceClaim, ClusterRole API group, DeviceClass selector limits, and OpaqueDeviceConfiguration.Driver field, with added tests and improved error messages.
- [pull/134211, pull/134288, pull/134302, pull/134286]
- Kubelet bug fixes and improvements: Several pull requests address kubelet issues including immediate readiness status update on container termination, fixing pod cache hang and duplicate container names due to backward time shifts, preserving pod failure status after admission rejection, and excluding init containers from triggering Pod phase transitions to improve status accuracy.
- [pull/134188, pull/134204, pull/134186, pull/134190]
- Code refactoring and cleanup: Multiple PRs refactor code for better structure and maintainability, such as reorganizing storage.getList function for future iterator migration, refactoring client-go functions to use WithContext versions, replacing MutationCache with SecretLister in serviceaccount-tokens-controller, and updating comments to clarify bug workaround origins.
- [pull/134202, pull/134208, pull/134267, pull/134201]
- Logging and scheduling improvements: Pull requests improve logging by migrating device manager components to contextual logging and enhance scheduler performance by marking assumed pods pending deletion to avoid redundant preemption API calls.
- [pull/134293, pull/134205]
- Test and CI enhancements: Some PRs focus on test improvements including adding missing skip messages for clarity, stabilizing flaky autoscaling HPA CPU tests by replacing immediate checks with polling, and experimental attempts to modify binding-related integration tests.
- [pull/134195, pull/134203, pull/134266]
- API and admission plugin updates: Updates include moving the OwnerReferencesPermissionEnforcement admission plugin to the apiserver module for better registration flexibility and making nodePortAddresses optional in kube-proxy-config.v1alpha1 to align with documentation and behavior.
- [pull/134246, pull/134206]
- Resource management and quota adjustments: A pull request adjusts the DRA extended resource quota to include device usages from regular resource claims, improving resource tracking.
- [pull/134210]
- Dependency and image updates: One PR removes runtime dependencies on gogo protobuf by updating the AST rewriter and replacing utilities, while another updates CSI sidecar container images to their latest versions and improves related code style.
- [pull/134256, pull/134304]
- Bug fixes in networking and warnings: Fixes include correcting an IPv6 allocator bug by replacing an improper cast to prevent out-of-range addresses and removing an incorrect warning about pod names with dots when hostname is explicitly set.
- [pull/134193, pull/134198, pull/134200]
3.2 Closed Pull Requests
This section provides a summary of pull requests that were closed in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.
Pull Requests Closed This Week: 28
Key Closed Pull Requests
1. feat(validation-gen): Introduce k8s:customUnique to control listmap uniqueness: This pull request introduces a new declarative validation tag, k8s:customUnique, in validation-gen to allow disabling uniqueness validation for listmap items, applies this tag to CertificateSigningRequestStatus.Conditions, and includes updates to tests and code generation to support this feature.
- URL: pull/134279
- Merged: Yes
2. gogo protobuf dependency cleanup: This pull request cleans up and isolates references to gogo protobuf packages by redefining the subset of methods relied upon in local interfaces and removing all non-generated code references to gogo dependencies, thereby reducing unnecessary gogo protobuf dependencies in the Kubernetes project.
- URL: pull/134228
- Merged: Yes
3. Automated cherry pick of #134265: kubeadm: ensure waiting for apiserver uses a local client: This pull request is an automated cherry pick that ensures kubeadm waits for the kube-apiserver by using a local client configured to connect directly to the local API server endpoint instead of the control plane endpoint, improving reliability during the init and join phases.
- URL: pull/134269
- Merged: Yes
Other Closed Pull Requests
3.3 Pull Request Discussion Insights
This section will analyze the tone and sentiment of discussions within this project's open and closed pull requests that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.
Based on our analysis, there are no instances of toxic discussions in the project's open or closed pull requests from the past week.
IV. Contributors
4.1 Contributors
Active Contributors:
We consider an active contributor in this project to be any contributor who has made at least 1 commit, opened at least 1 issue, created at least 1 pull request, or made more than 2 comments in the last month.
If there are more than 10 active contributors, the list is truncated to the top 10 based on contribution metrics for better clarity.
| Contributor | Commits | Pull Requests | Issues | Comments |
|---|---|---|---|---|
| pohly | 37 | 9 | 5 | 78 |
| BenTheElder | 4 | 1 | 3 | 52 |
| liggitt | 11 | 3 | 1 | 44 |
| p0lyn0mial | 51 | 3 | 0 | 0 |
| pacoxu | 11 | 6 | 1 | 33 |
| macsko | 13 | 8 | 3 | 27 |
| huww98 | 24 | 6 | 0 | 20 |
| dims | 14 | 4 | 4 | 23 |
| yongruilin | 25 | 4 | 3 | 10 |
| serathius | 18 | 4 | 1 | 16 |