Weekly GitHub Report for Kubernetes

Thank you for subscribing to our weekly newsletter! Each week, we deliver a comprehensive summary of your GitHub project's latest activity right to your inbox, including an overview of your project's issues, pull requests, contributors, and commit activity.

---

Table of Contents

• I. News
  • 1.1. Recent Version Releases
  • 1.2. Other Noteworthy Updates
• II. Issues
  • 2.1. Top 5 Active Issues
  • 2.2. Top 5 Stale Issues
  • 2.3. Open Issues
  • 2.4. Closed Issues
  • 2.5. Issue Discussion Insights

• III. Pull Requests
  • 3.1. Open Pull Requests
  • 3.2. Closed Pull Requests
  • 3.3. Pull Request Discussion Insights

• IV. Contributors
  • 4.1. Contributors

I. News

1.1 Recent Version Releases:

The current version of this repository is v1.32.3

1.2 Version Information:

The Kubernetes version released on March 11, 2025, introduces key updates detailed in the official CHANGELOG, with additional binary downloads available. For comprehensive information on new features and changes, users are encouraged to consult the Kubernetes announce forum and the linked CHANGELOG.

II. Issues

2.1 Top 5 Active Issues:

We consider active issues to be issues that that have been commented on most frequently within the last week. Bot comments are omitted.

1. ResourceQuota check for Terminating Pod scope uses wrong field: This issue raises a concern that the ResourceQuota scope check for terminating Pods in Kubernetes might be using an incorrect field, specifically .spec.activeDeadlineSeconds instead of .metadata.deletionGracePeriodSeconds, which is set when a Pod deletion is triggered. The reporter seeks confirmation on this potential mistake to ensure that terminating Pods are correctly accounted for in ResourceQuota scopes labeled as Terminating or NotTerminating.

   • The comments provide references to design proposals and related issues, discuss the confusing terminology of "Terminating" versus "Run once" Pods, suggest deprecating the current quota scopes in favor of clearer names like BoundedRunDuration and UnboundedRunDuration, and highlight the inherent difficulty in definitively classifying Pods based on their termination behavior, ultimately leaving the issue open pending more evidence.
   • Number of comments this week: 8

2. Rename AllowlistEntry.Name to Command to match kubeconfig: This issue addresses the need to rename the field AllowlistEntry.Name to Command in order to align with the naming convention used in kubeconfig for exec plugins. The change is planned to occur before the v1 promotion of kuberc to ensure consistency between related Kubernetes configuration types.

   • The comments include labeling the issue with appropriate SIG and priority tags, linking it to a related enhancement proposal, and a brief discussion about who will take ownership of the work, with one contributor offering to help and another being assigned to continue the feature development.
   • Number of comments this week: 6

3. watch request for an unrecognized resource version should return error: This issue questions whether watch requests in the Kubernetes API should return an error when given an unrecognized or large resource version, as opposed to waiting indefinitely until that version becomes available. The user expects watch requests with a future resource version to error out immediately, similar to how get/list requests handle such cases, and seeks clarification on the design rationale behind the current behavior.

   • The comments clarify that the observed behavior is intentional and aligns with the Kubernetes API design: watch requests with future resource versions wait for updates rather than erroring, while list requests return errors if the resource version is unavailable or too old. Testing confirms this distinction, and the issue is considered resolved as the behavior matches documented semantics.
   • Number of comments this week: 3

4. rewrite command line testing: This issue proposes rewriting the current Kubernetes command line interface (CLI) tests, which are implemented as shell scripts using Kubernetes-specific helpers, by adopting a testing framework similar to the one used by the Go team for their CLI commands. The motivation behind this change is to lower the learning curve for newcomers, improve integration with continuous integration systems by enabling parallelism and better failure reporting, and potentially create a more maintainable and understandable testing approach.

   • The comments express cautious support for the idea, emphasizing the need to carefully evaluate the best testing tool before proceeding and noting past struggles with CLI tests in related projects. There is also discussion about the appropriateness of existing end-to-end tests for kubectl and the potential benefits of a unified testing framework that could run against a live API server, with a commitment to further discuss the topic in upcoming SIG-CLI meetings.
   • Number of comments this week: 3

5. Enhancement request: Allows paramsRef to check parameters on multiple namespace: This issue requests an enhancement to allow the paramsRef field in ValidatingAdmissionPolicyBinding to reference parameters across multiple namespaces by using a matchNamespaces selector, enabling validation against resources in different namespaces of the same kind. The user provides a use case where this feature would prevent hostname conflicts across namespaces for Ingress resources, which currently can only be checked within a single namespace, and seeks feedback on the feasibility and potential complexity of implementing this feature from SIG API Machinery maintainers.

   • The comments include a SIG API Machinery label assignment and a detailed discussion highlighting Kubernetes' limitations around consistency and transaction guarantees across namespaces and APIs, skepticism about the scalability and feasibility of using all cluster objects as parameters, and a request for further insights from SIG API Machinery leads to better understand the technical challenges and potential implementation of the requested feature.
   • Number of comments this week: 3

2.2 Top 5 Stale Issues:

We consider stale issues to be issues that has had no activity within the last 30 days. The team should work together to get these issues resolved and closed as soon as possible.

1. Zone-aware down scaling behavior: This issue describes a problem with the horizontal pod autoscaler (HPA) scale-in behavior in a Kubernetes deployment that uses topology spread constraints to evenly distribute pods across zones. Specifically, during scale-in events, the pods become unevenly distributed with one zone having significantly fewer pods than allowed by the maxSkew: 1 setting, causing high CPU usage on the lone pod in that zone and violating the expected balanced pod distribution.
2. apimachinery's unstructured converter panics if the destination struct contains private fields: This issue describes a panic occurring in the apimachinery's DefaultUnstructuredConverter when it attempts to convert an unstructured object into a destination struct that contains private (non-exported) fields. The reporter expects the converter to safely ignore these private fields instead of panicking, as this problem arises particularly with protobuf-generated gRPC structs that include private fields for internal state.
3. Integration tests for kubelet image credential provider: This issue proposes adding integration tests for the kubelet image credential provider, similar to the existing tests for client-go credential plugins. It suggests that since there are already integration tests for pod certificate functionality, implementing tests for kubelet credential plugins would be a logical and beneficial extension.
4. conversion-gen generates code that leads to panics when fields are accessed after conversion: This issue describes a bug in the conversion-gen tool where it generates incorrect conversion code for structs that have changed field types between API versions, specifically causing unsafe pointer conversions instead of properly calling the conversion functions. As a result, accessing certain fields like ExclusiveMaximum after conversion leads to runtime panics, highlighting the need for conversion-gen to produce safe and correct conversion code to prevent such crashes.
5. Failure cluster [ff7a6495...] TestProgressNotify fails when etcd in k/k upgraded to 3.6.2: This issue describes a failure in the TestProgressNotify test that occurs when the etcd component in the Kubernetes project is upgraded to version 3.6.2. The test times out after 30 seconds waiting on a result channel, with multiple errors indicating that the embedded etcd server fails to set up serving due to closed network connections and server shutdowns.

2.3 Open Issues

This section lists, groups, and then summarizes issues that were created within the last week in the repository.

Issues Opened This Week: 20

Summarized Issues:

• Code Generation Consistency: The defaulter-gen tool generates a defaults file even without the // +k8s:defaulter-gen tag, unlike similar tools that only generate files when the tag is present. This inconsistency leads to unexpected file generation behavior in the Kubernetes code-generator.
• issues/135417

• Null Value Handling in kubectl: Using kubectl create with a field set to null converts the value to 0, causing validation errors, whereas kubectl apply correctly removes the field when set to null. This discrepancy results in inconsistent and problematic resource management behaviors.
• issues/135423

• Field Naming Consistency: There is a proposal to rename AllowlistEntry.Name to Command to align with the kubeconfig exec plugin field name. This change aims to ensure naming consistency before promoting kuberc to v1.
• issues/135424

• kubectl Timeout Issues over VPN: Users experience random timeouts running kubectl commands through a Tailscale VPN connected via a subnet router on an AWS VPC hosting an EKS cluster. The cause of these intermittent failures is unclear, involving potential issues with kubectl, AWS, or Tailscale configurations.
• issues/135430

• Kubelet Watchdog Improvements: Enhancements are proposed to improve logging visibility for SdNotify failures and add internal timeouts to prevent blocking and silent hangs in the kubelet watchdog. These changes aim to facilitate faster root cause identification of kubelet heartbeat failures and restarts.
• issues/135449

• Watch Request Error Handling: The current Kubernetes API behavior waits for unrecognized or future resource versions in watch requests instead of returning immediate errors. This issue questions whether watch requests should align with get/list requests by returning errors promptly for invalid resource versions.
• issues/135452

• Volume Expansion Test Failures: Volume expansion end-to-end tests fail when a CSI driver allocates a larger volume than requested due to minimum size constraints. The test incorrectly calculates the new size based on the user request rather than the actual allocated size, causing expansion failures.
• issues/135456

• Workqueue Metrics Stability: There is a request to graduate workqueue metrics from alpha to beta to provide users with greater confidence in their stability. This promotion would enable infrastructure providers to reliably expose these metrics.
• issues/135463

• Device Binding Conditions (DBC) Beta Coordination: A central discussion tracks sub-issues related to the beta release of KEP-5007 for Device Binding Conditions, coordinating improvements and considerations before beta launch. This effort aims to ensure readiness and address outstanding concerns.
• issues/135472

• Device Attachment Workflow Enhancement: The current fail-and-reschedule device attachment approach is proposed to be replaced with a "happy path" using device taints and re-allocation. This aims to reliably bind fabric-attached devices like GPUs to pods without race conditions or binding failures.
• issues/135473

• DRA Device Binding Conditions Feedback: Feedback is sought from DRA driver developers on the DRADeviceBindingConditions feature to ensure API suitability before beta promotion. Emphasis is placed on input based on expected "happy-path" usage to reduce risks during beta.
• issues/135474

• DRA Device Binding Metrics Implementation: Planned metrics will evaluate the production readiness of DRADeviceBindingConditions, tracking resource claim allocations, timeouts, and PreBind latency. These metrics aim to ensure reliability and provide actionable insights to operators.
• issues/135475

• CLI Test Rewrite Proposal: A proposal suggests rewriting Kubernetes CLI tests from shell scripts to the Go testscript framework to improve readability, maintainability, and CI integration. This change acknowledges significant effort and invites community discussion.
• issues/135478

• Deployment Rolling Update Stuck Bug: Deployments with maxSurge=0 get stuck indefinitely during rolling updates if scaled down mid-update due to stale ReplicaSet annotations. This causes the deployment controller to enter an infinite loop, preventing update completion.
• issues/135483

• Resource Quantity Parsing Issue: The resource.MustParse function fails to correctly parse very large quantity values near math.MaxInt64, producing incorrect outputs. Direct construction of such quantities works as expected, highlighting a parsing bug.
• issues/135487

• Premature Pod OOMKills in GitLab Runner: The Kubernetes executor in GitLab Runner prematurely kills Java build pods with OOMKilled errors due to rapid memory allocation rates triggering false positives. This occurs despite pods staying within configured memory limits.
• issues/135490

• Failed ReplicaSet Cleanup Inquiry: There is a request for information on whether Kubernetes provides automated methods to clean up failed ReplicaSets that accumulate in clusters. This reflects a need for cluster maintenance improvements.
• issues/135491

• Scheduler Rescheduling for Device Binding Failures: Enhancements to the scheduler in DRADeviceBindingConditions are proposed to handle binding failures by introducing error statuses that allow rescheduling on the same device. This aims to better manage transient provisioning failures while preserving resource allocations.
• issues/135493

• ValidatingAdmissionPolicyBinding Multi-Namespace Params: An enhancement is requested to allow paramsRef in ValidatingAdmissionPolicyBinding to reference parameters across multiple namespaces using a namespace selector. This would enable cluster-wide validation of resource conflicts like duplicate hostnames.
• issues/135497

• Daemonset Filter Context Bug in kubectl Drain: The daemonset filter in k8s.io/kubectl/pkg/drain incorrectly uses context.TODO() instead of the drain helper's context, causing issues with timeouts and logging consistency during node drain operations.
• issues/135512

2.4 Closed Issues

This section lists, groups, and then summarizes issues that were closed within the last week in the repository. This section also links the associated pull requests if applicable.

Issues Closed This Week: 10

Summarized Issues:

• Race conditions in resource access and namespace recognition: Multiple issues describe race conditions where Kubernetes components attempt to access resources before they are fully available or recognized, leading to authorization errors or admission rejections. These race conditions cause 403 Forbidden errors in EKS kubelets and spurious "namespace not found" errors in ValidatingAdmissionPolicy, both resolved by adding fallback mechanisms or updates to resource authorization flows.
• issues/135175, issues/135352

• Test flakiness due to resource conflicts and parallelism: Several test failures are caused by resource sharing conflicts or parallel test setups, resulting in intermittent errors and timeouts. These include device allocation inconsistencies in the TestDRA suite and mTLS certificate verification failures in the Projected PodCertificate e2e tests.
• issues/135422, issues/135459

• Performance bottlenecks from inefficient locking and string operations: Performance issues arise from lock contention on a global mutex during KMSV2 envelope encryption metric updates and from quadratic time complexity in string concatenation within the ParseEnv function. Both issues degrade Kubernetes API server responsiveness and are addressed by optimizing locking and string building approaches.
• issues/135408, issues/135435

• Inconsistent or incomplete feature implementations causing unexpected behavior: Incomplete or inconsistent application of feature gates and validation rules lead to unexpected Kubernetes behavior, such as relaxed service name validation not applying uniformly in Ingress resources and inconsistent handling of duplicate nodePort numbers during service updates.
• issues/135425, issues/135439

• Authentication failures due to expired tokens: Authentication errors occur in the Kamaji control plane components caused by expired service account tokens, resulting in invalid bearer token errors and repeated authentication failures.
• issues/135450

• Unclear or unintelligible issue content: One issue contains largely unintelligible descriptions filled with emojis and lacks clear or specific information about the problem or its context.
• issues/135453

2.5 Issue Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed issues that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed issues from the past week.

---

III. Pull Requests

3.1 Open Pull Requests

This section provides a summary of pull requests that were opened in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Opened This Week: 62

Key Open Pull Requests

1. Improve RemoveStatusCondition by performing the operation in-place: This pull request improves the RemoveStatusCondition function by refactoring it to perform the removal operation in-place without reallocating the entire conditions slice, thereby optimizing memory usage while maintaining the behavior of removing at most one condition of the specified type.

• URL: pull/135509

• Merged: No

• Associated Commits: 33f42, 1b87f, 0820a, 34dc3, 2d154, 9bfd7, d8270, dcf19

2. schedule pod availability checks at the correct time in StatefulSets: This pull request fixes a bug in the StatefulSet controller by scheduling pod availability checks at the correct time to ensure .status.availableReplicas is updated promptly, thereby preventing delays in StatefulSet rollout progress and improving the handling of pods that are slow or struggling to become Ready.

• URL: pull/135428

• Merged: No

• Associated Commits: f7b88, bb890, 00cd3, 7f333

3. Add utility to summarize feature gates in markdown: This pull request adds a new command-line utility cmd/genfeaturegates that generates a markdown table summarizing all registered Kubernetes feature gates by extracting their information using public APIs, with options to sort and filter the output and write it to a file.

• URL: pull/135510

• Merged: No

• Associated Commits: eb75c, 6e03c, 36630, 0f307

Other Open Pull Requests

• Declarative Validation Enhancements: Multiple pull requests enhance declarative validation in Kubernetes by wiring storage group handling and enabling validation for namespaces and Horizontal Pod Autoscaler fields. These changes improve validation accuracy and testing capabilities by integrating new validation rules and validation-gen support.
  pull/135438, pull/135498, pull/135412

• OpenAPI Vendor Extensions Documentation: Two pull requests add and improve documentation for Kubernetes-specific OpenAPI vendor extensions such as x-kubernetes-list-type, x-kubernetes-list-map-keys, and x-kubernetes-unions. These updates align with existing documentation style and provide clearer explanations to improve API definition consistency and validation.
  pull/135413, pull/135429

• ResourceQuota and Security Fixes: Pull requests fix semantic mismatches in ResourceQuota scopes and close a security loophole allowing quota bypass. These changes improve quota accuracy by adjusting pod lifecycle tracking and enhance system security by enforcing quota restrictions.
  pull/135415, pull/135434

• Flag and Role Permission Updates: Updates include validation for the --service-cluster-ip-range flag to ensure unicast ranges and additions of workload-related permissions to cluster roles. These changes prevent misconfiguration issues and address previous permission omissions across view, edit, and admin roles.
  pull/135416, pull/135418

• Error Handling and Logging Improvements: Pull requests clarify the ErrorHandler interface with a new helper function and introduce contextual logging for feature gates. These improvements fix error processing issues and enhance unit test diagnostics by associating warnings with specific tests and ensuring thread safety.
  pull/135419, pull/135432

• Test Stability and Flakiness Fixes: Changes address flaky test behavior by adjusting timeouts and channel buffering to prevent goroutine leaks and ensure deterministic test execution. These fixes improve test reliability, especially under race detector and CI conditions.
  pull/135427, [pull/135441](https://github.com/kubernetes/kubernetes/pull/135441]

• Namespace Lookup Fallback Fixes: Multiple automated cherry picks implement fallback to live namespace lookup during admission to resolve spurious "namespace not found" errors with ValidatingAdmissionPolicy and MutatingAdmissionPolicy. These fixes improve admission reliability for newly-created namespaces.
  pull/135442, [pull/135443](https://github.com/kubernetes/kubernetes/pull/135443], pull/135444

• Storage and CSI Improvements: Pull requests clean up Dockerfiles for test volume images, reimplement storage API conformance tests, remove dead code related to node ID length, and fix volume expansion tests by using actual allocated storage sizes. These changes reduce image size, improve test maintainability, and prevent test failures due to storage size discrepancies.
  pull/135446, pull/135451, pull/135455, pull/135448

• Kubelet and Device Plugin Manager Fixes: A pull request addresses a race condition in the device plugin manager by uniquely identifying plugin endpoints to prevent incorrect removal during updates, improving resource availability and scheduling reliability.
  pull/135437

• New kubectl Debug Subcommand: A new kubectl debug lifecycle subcommand is introduced to diagnose pod lifecycle issues by analyzing pod phases, related events, and resources, providing root cause analysis with confidence scores and actionable recommendations in multiple output formats.
  pull/135448

• Scheduler Performance Metrics Enhancement: A pull request modifies scheduler metrics to include threshold values, enabling visibility of "Threshold" data points in performance dashboards for metrics with defined thresholds.
  pull/135457

3.2 Closed Pull Requests

This section provides a summary of pull requests that were closed in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Closed This Week: 21

Key Closed Pull Requests

1. [WIP] KEP-4671: Add Workload API. Copy of PR #134564: This pull request proposes adding a new Workload API based on KEP-4671, including the introduction of a WorkloadReference to the Pod specification, along with related feature gates, kubectl enhancements, and end-to-end tests, while excluding the gang scheduling implementation.

• URL: pull/135143

• Merged: No

• Associated Commits: 55554, 8a560, 24a10, e5de6, 6c84e, 310be, af49e

2. Fix: Check defaultBackend in allowRelaxedServiceNameValidation: This pull request fixes a bug in the allowRelaxedServiceNameValidation() function by adding validation for the spec.defaultBackend.service.name field to ensure consistent handling of relaxed service names in Ingress resources, thereby maintaining backward compatibility for existing Ingresses with service names that are RFC 1123-compliant but RFC 1035-noncompliant.

• URL: pull/135426

• Merged: 2025-11-28T08:00:22Z

• Associated Commits: ac90e, 950df, dae1d

3. KEP-3659: Add Helm ApplySet integration controller: This pull request implements Helm ApplySet integration for Kubernetes by adding a controller that watches Helm release Secrets to automatically create ApplySet metadata, aggregates Helm release health status with Kubernetes conditions and Prometheus metrics, introduces validation webhooks and new kubectl commands, and includes integration and end-to-end test scaffolding to enable seamless interoperability between Helm and kubectl through the ApplySet specification.

• URL: pull/135447

• Merged: No

• Associated Commits: 4488d, 18c1c, 8415e

Other Closed Pull Requests

• Test Flakiness and Reliability Improvements: Multiple pull requests address flaky tests by introducing retry logic, unique identifiers, and preventing resource conflicts to improve test stability. These changes include deflaking CSI Mock volume expansion quota validation, preventing cross-test flaking in pod certificate tests, and fixing a flaky DRA integration test by isolating device allocation.
• [pull/135131, pull/135445, pull/135445, pull/135440]

• Etcd Client and Component Enhancements: Pull requests propose adding DNS-based service discovery, gRPC health checking, and round-robin load balancing to the etcd client for better high availability, along with updating the etcd component to version 3.6.6. These improvements maintain backward compatibility and rebuild the component accordingly.
• [pull/135047, pull/135347]

• Pod Management Features and Fixes: Several pull requests enhance pod management by adding new kubectl flags to filter pods by stale conditions or idle activity, fixing admission failure handling in kubelet, and correcting ResourceQuota scope semantics. These changes improve pod lifecycle tracking, admission error handling, and quota accuracy.
• [pull/135479, pull/135431, pull/135414, pull/135461]

• Test Additions and Updates: Some pull requests add or update tests related to CRI-O and cloud-controller-manager versions, as well as include test commits without detailed descriptions. These updates aim to ensure compatibility and coverage for new versions and components.
• [pull/135121, pull/135397, pull/135436]

• Documentation Improvements: One pull request adds comprehensive documentation for x-kubernetes OpenAPI extensions, providing detailed explanations and usage examples consistent with existing OpenAPI specification styles.
• [pull/135235]

• Build and Image Optimization: A pull request optimizes container image builds by skipping weak dependencies and documentation files and cleaning up cache directories, reducing image size and improving build efficiency without affecting functionality.
• [pull/135342]

• Feature Removal and Cleanup: Two pull requests remove deprecated features and related flaky tests, specifically the SeparateCacheWatchRPC feature and the TestWatchStreamSeparation test, to stabilize the codebase after deprecation.
• [pull/135420, pull/135421]

• Concurrency and Thread Safety Fixes: A pull request adds a read lock to the GetIndexers function to ensure thread-safe access and prevent concurrency issues.
• [pull/135440]

3.3 Pull Request Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed pull requests that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed pull requests from the past week.

---

IV. Contributors

4.1 Contributors

Active Contributors:

We consider an active contributor in this project to be any contributor who has made at least 1 commit, opened at least 1 issue, created at least 1 pull request, or made more than 2 comments in the last month.

If there are more than 10 active contributors, the list is truncated to the top 10 based on contribution metrics for better clarity.

Contributor	Commits	Pull Requests	Issues	Comments
pohly	25	3	14	40
carlory	34	12	1	24
macsko	28	2	1	18
aojea	4	4	1	34
liggitt	12	3	0	26
bwsalmon	37	2	1	0
serathius	2	2	0	30
harche	10	8	5	10
BenTheElder	11	2	0	18
HirazawaUi	5	0	2	21