Weekly GitHub Report for Kubernetes

Thank you for subscribing to our weekly newsletter! Each week, we deliver a comprehensive summary of your GitHub project's latest activity right to your inbox, including an overview of your project's issues, pull requests, contributors, and commit activity.

Table of Contents

• I. News
  • 1.1. Recent Version Releases
  • 1.2. Other Noteworthy Updates
• II. Issues
  • 2.1. Top 5 Active Issues
  • 2.2. Top 5 Stale Issues
  • 2.3. Open Issues
  • 2.4. Closed Issues
  • 2.5. Issue Discussion Insights

• III. Pull Requests
  • 3.1. Open Pull Requests
  • 3.2. Closed Pull Requests
  • 3.3. Pull Request Discussion Insights

• IV. Contributors
  • 4.1. Contributors

I. News

1.1 Recent Version Releases:

The current version of this repository is v1.32.3

1.2 Version Information:

The Kubernetes version released on March 11, 2025, introduces key updates detailed in the official CHANGELOG, with additional binary downloads available. For comprehensive information on new features and changes, users are encouraged to refer to the Kubernetes announce forum and the linked CHANGELOG.

II. Issues

2.1 Top 5 Active Issues:

We consider active issues to be issues that that have been commented on most frequently within the last week. Bot comments are omitted.

1. [KIND/BUG] [PRIORITY/CRITICAL-URGENT] [SIG/NODE] [TRIAGE/ACCEPTED] CPU Manager AllocatePod re-allocates restored pod CPUs after kubelet restart: This issue describes a problem where the static CPU Manager in Kubernetes is not idempotent across kubelet restarts when using pod-scope Topology Manager and PodLevelResourceManagers, causing restored pod CPU assignments to be overwritten and leading to pod admission failures or CPU leaks. The expected behavior is that after a kubelet restart, the CPU Manager should preserve the restored pod CPU sets and not reallocate CPUs, ensuring consistent and reliable pod CPU assignments.

   • The comments include acknowledgments of the bug, assignment and triage of the issue, verification of the problem with tests, submission of a fix via a pull request, and discussions about communication and prioritization, ultimately escalating the priority to critical-urgent to ensure the fix is included before the next release.
   • Number of comments this week: 11

2. [SIG/SCHEDULING] [KIND/FEATURE] [NEEDS-TRIAGE] [Feature][Scheduler] PodGroup/CompositePodGroup scheduling failure diagnosis is insufficient for debugging: This issue addresses the insufficient and misleading diagnostic information provided when PodGroup or CompositePodGroup gang scheduling fails, which hampers effective debugging by users. It highlights problems such as loss of individual pod failure details during group-level status propagation, lack of aggregated failure summaries at the PodGroup level, and the absence of condition updates for CompositePodGroups, proposing a layered diagnosis mechanism to improve clarity and context across Pod, PodGroup, and CompositePodGroup levels.

   • The comments discuss agreement on the root causes and the need for improvements, with contributors expressing interest in working on different parts of the fix, including message enhancements and condition patching; ongoing work on CompositePodGroup conditions is noted, and concerns about balancing message detail with size and complexity are raised.
   • Number of comments this week: 8

3. [SIG/NODE] [KIND/FLAKE] [PRIORITY/IMPORTANT-LONGTERM] [TRIAGE/ACCEPTED] ci-kubernetes-node-arm64-ubuntu-serial periodically doesn't come up: This issue describes a recurring problem where the ci-kubernetes-node-arm64-ubuntu-serial job periodically fails to start the cluster, with failures traced to three distinct root causes: GCE VM stockouts in a single zone, a dominant test suite hang causing a 3-hour timeout kill without logs, and a known arch-independent test flake. The main actionable item is to fix the timeout misalignment so that the test suite can self-terminate and upload logs before the external kill, improving diagnosability, while also addressing the VM stockout by enabling multi-zone fallback.

   • The comments include triage acceptance and prioritization, a detailed analysis breaking down the failures into three modes, recommendations for timeout and zone fallback fixes, and volunteers expressing intent to investigate further by reviewing logs and test history.
   • Number of comments this week: 7

4. [KIND/BUG] [PRIORITY/IMPORTANT-SOON] [SIG/NODE] [TRIAGE/ACCEPTED] In-place container memory resize does not work when pod is in status OOMKilled: This issue describes a problem where in-place container memory resizing does not resolve an OOMKilled pod status, causing the pod to remain in a crash loop despite having increased memory resources. The user reports that while the container's cgroup memory is updated correctly, the pod-level cgroup memory is not, leading to the failure to recover, and this behavior is reproducible across multiple Kubernetes versions and environments including AWS EKS and K3d.

   • The comments include requests for regression testing on earlier Kubernetes versions, confirmation that the issue is not a regression, sharing of reproducer reports on other platforms, assignment of the issue to a developer, discussions about the severity and priority of the bug, and plans for backporting the fix to multiple Kubernetes versions.
   • Number of comments this week: 6

5. [SIG/NODE] [NEEDS-TRIAGE] kubelet eviction manager: transient HasDedicatedImageFs failure permanently disables disk-pressure eviction signals: This issue describes a bug in the kubelet eviction manager where transient failures in the HasDedicatedImageFs check during initialization cause the eviction manager to permanently disable disk-pressure eviction signals by returning early and skipping critical evaluation steps. The reporter expects that such failures should not block memory and PID pressure evaluations or halt the entire synchronization loop, but rather skip disk signal evaluation for that cycle and retry in subsequent cycles to ensure proper eviction signal handling.

   • The comments include assignment of the issue, tagging it for the node SIG, a request and confirmation of AI usage disclosure, and a clarification on formatting and issue description updates to maintain clarity and compliance with project guidelines.
   • Number of comments this week: 5

2.2 Top 5 Stale Issues:

We consider stale issues to be issues that has had no activity within the last 30 days. The team should work together to get these issues resolved and closed as soon as possible.

As of our latest update, there are no stale issues for the project this week.

2.3 Open Issues

This section lists, groups, and then summarizes issues that were created within the last week in the repository.

Issues Opened This Week: 28

Summarized Issues:

• Probe cancellation and readiness during pod termination: Stopping a kubelet probe worker does not cancel ongoing exec probes, causing probes to continue running against terminating containers until timeout. Additionally, readiness probes lack end-to-end test coverage during pod graceful termination, risking incorrect Ready condition reporting after deletionTimestamp is set but before termination grace period ends.
• [issues/140977, issues/140978]

• Flaky and failing tests in various Kubernetes components: Multiple tests are flaky or failing intermittently, including a panic in apiserver/pkg/storage/cacher due to leaked watch goroutines, integration test failures in DRA podgroup scheduling, ci-kubernetes-e2e-capz-master-windows liveness probe restarts, and Windows-specific Security Context test failures due to Linux commands. These issues cause instability in CI and test reliability across different platforms and test suites.
• [issues/140979, issues/141072, issues/141073, issues/141085]

• Pod scheduling and resource allocation issues: DaemonSet pods can get stuck Pending due to lack of priority advantage, pod startup latency increases linked to FastFillPodStartup failures, pod scheduling failures due to resource claim conflicts, and in-place resizing of Guaranteed QoS pods is blocked on nodes using static CPU Manager policy. These problems affect pod lifecycle management and resource scheduling efficiency.
• [issues/140984, issues/141046, issues/141059, issues/141064]

• Kubelet and CPU/Memory Manager bugs after restarts: The CPU Manager incorrectly reallocates CPUs to restored pods after kubelet restarts, causing pod rejection or CPU leaks, while the Memory Manager fails to restore NUMA hints due to incorrect memory block comparisons, potentially leading to pod admission failures under certain topology policies. These bugs impact resource restoration and pod stability after node restarts.
• [issues/140989, issues/141069]

• Metrics and API server performance degradation: Unbounded cardinality in apiserver_request_terminations_total metric due to unvalidated request path labels causes excessive label growth, while high pod lifecycle event volumes during scale testing lead to API server throttling and etcd bloat, degrading control plane performance and increasing latency. These issues threaten cluster observability and scalability.
• [issues/141007, issues/141022]

• Gang scheduling diagnostics and StatefulSet update issues: Gang scheduling failures in PodGroup and CompositePodGroup lack sufficient diagnostic information, complicating debugging, while StatefulSet image updates after pod image pull errors do not trigger rolling updates unless pods are manually deleted, causing stuck error states. Both issues hinder effective workload management and troubleshooting.
• [issues/141025, issues/141095]

• Kubectl output error handling bugs: Errors from printer.PrintObj during commands like "kubectl rollout history" and "kubectl get" are silently ignored, causing commands to exit successfully despite output failures such as broken pipes. Proper error propagation is needed to ensure command reliability and user awareness of failures.
• [issues/141087, issues/141115]

• Resource allocation and eviction manager bugs: The ExtendedResourceCache serves stale mappings and drops winning DeviceClass due to improper ownership handling, while the kubelet eviction manager disables disk-pressure eviction signals after transient failures in HasDedicatedImageFs checks, blocking critical eviction logic and pressure evaluations. These bugs affect resource correctness and node health management.
• [issues/141103, issues/141119]

• ResourceSlice allocation inconsistency: The GatherPools function incorrectly determines pool generation completeness based on input ResourceSlice order when counts disagree, causing inconsistent allocation behavior depending on input permutation. This leads to unpredictable resource allocation outcomes.
• [issues/141117]

• Flaky LoadBalancer tests with ExternalTrafficPolicy Local: LoadBalancer tests intermittently fail to change UDP service types and ports or to consistently target nodes with endpoints, causing timeouts and HTTP service reachability failures during slow master job runs. These flakes reduce confidence in LoadBalancer feature stability.
• [issues/141133, issues/141136, issues/141138]

• Dependency instability causing build failures: The cyphar.com/go-pathrs module dependency is unstable and occasionally causes build failures, prompting consideration for migration to more stable infrastructure to maintain Kubernetes project security and reliability.
• [issues/141144]

• Documentation quality issues: The hack/README.md file contains multiple writing quality problems such as weasel words, comma splices, wordiness, missing spaces, and passive voice, requiring corrections for improved clarity and readability.
• [issues/141121]

2.4 Closed Issues

This section lists, groups, and then summarizes issues that were closed within the last week in the repository. This section also links the associated pull requests if applicable.

Issues Closed This Week: 35

Summarized Issues:

• Mount Security Options on emptyDir Volumes: Kubernetes does not allow specifying mount options like nosuid, nodev, and noexec on emptyDir volumes with medium: Memory, resulting in tmpfs mounts lacking these security flags. This raises concerns about potential attack vectors from executable files in writable tmpfs volumes.
  • issues/48912

• Node Shutdown and Systemd Configuration Conflicts: Kubernetes graceful node shutdown settings are overridden by the unattended-upgrades package's systemd logind configuration, causing errors due to higher precedence of InhibitDelayMaxSec overrides. This prevents Kubernetes from applying longer shutdown grace periods as intended.
  • issues/102818

• CPU Manager Admission and Metrics Flakiness: Pods using static CPUManager policy with full-pcpus-only fail admission after kubelet restarts due to SMTAlignmentError, indicating bugs in CPU allocation handling. Additionally, CPU Manager Metrics tests are flaky on nodes with Linux Kernel 6.12.41+ and containerd 2.1.3, blocking related updates.
  • issues/129078, issues/134236

• Multi-Tenancy and IP Address Allocation: There is a need to make cluster-level concepts like ServiceCIDR tenant-friendly by enabling control over IP allocation from multiple CIDR ranges per namespace or tenant. Proposed mechanisms include annotations or policies to assign IPs predictably and isolate usage, though this adds architectural complexity.
  • issues/130792

• Scheduler Performance and Flaky Scheduling Tests: Optimizations are proposed for Kubernetes scheduler inter-pod affinity/anti-affinity to isolate host-scoped rules and improve performance via caching and localized filtering. Flaky integration tests related to PodGroup gang scheduling and scheduler preemption also cause intermittent failures due to race conditions and context cancellations.
  • issues/138314, issues/139417, issues/141027

• Container Creation and Readiness Probe Regressions: Kubernetes v1.34+ regression causes pods to fail starting due to strict UTF-8 validation on environment variables from secrets, rejecting containers without clear error messages. Another regression in v1.35.0 causes exec readiness probes to fail silently during pod termination, leaving pods incorrectly marked as Ready.
  • issues/139132, issues/140881

• Flaky and Failing Tests in Various Environments: Multiple flaky and failing tests affect Kubernetes CI, including Windows node CPUManager tests failing due to Azure image changes, containerd alpha feature tests failing from incorrect feature gate settings, and flaky tests in publishing-bot smoke tests, EC2 conformance, init container startup order, and node conformance liveness probes.
  • issues/140777, issues/140813, issues/140859, issues/140270, issues/140272, issues/140943

• Kube-proxy Crashes and Networking Issues: Kube-proxy running in nftables mode on AWS EKS with Bottlerocket OS crashes with segmentation faults during sync, breaking Service DNAT/routing and causing all Service traffic to fail on affected nodes. Additionally, nftables sync failures occur due to assertion errors with IPv6 DNAT rules.
  • issues/140898, issues/140951

• Test Infrastructure and Compilation Failures: Compilation failures occur due to unupdated test doubles after interface changes, and concurrency race conditions in AWS deployer cause panics preventing conformance tests. Other infrastructure flakes include containerd service startup failures on ubuntu-2404 EC2 instances and master instance creation failures from bad COS image releases.
  • issues/140965, issues/140987, issues/141013, issues/141080

• Pod and PVC Scheduling Issues: PVCs and PVs are not cleaned up when nodes go offline, causing pods to be scheduled onto offline nodes and remain pending. StatefulSet pods also fail to enter running state due to context deadline exceeded errors during e2e testing.
  • issues/141076, issues/140910

• Admission Plugin Compatibility and API Changes: Restoration of the admissioninitializer.WantsAuthorizer interface in v1.37 is required to prevent compile errors and runtime failures in admission plugins relying on the previous method, ensuring backward compatibility after a mechanical rename.
  • issues/140915

• Windows-Specific Test Failures: Windows end-to-end tests fail due to hostnameOverride fields not working as expected and DNS resolution timing issues causing mTLS connection failures in Projected PodCertificate tests, leading to deterministic and intermittent test failures on Windows and Linux respectively.
  • issues/140992, issues/140993

• Conformance Test Algorithm Support Mismatch: The ServiceAccountIssuerDiscovery conformance test fails because it only allows certain token signing algorithms (RS256, ES256), while the cluster supports additional algorithms like ES384, requiring test updates to align with actual supported algorithms.
  • issues/141010

• Code Formatting and Documentation Issues: The gofmt tool in Go 1.26.5 produces different outputs on amd64 and arm64 architectures causing verify job failures, and punctuation errors in hack/README.md reduce documentation readability.
  • issues/141074, issues/141120

• kubectl Patch Error Messaging Improvements: The error message from kubectl patch when a strategic merge patch on a list field is missing the required merge key lacks actionable context. A proposal suggests clarifying which field requires the merge key, explaining its necessity, and providing examples to help users fix the issue.
  • issues/141112

• Test Job Failures and Job Removal Considerations: Alpha EC2 node end-to-end test jobs are permanently failing with no recent failure history, raising questions about whether these jobs should be removed due to persistent startup failures.
  • issues/141038

• Unit Test Failures Due to Upstream Changes: The TestCelCostStability unit test fails because of changes in runtime cost calculations in cel-go v0.30.0, which lowered expected costs due to lazy evaluation modifications, requiring upstream patches or pinned value adjustments.
  • issues/141056

2.5 Issue Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed issues that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed issues from the past week.

III. Pull Requests

3.1 Open Pull Requests

This section provides a summary of pull requests that were opened in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Opened This Week: 108

Key Open Pull Requests

1. Extending integration tests for WAP to match ones for pod preemption: This pull request extends the integration tests for pod group preemption in the WAP component by porting tests from pod-by-pod preemption, restructuring the podgrouppreemption_test.go file for improved readability, and adding several new test cases to ensure comprehensive coverage.

• URL: pull/141139

• Associated Commits: d0bde, 2440a, f5d52, caf2d, 86d58, 59c92, 0457f

2. Make MoreImportantVictim deterministic for equal start times: This pull request makes the MoreImportantVictim comparator deterministic for pods with equal start times by adding a tie-breaker based on pod UID, ensuring a total order that prevents flaky test failures caused by non-deterministic victim ordering on platforms with low-resolution timers, while preserving existing priority and workload semantics.

• URL: pull/140999

• Associated Commits: df58f, d843e, 00fe4

3. Fix CPU Manager re-allocations for pods using pod-level resources after kubelet restarts: This pull request fixes the kubelet CPU manager to correctly restore pod-level exclusive CPU allocations from checkpointed state after a kubelet restart, preventing pods from being rejected or silently moved to different CPUs, thereby avoiding CPU leaks by restoring existing assignments as-is and only allocating missing ones while maintaining consistent CPU accounting and topology hints.

• URL: pull/141041

• Associated Commits: d5bb9, ab26d, 4777b

Other Open Pull Requests

• Kubelet probe worker fixes for pod shutdown: These pull requests fix a regression in kubelet where exec readiness probes failed with "context canceled" during pod graceful termination. They detach probe workers from the pod sync context to ensure the pod's Ready condition updates correctly during shutdown.
  [pull/141005, pull/141006]

• End-to-end test error handling cleanup: Multiple pull requests replace the anti-patterns ExpectNoError and Expect(err).NotTo(HaveOccurred()) in various end-to-end tests with versions that include explanatory error messages. This improves test clarity and prevents loss of error context across storage, network, node, apps, api, miscellaneous, and framework tests.
  [pull/140964, pull/140969, pull/140970, pull/140971, pull/140972, pull/140973, pull/140974]

• CompositePodGroup scheduling improvements: These pull requests enhance CompositePodGroup scheduling by making MinGroupCount mutable across API versions and adding a scheduler plugin queueing hint for dynamic resizing. They also improve PreEnqueue phase performance by introducing incremental caching and refactoring readiness update logic to reduce complexity and increase throughput.
  [pull/141023, pull/141030]

• Kubelet and pod lifecycle enhancements: Pull requests add comprehensive unit tests for container lifecycle methods, fix pod stop signal handling to respect shorter grace periods, and optimize HTTP probing by caching requests to reduce overhead. These changes improve container lifecycle correctness and kubelet efficiency.
  [pull/141019, pull/141040, pull/141053]

• ExtendedResourceCache collision handling fix: This pull request fixes the ExtendedResourceCache to handle collisions between DeviceClass objects by storing all candidates per resource and recomputing the winning DeviceClass after changes. This prevents stale mappings and ensures correct resource promotion.
  [pull/141081]

• kubectl impersonation resource resolution fix: This pull request fixes an issue where impersonation of users and groups was incorrectly resolved through the RESTMapper, causing conflicts. It returns non-discoverable impersonation resources directly to ensure correct API group usage in SelfSubjectAccessReview requests.
  [pull/141110]

• TestValidateEvictionStatusUpdate timestamp determinism: These pull requests make eviction status update timestamps deterministic on Windows by using distinct fake-clock start times. This prevents flaky test behavior caused by timestamp equality due to Windows clock resolution.
  [pull/141051, pull/140981]

• AbstractPodGroup unification in scheduler queue: This pull request unifies handling of PodGroup and CompositePodGroup objects by introducing an AbstractPodGroup type that abstracts both, simplifying scheduling queue and workload forest management.
  [pull/141081]

• Code refactoring and dependency update: One pull request refactors code by adding missing nil checks, improving test helpers, and removing redundant functions. Another updates the github.com/google/cel-go dependency to v0.30.0, adjusting pinned test values to reflect runtime cost changes and ensuring compatibility.
  [pull/141052, pull/141058]

• PlacementCycleState clearing and function rename: This pull request ensures PlacementCycleState is cleared for pods transitioning to the binding phase to maintain consistency with PodGroupCycleState. It also renames the SetPodGroupCycleState function to better reflect its purpose.
  [pull/140986]

• verify-test-code.sh script update: This pull request updates the verify-test-code.sh script to prevent the use of the ExpectNoError anti-pattern without explanation, improving test code quality and requiring coordination with related PRs to avoid CI failures.
  [pull/140975]

• Readiness probe coverage for pod termination: This pull request adds and tightens end-to-end tests to assert readiness probes correctly run and cause pods to become NotReady during graceful termination, addressing a coverage gap that allowed regressions where pods remained Ready until container termination.
  [pull/140976]

• etcd3 watcher fatalOnDecodeError scoping: This pull request scopes the fatalOnDecodeError flag to individual watchers by snapshotting its value at construction, preventing leaked watch goroutines from causing panics in unrelated tests due to global state changes during cleanup.
  [pull/140980]

3.2 Closed Pull Requests

This section provides a summary of pull requests that were closed in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Closed This Week: 92

Key Closed Pull Requests

1. Opportunistic batching rescore: This pull request implements the rescoring path for opportunistic batching as described in KEP-5598, providing an overview of the complete flow to facilitate understanding and review of subsequent smaller PRs that incrementally build this feature into the Kubernetes scheduler.

• URL: pull/140289

• Associated Commits: 16ead, 3d5da, a51f2, 38c29, 4ab14, 429b6, 6b901, 99fbb, cdae7, 078d5, 80e1a, 6cc52, a6923, 93f80

2. test: add conformance coverage for EvictionRequest: This pull request adds comprehensive end-to-end conformance test coverage for the EvictionRequest API in Kubernetes, verifying all key resource operations such as create, get, list, watch, patch, update, status updates, and deletion, while also updating test data to ensure lifecycle API discovery compliance.

• URL: pull/140897

• Associated Commits: 3ce50, 6d7d6, 3d1f3, 2710e, 11d31, 9d830, 45ee7, 19d41, c3866

3. fix(nestedpendingoperations): Wait only blocks while pending operations exist: This pull request addresses a deadlock issue in the Wait() function of the nestedpendingoperations component by modifying it to block only while there are actually pending operations, thereby preventing indefinite blocking caused by leftover non-pending entries after errors with exponential backoff enabled.

• URL: pull/140985

• Associated Commits: a8ca7, d7b45, 7361d, d7bab, ec291, 16375, cc500, d883b

Other Closed Pull Requests

3.3 Pull Request Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed pull requests that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed pull requests from the past week.

IV. Contributors

4.1 Contributors

Active Contributors:

We consider an active contributor in this project to be any contributor who has made at least 1 commit, opened at least 1 issue, created at least 1 pull request, or made more than 2 comments in the last month.

If there are more than 10 active contributors, the list is truncated to the top 10 based on contribution metrics for better clarity.

Access Last Week's Newsletter:

• Link