Weekly GitHub Report for Xla

Thank you for subscribing to our weekly newsletter! Each week, we deliver a comprehensive summary of your GitHub project's latest activity right to your inbox, including an overview of your project's issues, pull requests, contributors, and commit activity.

Table of Contents

• I. News
  • 1.1. Recent Version Releases
  • 1.2. Other Noteworthy Updates
• II. Issues
  • 2.1. Top 5 Active Issues
  • 2.2. Top 5 Stale Issues
  • 2.3. Open Issues
  • 2.4. Closed Issues
  • 2.5. Issue Discussion Insights

• III. Pull Requests
  • 3.1. Open Pull Requests
  • 3.2. Closed Pull Requests
  • 3.3. Pull Request Discussion Insights

• IV. Contributors
  • 4.1. Contributors

I. News

1.1 Recent Version Releases:

No recent version releases were found.

1.2 Version Information:

Please provide the version release information you would like me to analyze and summarize.

II. Issues

2.1 Top 5 Active Issues:

We consider active issues to be issues that that have been commented on most frequently within the last week. Bot comments are omitted.

1. [XLA:GPU] CollectiveMemoryCache can cause rank-divergent NCCL window registration: This issue describes a deadlock problem in the XLA GPU backend where the CollectiveMemoryCache can cause rank-divergent NCCL window registration due to inconsistent cache hit/miss results across ranks during concurrent executions. The root cause is that buffer allocation happens before acquiring the exclusive clique lock, leading to different runtime addresses and thus some ranks performing collective registration while others skip it, resulting in deadlock.

   • The comments discuss the challenges of fixing this issue, noting that exclusive locks on GPU cliques serialize execution but do not prevent address divergence because buffer allocation precedes lock acquisition. Suggestions include reordering execution steps or introducing new locking mechanisms, but multi-host synchronization is considered prohibitively expensive, and some contributors view concurrent collective executions as unsupported or user errors.
   • Number of comments this week: 8

2. [NVIDIA-GPU] [ERR: RUNTIME] HLO verifier failure in multi-device FFT with x64 enabled: This issue describes a failure in the HLO verifier when running a multi-device FFT operation with 64-bit floating point precision enabled in JAX, resulting in a type mismatch error between complex64 and complex128 during a multiply operation. The user suspects the problem arises from hardcoded complex64 constants in the XLA FFT handler code that do not adapt to the input data type, causing the verifier to reject the computation when multiple devices and higher precision are used.

   • The comment confirms the issue can be reproduced on the CPU backend with multiple devices and x64 enabled, showing the same type mismatch error; it also provides a minimal example and notes that the error does not occur with single-device or float32 inputs.
   • Number of comments this week: 1

Since there were fewer than 5 open issues, all of the open issues have been listed above.

2.2 Top 5 Stale Issues:

We consider stale issues to be issues that has had no activity within the last 30 days. The team should work together to get these issues resolved and closed as soon as possible.

As of our latest update, there are no stale issues for the project this week.

2.3 Open Issues

This section lists, groups, and then summarizes issues that were created within the last week in the repository.

Issues Opened This Week: 3

Summarized Issues:

• Collective Operations Optimization: This topic covers efforts to improve the efficiency of collective operations in distributed systems by enabling overlapping of independent collective operations with compute instructions, which aims to reduce network-bound pipeline bubbles. It also addresses synchronization issues in collective memory caching that can cause deadlocks due to inconsistent cache states across ranks during concurrent executions.
• issues/45315, issues/45406

• Thread Binding Failures on SLURM Clusters: This topic involves problems with thread binding in XLA on SLURM clusters using standard cgroups, where attempts to bind threads with hwloc_set_cpubind() fail with invalid argument errors. The issue results in misleading error-level logs despite normal GPU execution and suggests reducing log severity or silently handling the failure to avoid confusion.
• issues/45484

2.4 Closed Issues

This section lists, groups, and then summarizes issues that were closed within the last week in the repository. This section also links the associated pull requests if applicable.

Issues Closed This Week: 4

Summarized Issues:

• Compilation Failures on ARM64 NEON with GCC: The xla::ConvertF64ToEf57 implementation fails to compile on ARM64 NEON when using GCC, an issue introduced in a specific commit and partially addressed for Clang but not resolved for GCC. This results in persistent compilation errors that affect ARM64 builds using GCC.
• issues/8247

• GPU Memory Over-allocation and OOM Errors in Newer JAX Versions: Newer JAX versions cause unexpectedly large GPU memory allocations and out-of-memory errors on NVIDIA GPUs during certain XLA computations, despite similar code running successfully on older versions. This is likely due to changes in fusion and optimization logic that increase memory usage without improving runtime efficiency.
• issues/17124

• SPMD Partitioning and Type Mismatch Errors in Dynamic Update Slice: The dynamic_update_slice operation fails when used with sharding and double precision due to type mismatches between 64-bit constants and 32-bit partition IDs during SPMD partitioning, causing LLVM compilation errors. A workaround involves explicit type conversions to ensure consistent integer types in comparisons, highlighting a bug in type handling during partitioning.
• issues/24186

• Incorrect Operation Ordering in SPMD Partitioner Causing Gradient Leakage: The SPMD partitioner incorrectly orders all-reduce and dynamic-slice operations in the backward pass of jnp.take (scatter-add), leading to gradients leaking to unrelated vocabulary rows. This happens because mismatched shards are summed before slicing, causing incorrect gradient aggregation across tensor parallel devices.
• issues/42481

2.5 Issue Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed issues that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed issues from the past week.

III. Pull Requests

3.1 Open Pull Requests

This section provides a summary of pull requests that were opened in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Opened This Week: 24

Key Open Pull Requests

1. [XLA:GPU] Add CuTeDSL FFI v3 runtime integration: This pull request integrates the CuTeDSL FFI v3 runtime into the XLA GPU backend by linking public CuTeDSL runtime entry points through a Bazel-selectable provider, implementing a strict ten-attribute collective v3 schema with lifecycle management for collective operations, supporting both shared and static runtime linkage, enforcing safety boundaries in execution, and validating the integration through extensive testing across multiple configurations and hardware setups.

• URL: pull/45512

• Associated Commits: b20b9, 18dd4, 48f1a, 0280f, 04121, fa5dd, 36a71, 95562, 56032, e6b0f

2. [XLA:SE] Consolidate VMM allocator map resolution: This pull request refactors the VMM allocator map resolution by consolidating multiple map-related operations into a single, streamlined function to simplify the lock-protected lifecycle and reduce duplicated control flow without changing any public API behavior.

• URL: pull/45420

• Associated Commits: 42c81, f69ff, 5a8c0

3. [XLA:SE] Simplify VMM allocator record and pending state: This pull request simplifies the VMM allocator by consolidating the active and stale states into a single sequence-number-based identity within allocation records, unifying reservation record indexing, and reducing synchronization complexity to improve state management without changing existing behavior.

• URL: pull/45421

• Associated Commits: eb774, c278f, 7d610

Other Open Pull Requests

• DeviceAddressVmmAllocator refactoring and simplification: Multiple pull requests improve the DeviceAddressVmmAllocator by consolidating mapped allocation reuse and creation paths, simplifying allocation bookkeeping, and removing redundant helpers and branches. These changes preserve existing behavior while enhancing code clarity, maintenance, and accounting for physical address budgets.
  • pull/45422, pull/45323

• GPU output buffer and command buffer allocation management: Several pull requests refactor GPU output buffer management and command buffer allocation by consolidating output donation validation, copy protection, live-output tracking, and virtual-address remapping logic into dedicated allocators. These changes unify output-buffer policies, separate address-policy-specific behavior, and improve code structure without altering GPU program generation or execution schedules.
  • pull/45478, pull/45567

• SKIP_PROFILED command buffer update mode for XLA GPU: Two pull requests introduce and extend the SKIP_PROFILED command buffer update mode that profiles allocation-address stability over multiple executions to enable virtual address remapping of stable allocations. This reduces unnecessary address updates in repeated GPU workloads while maintaining fallback behavior for unstable or unsupported allocations, including protocol changes and comprehensive tests.
  • pull/45568, pull/45597

• Virtual address remapping support for XLA GPU backend buffers: Multiple pull requests add support for virtual address remapping of parameter and output buffer allocations in the XLA GPU backend, enabling stable reservation addresses by aliasing caller-owned or physical allocations during execution. These changes include new allocation methods, remapping logic, hooks, and testing infrastructure to ensure consistent address resolution and proper alias release.
  • pull/45582, pull/45595

• LLVM symbolizer integration and sanitizer_wrapper improvements: One pull request integrates the llvm symbolizer from a hermetic LLVM toolchain into the sanitizer_wrapper within rules_ml_toolchain to support asan/tsan pipelines properly. It also refactors related scripts to remove dependencies on external ignore_list files, improving pipeline robustness.
  • pull/45486

• cuDNN convolution plan probing without GPU executor: A pull request adds deviceless cuDNN convolution plan probing APIs to the CUDA stream-executor, enabling heuristic probing of cuDNN convolution graphs without requiring a GPU executor or cuDNN handle. This facilitates compile-time convolution fallback decisions and improves support for configurations that previously caused failures on certain GPUs.
  • pull/45534

• Fixes and improvements in HLO text parser error handling: Several pull requests fix out-of-bounds bugs and improve error handling in the HLO text parser by returning parse errors instead of aborting or causing memory corruption when encountering malformed attributes or collective-device lists. These changes validate axis references, indices, and device ID counts to prevent crashes and unbounded memory usage.
  • pull/45559, pull/45560

• Algebraic simplifier correctness improvements: One pull request fixes unsound algebraic simplifier folds for inverse-function identities by gating them behind the fast-math option to preserve correct IEEE 754 semantics and gradients. This prevents incorrect constant folding that caused discrepancies between eager and JIT-compiled executions.
  • pull/45560

• Enhancements to autotuner backends and thread-safety: A pull request improves autotuner backends by enforcing const-correctness, enhancing thread-safety with per-call rewriter pipelines and scratch cost analysis, and optimizing resource management by borrowing MLIRContext instances from a pool in the Triton and Fission backends.
  • pull/45514

• Fix data race in FissionBackend rewriter pipeline: One pull request fixes a data race by replacing a shared HloPassPipeline instance with a factory that creates a fresh pipeline on each call, preventing concurrent mutation of non-atomic internal state. Related backend factories and tests are updated accordingly.
  • pull/45334

• PJRT GPU custom call extension enhancements: A pull request enhances the PJRT GPU custom call extension by adding a traits field to the registration arguments and updating the extension version. This allows passing XLA FFI handler traits dynamically at registration time, enabling handlers to be marked as command buffer compatible rather than fixed per shared library.
  • pull/45377

• Removal of obsolete allocation_granularity attribute: One pull request removes the obsolete allocation_granularity attribute from the GpuExecutableBufferAllocator, which remained after a prior cleanup due to allocator forking.
  • pull/45475

• HLO text parser improvements for malformed collective-device lists: A pull request modifies the HLO text parser to return parse errors instead of aborting on malformed replica_groups=mesh[...] collective-device lists. It validates axis references, indices, and device ID counts before allocation to prevent crashes and unbounded memory usage.
  • pull/45559

• Unit test addition for transpose folding in dot operations: One pull request adds a unit test verifying that the function CanFoldTransposeOperandIntoDot correctly folds a transpose on the right-hand side operand of a dot operation. This fills a gap in test coverage that previously included only positive left-hand side cases and negative right-hand side cases.
  • pull/45563

• Removal of pinned rules_ml_toolchain version in CI build: A pull request removes the pinned override of the rules_ml_toolchain version from the bzlmod CI build for JAX, as it is no longer necessary after JAX migrated to the latest rules_ml_toolchain.
  • pull/45582

• Unification of ROCm CI testing scripts: One pull request unifies ROCm continuous integration testing scripts by redirecting calls from run_xla.sh and run_multigpu.sh to a single script, run_xla_ci_build.sh, streamlining the unit test execution process.
  • pull/45599

3.2 Closed Pull Requests

This section provides a summary of pull requests that were closed in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Closed This Week: 50

Key Closed Pull Requests

1. [DO NOT MERGE] Test pr clean up ignore list: This pull request is a cleanup effort focused on updating and managing the ignore list within the ROCm continuous integration workflow configuration files, as indicated by multiple commits modifying the rocm_ci.yml and related CI build URLs, but it was not merged.

• URL: pull/43432

• Associated Commits: 71169, fe6ca, a48c2, e7e35, fa436, 93f39, 45772, 13db2, 0a35d, 67373, 486ca, 90b6e, 87170, d8e32, 4a163, f4dec, e5669, d999c, 62c82, 3458f, b79bf, a1c09, 1275d, 72471, 9522d, d9a72

• Associated Commits: 71169, fe6ca, a48c2, e7e35, fa436, 93f39, 45772, 13db2, 0a35d, 67373, 486ca, 90b6e, 87170, d8e32, 4a163, f4dec, e5669, d999c, 62c82, 3458f, b79bf, a1c09, 1275d, 72471, 9522d, d9a72

2. [ROCm] Download rocm redist inside xla: This pull request proposes moving the download and management of the ROCm redistributable libraries from the external rules_ml_toolchain into the XLA project itself to simplify updates, synchronize sysroot versions with the used distros, and improve initialization by injecting the extracted directories directly into the ROCm toolchain.

• URL: pull/44825

• Associated Commits: 576e9, 7cdba, ca256, 34ee8, 39990, 4dee5, b5996, aea2e, 21100, d355c, 91508, 10c29, 41e5d, fa925, 59462, 963be, 712ad, 5c9aa, c079c, ac5ef, bbffc, 225fd, 17b6c, 2356e, b9746, 36b94

• Associated Commits: 576e9, 7cdba, ca256, 34ee8, 39990, 4dee5, b5996, aea2e, 21100, d355c, 91508, 10c29, 41e5d, fa925, 59462, 963be, 712ad, 5c9aa, c079c, ac5ef, bbffc, 225fd, 17b6c, 2356e, b9746, 36b94

3. [XLA:GPU] Add VMM-persistent command buffer temp allocations: This pull request proposes a refinement of the GPU command buffer allocation policy by introducing VMM-persistent temporary buffer allocations with two update modes—ALWAYS_UPDATE and SKIP_TEMP—to make virtual address remapping more conservative and stable for temporary buffers, simplifying allocator state and improving maintainability while preserving normal parameter and output semantics.

• URL: pull/45062

• Associated Commits: ede7c, 5cce8, 61218, 61bb7, f5904, e9f2c, d7bab, acfd7, d1bb5, 2ac03, a1a71, 28342, e9b8b, 3f1b1

• Associated Commits: ede7c, 5cce8, 61218, 61bb7, f5904, e9f2c, d7bab, acfd7, d1bb5, 2ac03, a1a71, 28342, e9b8b, 3f1b1

Other Closed Pull Requests

• ROCm-specific XLA GPU VMM optimizations: These pull requests introduce ROCm-specific improvements to the XLA GPU VMM command-buffer, including a skip-remap mechanism that caches and only remaps changed virtual address slots and a copy-into-shadow strategy for small, frequently changing slices. They also refactor allocation ownership and state management, improve budget accounting, and simplify map resolution to enhance allocator correctness and maintainability without changing performance.
• [pull/45070, pull/45258, pull/45419, pull/45414]

• Collective operations and ROCm backend enhancements: Multiple pull requests enable and improve collective tests for the ROCm backend, add support for dynamic roots in CollectiveBroadcast, introduce an all_gather module optimized for AMD/ROCm, and fix bugs in ROCm AllReduce kernels related to non-uniform replica groups. These changes improve test coverage, flexibility, and stability of collective operations on ROCm hardware.
• [pull/44641, pull/44145, pull/44559, pull/45332, pull/45398]

• XLA GPU backend kernel and address-space handling: These pull requests harden SPIR-V kernel argument address-space rewriting by restricting rewrites to default LLVM address space pointers and add regression tests for mixed kernel signatures. They also add backend-specific kernel annotations and simplify noalias test checks to fix GPU backend filecheck errors.
• [pull/45282, pull/45480]

• Command-buffer profiling and remapping improvements: An experimental command-buffer update mode profiles allocation addresses to identify persistent allocations, enabling stable virtual address remapping and skipping redundant updates. Additionally, the GPU command buffer allocation system is refactored to isolate VA remapping logic into a dedicated subclass, improving code organization without functional changes.
• [pull/45366, pull/45566]

• Bug fixes for ROCm and hipBLASLt GEMM operations: These pull requests fix bugs in ROCm GEMM rewriter visitors to detect unsupported epilogues with f64 types, prevent GPU page faults on gfx950 hardware by guarding against non-uniform replica groups in AllReduce, and refactor operand handling in hipBLASLt MX backend to correctly lower Triton GEMM fusions with scaled-dot operations.
• [pull/45327, pull/45332, pull/45507]

• Clang-format tooling improvements: Pull requests restructure the clang-format check process to allow local runs, add a --fix mode to automate formatting corrections, and unify GitHub workflow configurations for clang-format, enhancing developer workflow and code consistency.
• [pull/45275, pull/45278, pull/45274]

• ROCm device and library updates: These pull requests address concurrency issues in the ROCm SMI library by serializing access during device initialization and update the rocm_device_libs to a newer commit while removing build workarounds to align with the standalone CMake build.
• [pull/44992, pull/45273]

• Miscellaneous bug fixes and test coverage improvements: Additional pull requests fix an out-of-bounds read bug in AllReduceSplitter's ReplicaGroups comparator, resolve a crash in GatherSimplifier by rewriting scalar gathers, add tests for zero-sized HLO buffers in DUS fusion, and add native bf16 log support on AMD gfx1250 hardware.
• [pull/45398, pull/45401, pull/45302, pull/45280]

• CUDA profiling path disablement: One pull request disables the CUDA 13.2+ CUPTI V2 multi-subscriber profiling path and its unit tests by default due to compatibility issues causing profiling failures on CUDA 13.3.
• [pull/44853]

3.3 Pull Request Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed pull requests that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed pull requests from the past week.

IV. Contributors

4.1 Contributors

Active Contributors:

We consider an active contributor in this project to be any contributor who has made at least 1 commit, opened at least 1 issue, created at least 1 pull request, or made more than 2 comments in the last month.

If there are more than 10 active contributors, the list is truncated to the top 10 based on contribution metrics for better clarity.