Weekly GitHub Report for Xla

Thank you for subscribing to our weekly newsletter! Each week, we deliver a comprehensive summary of your GitHub project's latest activity right to your inbox, including an overview of your project's issues, pull requests, contributors, and commit activity.

Table of Contents

• I. News
  • 1.1. Recent Version Releases
  • 1.2. Other Noteworthy Updates
• II. Issues
  • 2.1. Top 5 Active Issues
  • 2.2. Top 5 Stale Issues
  • 2.3. Open Issues
  • 2.4. Closed Issues
  • 2.5. Issue Discussion Insights

• III. Pull Requests
  • 3.1. Open Pull Requests
  • 3.2. Closed Pull Requests
  • 3.3. Pull Request Discussion Insights

• IV. Contributors
  • 4.1. Contributors

I. News

1.1 Recent Version Releases:

No recent version releases were found.

1.2 Version Information:

Please provide the version release information you would like me to analyze and summarize.

II. Issues

2.1 Top 5 Active Issues:

We consider active issues to be issues that that have been commented on most frequently within the last week. Bot comments are omitted.

1. [CPU] [ERR: RUNTIME] [STAT:AWAITING RESPONSE FROM CONTRIBUTOR] XLA CPU: jit_compile=True causes a segmentation fault for TruncateMod → Where → Cumprod: This issue describes a segmentation fault occurring when using XLA's JIT compilation on CPU for a TensorFlow graph involving the operations TruncateMod, Where, and Cumprod, which runs successfully in eager mode but crashes when compiled. The fault is traced to a bug in the SliceToDynamic kernel related to incorrect size inference and tiling in reduce-window operations with dynamic dimensions, causing out-of-bounds memory access.

   • The comments include a contributor expressing interest in working on the issue and asking about the process for picking up issues, a detailed explanation of the root cause and proposed fixes by another contributor, and guidance encouraging new contributors to self-assign and comment when taking on issues.
   • Number of comments this week: 4

2. [STAT:AWAITING OPENXLA-ENG] GroupDisjointReductions materializes O(V*R) reachability sets and derives a non-canonical group order: This issue discusses inefficiencies and non-determinism in the GroupDisjointReductions function within the XLA GPU backend, which currently materializes large reachability sets leading to quadratic complexity and produces a non-canonical group order dependent on pointer-based hash iteration. The user proposes a revised approach to reduce traversal count and storage while ensuring deterministic group ordering by selecting a canonical representative, but a correction clarifies that some set materialization is still necessary due to filtering logic, and a model quantifies the rarity of the ordering variance in practice.

   • The comments refine the proposed optimization by acknowledging the necessity of full reachable sets for correctness, present a probabilistic model showing the ordering variance occurs in about 0.4% of cases, and note a draft change to enforce deterministic group ordering is underway, with reviewers assigned to ensure deterministic compile results.
   • Number of comments this week: 4

3. [AMD-GPU] [ERR:BUILD] ROCm PJRT plugin build failure: This issue reports a build failure when compiling the ROCm PJRT plugin on Ubuntu 24.04 using ROCm 7.14, with errors related to missing or undeclared identifiers in the ROCm codebase and invalid attribute arguments during compilation. The user provides detailed environment setup and build commands, highlighting that the problem may also occur with earlier ROCm versions, and includes a lengthy error log showing multiple compilation errors.

   • The comments discuss the availability and validity of ROCm versions, noting that versions before 7.14 may no longer install, and one commenter confirms successful building on a specific ROCm branch with a simplified target architecture, suggesting that building for all AMDGPU targets may be unnecessary and slower.
   • Number of comments this week: 2

4. [CPU] [GPU] [STAT:AWAITING RESPONSE FROM CONTRIBUTOR] [ERR:PERFORMANCE] NaNs from bitwise_xor & atan2 after compilation: This issue reports that after JAX compilation, the function involving bitwise_xor and atan2 produces NaN values on CPU and GPU, whereas the eager execution and TPU do not exhibit this problem. The root cause is identified as an LLVM optimization that incorrectly folds atan2(0, 0) to NaN, which contradicts the IEEE 754 standard where atan2(+0, +0) should be +0.

   • The comment explains that the NaN arises during compile-time constant folding in LLVM, not in XLA, and a fix has been proposed on the XLA side to handle the algebraic simplification correctly, while the LLVM issue still needs an upstream fix.
   • Number of comments this week: 1

5. [STAT:AWAITING RESPONSE FROM CONTRIBUTOR] [ERR: RUNTIME] [XLA] Heap corruption (free(): invalid pointer) during JIT compilation when running WhereOp with float16: This issue describes a heap corruption error occurring during JIT compilation with XLA when running the WhereOp on float16 inputs on CPU, which leads to a crash with invalid pointer free or segmentation fault. The problem arises because XLA disables runtime shape checks on CPU, causing a dynamic size to exceed its buffer bound during a logical_and operation, resulting in out-of-bounds memory writes and subsequent crashes.

   • The comment explains that the root cause is unchecked dynamic size bounds in the XLA compilation pipeline, specifically in the SliceToDynamic kernel, and proposes a fix that clamps dynamic sizes to their valid bounds to prevent memory corruption, with future work planned to add proper runtime shape checks.
   • Number of comments this week: 1

2.2 Top 5 Stale Issues:

We consider stale issues to be issues that has had no activity within the last 30 days. The team should work together to get these issues resolved and closed as soon as possible.

As of our latest update, there are no stale issues for the project this week.

2.3 Open Issues

This section lists, groups, and then summarizes issues that were created within the last week in the repository.

Issues Opened This Week: 7

Summarized Issues:

• Performance regressions and inefficiencies in XLA backend: Several issues highlight performance problems in the XLA backend, including a regression caused by slice operations being excluded from YNNPACK fusion, leading to inefficient kernel execution. Additionally, the GroupDisjointReductions function suffers from quadratic complexity and non-deterministic group ordering, causing inconsistent reduction assignments and inefficiencies.
• issues/46351, issues/46537

• Timeouts and long-running GPU tests on NVIDIA hardware: Multiple GPU tests on NVIDIA B200 and GB200 devices experience severe timeout issues, with tests like collective_pipeline_parallelism_test_gb200, buffer_comparator_test_b200, and topk_test_b200 exceeding the 900-second timeout limit. These prolonged runtimes cause CI failures and indicate a need for optimization or test sharding to meet time constraints.
• issues/46626, issues/46629, issues/46630

• Build and test environment failures on ROCm and CUDA platforms: There are build failures when compiling the ROCm PJRT plugin on Ubuntu 24.04 with ROCm 7.14 due to missing identifiers and invalid attributes. Additionally, CUDA GPU LIT tests fail hermetically because the --xla_gpu_cuda_data_dir path does not expose necessary binaries like ptxas or nvlink, preventing PTX compilation on NVIDIA B200 and GB200 platforms.
• issues/46610, issues/46627

2.4 Closed Issues

This section lists, groups, and then summarizes issues that were closed within the last week in the repository. This section also links the associated pull requests if applicable.

Issues Closed This Week: 6

Summarized Issues:

• Numerical correctness and overflow handling in algebraic simplification: The algebraic simplifier in XLA incorrectly folds inverse function pairs like log(exp(x)) and sqrt(x²) without considering IEEE 754 float32 intermediate overflow or underflow, leading to discrepancies between XLA-compiled and eager execution results. This causes finite values to be produced instead of infinities or NaNs when intermediate computations overflow or underflow.
• issues/42732

• Type mismatches and verifier failures in multi-device FFT with x64: Running jax.numpy.fft.fft on multiple devices with x64 enabled triggers an HLO verifier failure due to a type mismatch between complex64 and complex128 in a multiply operation during multi-device FFT execution. This issue specifically occurs with float64 inputs and three or more dimensions, causing the verifier to fail.
• issues/44541

• Unsupported operations on complex128 causing evaluator crashes: The XLA HloEvaluator crashes fatally when performing a less-than comparison on complex128 tensors during constant folding, as it aborts instead of returning a proper error for unsupported partial ordering operations on complex numbers. This results in an unexpected termination rather than graceful error handling.
• issues/44936

• Compiler crashes due to resource exhaustion in ROCm backend: The ROCm backend compiler experiences a segmentation fault when compiling a Pallas/Triton kernel that exceeds the shared-memory-per-block limit, caused by a null function pointer call in CompileTritonToLLVM. Instead of returning a proper resource exhaustion error, the compiler crashes unexpectedly.
• issues/45897

• Memory regression from buffer donation changes in jaxlib 0.7.2: A regression introduced in jaxlib 0.7.2 causes donated buffers to no longer be updated in place during loops that read intermediate values, leading to significantly increased peak memory usage. This happens due to extra temporary buffer allocations on both CPU and GPU backends.
• issues/45912

• Performance degradation from serialized FFT operations on CPU: FFT operations on the CPU are serialized to a single thread when preceded by another operation because of an overly aggressive thread pool check in XLA. The issue was fixed by modifying the EigenThreadPool to restrict parallelism only if the thread is already inside a parallel region, improving performance.
• issues/46345

2.5 Issue Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed issues that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed issues from the past week.

III. Pull Requests

3.1 Open Pull Requests

This section provides a summary of pull requests that were opened in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Opened This Week: 29

Key Open Pull Requests

1. [WIP][ROCm] Use therock 7.14 as a main platform: This pull request updates the ROCm continuous integration setup to use the hermetic therock 7.14 platform as the main environment for PR checks, including changes to build configurations, runtime libraries, and pipeline targets to ensure consistent and reliable testing.

• URL: pull/46510

• Associated Commits: 2a008, 2a0b2, 607b6, b6ad6, 514a0, fb416, 34b3a, d2b63, 12a64

2. rocm sol latency estimator: This pull request enables the SOL latency estimator for ROCm platforms, specifically ungating it for gfx942 and gfx950 architectures while ensuring graceful degradation when specifications are unavailable, and includes updates to matmul and collective interpolator data as well as the matmul interpolator table.

• URL: pull/46575

• Associated Commits: 082af, 336bb, 4fb75, 106eb, b2c78

3. [ROCm] Fix gpu_collectives_test: This pull request fixes several failing subtests in the gpu_collectives_test on the ROCm platform by skipping unsupported tests on non-CUDA platforms, adding missing platform communication support, implementing registered memory support for RCCL, and enabling the use of multiple clique IDs.

• URL: pull/46650

• Associated Commits: af874, 044db, 58b1b, f6a94, d93af

Other Open Pull Requests

• Bug fixes in GPU backend and buffer management: Several pull requests address bugs in GPU backend behavior and buffer assignment. These include fixing canonical ordering of reduction groups for deterministic GPU launch grid indices, updating cross-color temporary-buffer reuse logic to prevent unsafe overlaps, and restoring ROCm multi-GPU execution behavior by reverting improper memory space handling.
  • pull/46539, pull/46543, pull/46445

• Improvements and fixes in XLA GPU collective and communication handling: Multiple pull requests enhance GPU collective operations and communication domains. They add a typed collective communication domain for better stream scheduling, restructure synchronous collective handling to simplify thunk emission, and implement multi-process eager initialization for MORI GPUcollectives backend to improve memory allocation and initialization.
  • pull/46541, pull/46463, pull/46636

• Enhancements and fixes in XLA GPU backend for NVPTX and CUDA toolchains: These pull requests add support for new GPU targets and improve compliance with CUDA toolchains. They introduce SM107 NVPTX target support, enable correct PTX version selection for CUDA 13 toolchains, and fix float32 division mode to ensure IEEE-754 compliance on GPUs.
  • pull/46646, pull/46647, pull/46400

• Fixes and improvements in XLA CPU backend and dynamic dimension handling: Several pull requests fix crashes and incorrect results related to dynamic dimensions and memory safety in the CPU backend. They clamp dynamic dimension sizes to static bounds, correct output size calculations for scan-like reduce-window operations, and prevent unsound nonnegative inference for signed integer operations.
  • pull/46625, pull/46618, pull/46606

• Algebraic simplifications and correctness improvements in XLA: One pull request introduces an algebraic simplification folding xor(a, a) to zero, handling side-effect-free operands and trivial xor-with-zero identities. This prevents incorrect NaN results from MLIR’s atan2 constant folding and ensures IEEE 754-compliant behavior on CPU and GPU.
  • pull/46594

• Performance improvements and fusion enhancements: Pull requests improve performance by enabling fusion of slice operations into YNNPACK kernels and introducing a fused split-K kernel for narrow dot products on GPUs. These changes significantly reduce execution time and improve memory bandwidth utilization for specific workloads.
  • pull/46353, pull/46645

• Code quality and developer tooling improvements: One pull request adds a pre-commit configuration to run clang-format, clang-tidy, and buildifier checks locally, improving developer efficiency by enabling optional git hooks for automated code quality checks. Another fixes .clang-format to correctly sort C system headers before C++ headers.
  • pull/46417, pull/46358

• Fixes for device description and parameter handling: Pull requests fix the device description string for the A100 PCIe 80GB to match JAX device_kind and implement conditional wrapping of fusion parameters into replica-id pointer tables based on data type. These changes fix incorrect promotions and GPU memory-access faults.
  • pull/46339, pull/46352

• Security and safety fixes in memory handling: A pull request fixes an integer overflow vulnerability in the IFRT proxy's byte-strides size check by replacing unchecked multiplication with overflow-checked multiplication, preventing out-of-bounds reads and heap buffer overflows in gRPC handlers.
  • pull/46604

• Validation and parsing robustness improvements: One pull request adds validation in the HLO text parser to ensure consistent iota tile assignment dimensions, preventing silent parsing of inconsistent tile assignments and improving error diagnosis for malformed input.
  • pull/46606

• GPU command buffer efficiency improvements: A pull request improves GPU command buffer efficiency by tracing fully-persistent traced commands directly into the parent command buffer using a new CUDA ≥ 12.3 API, eliminating unnecessary nested command buffers and preserving rank-consistency for collective operations.
  • pull/46394

• Optimization pass enhancements: One pull request introduces a new flag to bypass the profitability heuristic in the all_reduce_splitter optimization pass and extends support for splitting all-reduce operations involving multiple replica groups, enabling improved loop hoisting and optimization in complex training scenarios.
  • pull/46521

3.2 Closed Pull Requests

This section provides a summary of pull requests that were closed in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Closed This Week: 55

Key Closed Pull Requests

1. Add BF16 fallback for FP8 cuDNN convolutions without valid plans: This pull request implements a fallback mechanism that, when cuDNN autotuning fails to find valid FP8 convolution plans—particularly on NVIDIA RTX 5090 GPUs with certain channel/group configurations—automatically probes and uses BF16 convolution plans instead via an FP8-to-BF16 rewrite path, thereby preventing crashes in JAX by ensuring convolution operations remain executable and logging warnings whenever this fallback occurs.

• URL: pull/40021

• Associated Commits: 6253c, ee763, 37301, 84580, 9216b, c3e14, d38fc, 96405, 96d4c, c36d8

2. [XLA] Fix asymmetric copy elision in conditional branches: This pull request addresses the asymmetric copy elision issue in conditional branches within XLA by gating exclusive-branch cases on actual escape analysis rather than branch position, enabling symmetric copy elision that eliminates unnecessary copies regardless of branch order, and includes new parameterized tests to verify zero-copy behavior and improve performance in scan and conditional patterns.

• URL: pull/46314

• Associated Commits: d5bf8, e59ef, 1f70f, 40e0b, 3ac0e, 6830a, 3c8e6

3. [ROCM][NFC] Improving & making nvshmem tests platform-agnostic: This pull request aims to improve and make nvshmem tests platform-agnostic in preparation for integrating a shmem-like GPU library for ROCM, while also enhancing the nvshmem GPU collectives test to use more concise self-verifying HLOs and fixing the nvshmem sendrecv subtest by addressing the one-sided nature of nvshmem collectives.

• URL: pull/39788

• Associated Commits: d8c8a, 71aac, 9f150, 9dd07, 93834, f5365

Other Closed Pull Requests

• GPU Collective and Communication Enhancements: Multiple pull requests improve GPU collective operations and communication by introducing dynamic root collective broadcast thunk modifications, adding the GroupCollectivesByKey GPU HLO pass for grouped collective execution, foundational AMD MORI collectives backend integration for ROCm, and stage-dependent context decoding in the XLA FFI for collective communication contexts. These changes enable optimized collective execution, backend kernel fusion, and support for new ROCm collectives backends while improving asynchronous and collective operation handling in the GPU backend.
  • pull/44235, pull/46449, pull/46449, pull/46011, pull/46188, pull/46235

• Memory Management and Allocation Improvements: Several pull requests address memory management by adding a SchedulerMemoryFencing pass to limit buffer live ranges and reduce peak memory usage, re-enabling collective memory allocations in ROCm with pointer tracking, introducing batch deferred-deallocation in the DeviceAddressVmmAllocator to reduce GPU timeline write overhead, and adding a RedzoneAllocator wrapper in the ROCm autotuner to detect out-of-bounds writes. These enhancements improve memory usage efficiency, prevent out-of-memory errors, and increase robustness against memory corruption.
  • pull/45929, pull/46501, pull/44170, pull/46078

• Bug Fixes and Stability Improvements: Multiple pull requests fix critical bugs including preventing splitting of scaled softmax logits fusion on CPU to avoid NaNs, fixing a race condition in two-phase launch of CommonPjRtLoadedExecutable::Execute, correcting dynamic-slice offset reversal in GPU backend to fix gradient errors, and adding overflow detection in shape dimension product validation to prevent corrupted element counts. These fixes enhance numerical correctness, execution stability, and validation robustness across CPU and GPU backends.
  • pull/44738, pull/45683, pull/46231, pull/46403

• Toolchain and Build System Updates: Several pull requests update the ROCm and CUDA toolchains and build configurations by fixing the non-hermetic ROCm toolchain setup, adding support for libcurand.so.11 in Bazel for CUDA, enabling transitive data file dependencies for hermetic ROCm builds, and updating BANT_VERSION to fix dependency removal issues. These changes ensure proper toolchain configuration and compatibility with new CUDA and ROCm releases.
  • pull/46297, pull/46410, pull/45767, pull/46296

• Performance and Code Quality Enhancements: Pull requests improve performance and code quality by refactoring the ThunkEmitter in the GPU backend for clearer async execution management, improving the DotMerger to preserve minor dimension order and avoid performance regressions, and fixing unsound algebraic simplifications by gating inverse-function folds behind a fast-math option to maintain IEEE 754 semantics. These changes enhance maintainability, runtime efficiency, and numerical correctness.
  • pull/46188, pull/46013, pull/45560

• CUDA and cuDNN Support Enhancements: Pull requests add deviceless cuDNN convolution plan probing APIs to enable heuristic probing without GPU executors, and restore a temporary workaround for the Triton TMEM scale padding issue to maintain CUDA 12.8 compatibility on certain GPUs. These updates improve convolution support and maintain compatibility with recent CUDA versions.
  • pull/45534, pull/46159

3.3 Pull Request Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed pull requests that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed pull requests from the past week.

IV. Contributors

4.1 Contributors

Active Contributors:

We consider an active contributor in this project to be any contributor who has made at least 1 commit, opened at least 1 issue, created at least 1 pull request, or made more than 2 comments in the last month.

If there are more than 10 active contributors, the list is truncated to the top 10 based on contribution metrics for better clarity.