Weekly GitHub Report for Xla

Thank you for subscribing to our weekly newsletter! Each week, we deliver a comprehensive summary of your GitHub project's latest activity right to your inbox, including an overview of your project's issues, pull requests, contributors, and commit activity.

Table of Contents

• I. News
  • 1.1. Recent Version Releases
  • 1.2. Other Noteworthy Updates
• II. Issues
  • 2.1. Top 5 Active Issues
  • 2.2. Top 5 Stale Issues
  • 2.3. Open Issues
  • 2.4. Closed Issues
  • 2.5. Issue Discussion Insights

• III. Pull Requests
  • 3.1. Open Pull Requests
  • 3.2. Closed Pull Requests
  • 3.3. Pull Request Discussion Insights

• IV. Contributors
  • 4.1. Contributors

I. News

1.1 Recent Version Releases:

No recent version releases were found.

1.2 Version Information:

Please provide the version release information you would like me to analyze and summarize.

II. Issues

2.1 Top 5 Active Issues:

We consider active issues to be issues that that have been commented on most frequently within the last week. Bot comments are omitted.

1. [NVIDIA-GPU] [GPU] [STAT:AWAITING RESPONSE FROM CONTRIBUTOR] [ERR:PERFORMANCE] Delay kernel timer timeout on Blackwell (SM 12.0a) — hardcoded cycle count assumes 2GHz: This issue addresses a problem with the delay kernel timer in the CUDA code for Blackwell GPUs, where a hardcoded cycle count assumes a 2GHz clock, causing timeouts on RTX 5090 GPUs running at higher clock speeds around 3.09GHz. The user suggests that the timeout value should be scaled according to the actual SM clock frequency or increased to accommodate faster GPUs to prevent sub-optimal timing accuracy warnings during tests.
   • The comments discuss a related pull request aimed at fixing recent delay kernel errors, with suggestions to include it in the next JAX release, and a request for the original reporter to confirm if the issue is resolved so it can be closed.
   • Number of comments this week: 1

Since there were fewer than 5 open issues, all of the open issues have been listed above.

2.2 Top 5 Stale Issues:

We consider stale issues to be issues that has had no activity within the last 30 days. The team should work together to get these issues resolved and closed as soon as possible.

As of our latest update, there are no stale issues for the project this week.

2.3 Open Issues

This section lists, groups, and then summarizes issues that were created within the last week in the repository.

Issues Opened This Week: 4

Summarized Issues:

• XLA compilation errors due to type and operation mismatches: Multiple issues describe failures in XLA compilation caused by type mismatches and unsupported operations during compile-time evaluation. These errors include a type mismatch between float32 and complex64 in a binary multiply operation and an InvalidArgumentError due to unsupported dynamic inferencing in a custom call during QR decomposition used with tf.sequence_mask.
• issues/45837, issues/45839

• Backend crashes and memory management regressions: There are critical backend issues including a segmentation fault in the ROCm backend caused by exceeding shared memory limits, and a regression in jaxlib 0.7.2 where donated buffers are not updated in place, leading to increased peak memory usage. These problems result in compiler crashes and inefficient memory usage on both CPU and GPU.
• issues/45897, issues/45912

2.4 Closed Issues

This section lists, groups, and then summarizes issues that were closed within the last week in the repository. This section also links the associated pull requests if applicable.

Issues Closed This Week: 2

Summarized Issues:

• Backend Float Handling Issues: The XLA CPU backend had a problem where specific round decimal float values in the protobuf configuration were incorrectly retrieved as zero, causing failures in further processing. This issue was resolved by removing a workaround and improving float handling to ensure correct value retrieval.
• issues/9010

• GPU Backend Address Space Handling: The XLA GPU backend required improvements in handling LLVM to SPIR-V address spaces, focusing on the correctness and safety of kernel argument rewriting. These changes aimed to better align with Khronos SPIR-V conventions and Intel TritonGEN's memory space mappings for safer and more accurate backend processing.
• issues/45184

2.5 Issue Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed issues that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed issues from the past week.

III. Pull Requests

3.1 Open Pull Requests

This section provides a summary of pull requests that were opened in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Opened This Week: 22

Key Open Pull Requests

1. [WIP] Expose NCCL Collective Resources from FFI: This pull request introduces a new feature that exposes NCCL collective resources through an external FFI extension API, enabling XLA to manage collective resources for custom kernels by refining the FFI ABI, adding runtime metadata derivation, and supporting host resolution for collective peer addresses.

• URL: pull/45772

• Associated Commits: f33cc, e9518, a758b, aa034, 0e8e3, fb0e0, 4ac93

2. [XLA:GPU] Add SchedulerMemoryFencing pass to bound LHS peak memory: This pull request introduces a new SchedulerMemoryFencing pass to the XLA GPU latency-hiding scheduler pipeline that adds control dependencies to bound the peak memory usage by preventing the scheduler from deferring the last users of large buffers across many async collective windows, thereby addressing out-of-memory issues caused by excessive buffer live ranges during transformer training workloads.

• URL: pull/45929

• Associated Commits: 24470, 95d7c, 949d9, ed9a7, 36482

3. [XLA:GPU][oneAPI/SPIR-V] Skip compilation and loading of empty constants modules: This pull request improves the XLA GPU backend by skipping the compilation and loading of empty constants LLVM modules across all GPU platforms, including oneAPI/SPIR-V, thereby avoiding unnecessary backend compilation and module loading steps for device-constant-free workloads and enhancing platform-independent efficiency.

• URL: pull/45877

• Associated Commits: 03d75, 1fdbf, e2c7c, b7446

Other Open Pull Requests

• GPU codebase refactoring and fusion improvements: Multiple pull requests refactor and improve GPU fusion utilities and rewriters to enhance code maintainability and performance. These include extracting common cuDNN fusion utilities for reuse, fixing DSFv2 rewriter bugs to remove duplicated hero operations, and adding a unit test for GEMM rewriter transpose folding to ensure correctness in operand handling.
  • pull/45707, pull/45799, pull/45822

• Race condition and error handling fixes in execution paths: Several pull requests address race conditions and error propagation issues in multi-device and two-phase launch execution paths. They ensure proper failure detection before proceeding with launches and prevent crashes by adding error handling similar to existing failure paths.
  • pull/45683, pull/45682

• GPU plugin and client configuration enhancements: Pull requests add new configuration options and support for GPU plugins and clients, including enabling max_inflight_computations in PJRT C API GPU plugin and adding SYCL support for PJRT_Client_DmaMap on Intel GPUs. These changes improve configurability and leverage new hardware capabilities for better performance.
  • pull/45684, pull/45954

• GPU lowering and optimization improvements: Updates to AMD TDM lowering and GPU slice-fusion logic improve performance and correctness by handling singleton dimensions properly and avoiding expensive recomputation on shared non-elementwise operands. These changes enable valid TDM paths and restore inference speed regressions.
  • pull/45673, pull/45950

• HLO parser validation and robustness enhancements: Multiple pull requests add validation checks to the HLO parser for attributes like dimensions, operand counts, and shapes in various opcode handlers. These prevent crashes on malformed inputs and ensure proper error statuses are returned instead of exceptions or out-of-bounds reads.
  • pull/45821, pull/45871

• Sharding and compilation bug fixes: Fixes in the StableHLO to HLO import and GPU backend prevent compilation failures by dropping rank-mismatched tiled shardings and avoiding invalid scalar sharding inheritance. These changes resolve regressions in models like MaxText MoE and sharded MoE models.
  • pull/45717, pull/45816

• ROCm and ROCm-related build and runtime support: Enhancements include passing ROCm-related HSA environment variables through lit tests, adding support for waves_per_eu in Triton custom calls for kernel occupancy tuning, and enabling transitive data file dependencies in wheel targets for hermetic ROCm builds. These ensure better ROCm CI job customization and runtime support.
  • pull/45735, pull/45757, pull/45767

• Allocator API simplification: The DeviceAddressVmmAllocator mapped Allocate() API is simplified by removing a rarely used mode and consolidating allocation record kinds. This streamlines the allocation process and removes unused code paths and tests, improving maintainability.
  • pull/45938

• Collective operation pipelining control: A new CollectivePipeliningMode is introduced to provide fine-grained control over pipelining of collective operations in GPU computations. This allows users to optimize loop state, buffering, and communication overhead by choosing between no pipelining, automatic pipelining, or explicit pipelining based on frontend attributes.
  • pull/45836

3.2 Closed Pull Requests

This section provides a summary of pull requests that were closed in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Closed This Week: 34

Key Closed Pull Requests

1. [XLA:GPU] Support VA remapping of parameter buffer allocations: This pull request introduces support for virtual address (VA) remapping of parameter buffer allocations in the XLA GPU backend, enabling stable reservation addresses for entry-computation-parameter buffers by aliasing caller-owned buffers into VA reservations during execution, with mechanisms to record, unmap, and rewrite these aliases to maintain consistent addresses across executions and ensure proper resource handling without exposing reservation addresses.

• URL: pull/45595

• Associated Commits: 09e4b, a0de7, 9e7c7, 7ec80, b4db6, 4d5aa

• Associated Commits: 09e4b, a0de7, 9e7c7, 7ec80, b4db6, 4d5aa

2. [XLA:GPU] Trigger command buffers update when persistent allocations set changed.: This pull request addresses a correctness fix in the XLA GPU backend by ensuring that command buffers are properly updated when the set of persistent allocation indices transitions from absent to present during execution steps, including adding logic to trigger full command-buffer updates upon this transition and recover from mismatches in allocation policy validity to prevent stale addresses in command buffers.

• URL: pull/45130

• Associated Commits: 789a0, 1926c, 3a40f, 6be84, 3aa52

• Associated Commits: 789a0, 1926c, 3a40f, 6be84, 3aa52

3. [XLA:SE] Consolidate VMM allocator map resolution: This pull request refactors the VMM allocator map resolution by consolidating multiple map-related operations into a single, streamlined function to simplify the lock-protected lifecycle and reduce duplicated control flow without changing the public API or behavior.

• URL: pull/45420

• Associated Commits: 42c81, f69ff, 5a8c0

• Associated Commits: 42c81, f69ff, 5a8c0

Other Closed Pull Requests

• VMM Allocator Simplification and Consolidation: Multiple pull requests improve the VMM allocator by simplifying state management with sequence numbers, consolidating reservation data, and unifying indexing. These changes reduce synchronization complexity and improve maintainability without altering behavior.
• [pull/45421, pull/45422]

• GPU Backend Virtual Address Remapping Enhancements: Several pull requests introduce and refactor virtual address remapping in the XLA GPU backend, including a new SKIP_PROFILED mode to optimize command-buffer updates and a dedicated subclass for remapping logic. These updates enable stable reservation addresses for output buffers and improve code structure while maintaining functionality.
• [pull/45568, pull/45567, pull/45596]

• XLA GPU Latency-Hiding Scheduler Memory Limit Debug Feature: A pull request adds an optional debug feature that enforces a soft-capped memory limit in the GPU latency-hiding scheduler, preventing node selection that would exceed memory limits when alternatives exist. This reduces excessive memory consumption and avoids out-of-memory errors on large graphs while ensuring scheduling progress.
• [pull/45753]

• oneDNN Asynchronous Execution and ThreadPool Fixes: Pull requests update the default oneDNN asynchronous execution engine for x86_64 Linux and Windows and fix a bug in the OneDnnThreadPool's parallel_for by capping worker threads to the number of work items. These changes restore intended parallelism and fix CPU performance regressions.
• [pull/41991, pull/44403]

• Sanitizer Wrapper and LLVM Symbolizer Integration: A pull request integrates the llvm symbolizer from a hermetic LLVM toolchain into the sanitizer_wrapper to support asan/tsan pipelines properly. It also refactors scripts to remove dependencies on ignore_list files.
• [pull/45486]

• Removal of Obsolete Attributes and Code Cleanup: One pull request removes the obsolete allocation_granularity attribute from the GpuExecutableBufferAllocator, which remained after a prior cleanup.
• [pull/45475]

• HLO Text Parser Robustness Improvements: A pull request modifies the HLO text parser to return parse errors instead of aborting on malformed replica_groups=mesh[...] collective-device lists, preventing crashes and unbounded memory allocations by validating inputs before allocation.
• [pull/45559]

• Unit Test Additions for GPU Backend Operations: Pull requests add unit tests for folding transpose operands into dot operations and for ROCm rocBLAS complex GEMM fallback covering multiple transpose combinations and batched cases. These improve test coverage and correctness verification.
• [pull/45563, pull/45669]

• Rules_ml_toolchain and JAX CI Build Updates: Pull requests remove the pinned override of the rules_ml_toolchain version after JAX migration and unify ROCm CI testing scripts by consolidating multiple scripts into one. Another PR attempts to revert a previous JAX CI checkout change to fix new issues.
• [pull/45582, pull/45599, pull/45608]

• SYCL Platform Support and Test Skips: Pull requests add support for consistent canonical platform naming to fix JAX runtime errors on SYCL, skip Grouped GEMM tests on SYCL due to lack of support, and skip Mosaic GPU and AddI32 PTX kernel tests on the oneAPI backend because they are unsupported.
• [pull/45613, pull/45607, pull/45687]

• NCCL Communicator API and Logging Updates: A pull request adds a new API to xla::Communicator for waiting on multiple peers simultaneously using a single NCCL call and updates all VLOG statements to XLA_VLOG_DEVICE in the nccl_communicator.cc file.
• [pull/45629]

• oneDNN Build Fix for Test Binary Linking: A pull request updates the oneDNN BUILD file to exclude planner_main.cpp from the main library target to prevent test binaries from linking oneDNN's main() function, fixing test failures.
• [pull/45638]

• Triton Codegen Restriction for FP4 Values: A pull request restricts generic Triton code generation for f4E2M1FN FP4 values to storage-preserving operations only, rejecting generic FP4 arithmetic and fusion outputs before lowering to prevent incorrect codegen and fix test regressions.
• [pull/45646]

• oneAPI Backend Stream Limits and Pointer Memory Space: Pull requests remove the limit on the number of streams per device for the oneAPI backend to align with other GPU backends and propose pointer memory space functionality to identify pointer locations (device, host, or shared memory), including enabling related unit tests.
• [pull/45691, pull/45692]

3.3 Pull Request Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed pull requests that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed pull requests from the past week.

IV. Contributors

4.1 Contributors

Active Contributors:

We consider an active contributor in this project to be any contributor who has made at least 1 commit, opened at least 1 issue, created at least 1 pull request, or made more than 2 comments in the last month.

If there are more than 10 active contributors, the list is truncated to the top 10 based on contribution metrics for better clarity.