Weekly GitHub Report for Xla

Thank you for subscribing to our weekly newsletter! Each week, we deliver a comprehensive summary of your GitHub project's latest activity right to your inbox, including an overview of your project's issues, pull requests, contributors, and commit activity.

Table of Contents

• I. News
  • 1.1. Recent Version Releases
  • 1.2. Other Noteworthy Updates
• II. Issues
  • 2.1. Top 5 Active Issues
  • 2.2. Top 5 Stale Issues
  • 2.3. Open Issues
  • 2.4. Closed Issues
  • 2.5. Issue Discussion Insights

• III. Pull Requests
  • 3.1. Open Pull Requests
  • 3.2. Closed Pull Requests
  • 3.3. Pull Request Discussion Insights

• IV. Contributors
  • 4.1. Contributors

I. News

1.1 Recent Version Releases:

No recent version releases were found.

1.2 Version Information:

Please provide the version release information you would like me to analyze and summarize.

II. Issues

2.1 Top 5 Active Issues:

We consider active issues to be issues that that have been commented on most frequently within the last week. Bot comments are omitted.

1. [CPU] TfrtCpuClient fails to allocate int2/int4 buffer: This issue describes a failure in the TfrtCpuClient when attempting to allocate a 1D tensor of type int4 using BufferFromHostBuffer with byte_strides set to std::nullopt, resulting in a mismatch error between dimensions and input strides. The problem arises because the code path does not handle the absence of byte_strides correctly, causing allocation to fail unless explicit strides are provided.

   • The comments discuss the current lack of support for packed sub-byte buffers in XLA:CPU, which always unpacks data to byte granularity, and express interest in future support for true sub-byte types like int2/int4 and packed FP4/FP6 formats; updates indicate this feature is planned but likely months away, with ongoing community inquiries about timelines and implementation details.
   • Number of comments this week: 2

2. [BUG] [ERR:PERFORMANCE] Involuntary Full Rematerialization: This issue describes a problem where the XLA compiler performs an involuntary full rematerialization of a tensor due to conflicts in sharding annotations when using JAX's sharding constraints, leading to performance inefficiencies. The user reports that they had to implement custom resharding logic to avoid this problem, as the compiler cannot automatically handle the transition between certain sharding patterns without rematerializing the entire tensor.

   • The comments explain that the rematerialization is caused by sharding conflicts propagated from intermediate instructions, and a pull request has been made to improve logging to identify the specific Hlo Instruction causing the conflict. Additional debugging advice is provided to inspect the Hlo dump and sharding metadata, and an internal reproducer has been submitted for further investigation.
   • Number of comments this week: 1

3. [CPU] Copies supersede OptimizationBarrier: This issue discusses how the XLA compiler on the CPU backend unnecessarily inserts multiple copies when scheduling operations involving slicing and in-place updates, despite the use of lax.optimization_barrier which is expected to enforce a specific execution order and eliminate redundant copies. The reporter notes that enabling the --xla_cpu_copy_insertion_use_region_analysis=true flag removes these extra copies and questions why this flag, which enables a region analysis introduced years ago, is only now resolving the issue, especially since the core copy elimination logic has not significantly changed.

   • The comments provide an additional example where the flag does not fully resolve copy insertion issues on CPU but works correctly on GPU, and mention that an internal reproducer has been submitted for further investigation.
   • Number of comments this week: 1

4. [STAT:AWAITING OPENXLA-ENG] [CPU] Enabling CPU backend optimization reveals a possible inconsistency in computing float32 power differences: This issue reports a discrepancy in the computation of float32 power differences when enabling CPU backend optimization in JAX, where the difference of equal squares unexpectedly yields a small non-zero result due to mixed precision in fused multiply-add (FMA) operations. The problem arises specifically with integer exponents optimized via multiplication and FMA, while non-integer exponents do not exhibit this inconsistency, suggesting that disabling FMA in these cases might be necessary to maintain numerical correctness.

   • The comments clarify that integer powers are lowered to multiplication operations optimized with FMA, which can introduce subtle precision errors as demonstrated by a provided script modeling the issue; it is noted that while FMA generally improves accuracy, it can cause problems in specific cases like this, leading to a recommendation to avoid FMA for these computations to ensure correct results.
   • Number of comments this week: 1

5. [CPU] [ERR:PERFORMANCE] Pmap slower with new CPU runtime: This issue reports a performance regression where the pmap function is significantly slower with the new CPU runtime in recent versions of JAX, as demonstrated by a minimal example involving adaptive solvers used in diffrax. The user provides timing comparisons across different JAX versions and notes that disabling the new CPU thunk runtime currently mitigates the slowdown, hoping for a future fix.

   • The comment section contains a single response indicating that an internal reproducer has been submitted to help investigate the issue further.
   • Number of comments this week: 1

2.2 Top 5 Stale Issues:

We consider stale issues to be issues that has had no activity within the last 30 days. The team should work together to get these issues resolved and closed as soon as possible.

As of our latest update, there are no stale issues for the project this week.

2.3 Open Issues

This section lists, groups, and then summarizes issues that were created within the last week in the repository.

Issues Opened This Week: 5

Summarized Issues:

• Kernel Performance and Workload Distribution: This topic covers issues related to improving kernel throughput and balancing workload distribution to reduce idle time and inefficiencies. The problems include fixed assignment strategies causing skew and quantization issues, as well as redundant communication and computation in batch-sharded grouped convolutions that increase memory usage unnecessarily.
• [issues/47868, issues/47945]

• GPU Compilation and Compatibility: This topic involves a compilation failure on SM107 GPUs due to an incorrect PTX version selection, which causes the LLVM compiler to abort instead of handling the error gracefully. The incompatibility between the device requirements and the selected PTX version leads to build failures on CUDA 12.9.
• [issues/47982]

• Benchmark Input Variability: This topic addresses the issue of benchmark inputs being identical across runs because of a fixed random seed, which reduces input coverage and may hide input-sensitive behaviors. The proposed solution is to modify the seed by incorporating the run index to ensure distinct inputs while maintaining reproducibility.
• [issues/48047]

• Floating-Point Comparison and Output Correctness: This topic highlights a violation of the IEEE 754 TotalOrder standard in a custom CUDA kernel for the TopK operation, where raw floating-point comparisons cause corrupted output in the presence of -NaN values. The issue contrasts with the correct behavior of an alternative implementation that respects the TotalOrder standard.
• [issues/48119]

2.4 Closed Issues

This section lists, groups, and then summarizes issues that were closed within the last week in the repository. This section also links the associated pull requests if applicable.

Issues Closed This Week: 2

Summarized Issues:

• Formatter and Testing Inconsistencies: This issue highlights a discrepancy between open-source and Google internal formatter checks, specifically about the requirement for new lines in newly added files. This inconsistency leads to delays in merging because internal test failures are not visible externally.
• issues/30652

• HLO Transform and Sharding Validation Errors: Registering a pre-scheduler HLO transform that returns the input unchanged causes the post-transform HloVerifier to reject valid multi-slice sharding on a two-slice TPU compile topology. The rejection stems from a mismatch in device count validation between localized per-slice device assignments and global multi-slice shardings.
• issues/47778

2.5 Issue Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed issues that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed issues from the past week.

III. Pull Requests

3.1 Open Pull Requests

This section provides a summary of pull requests that were opened in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Opened This Week: 30

Key Open Pull Requests

1. [XLA:GPU] Add cuDNN non-GEMM pointwise fusion support via TensorIR engine: This pull request adds support for lowering non-GEMM HLO pointwise fusions to the cuDNN graph API using the dedicated memory-bound TensorIR engine, introduces a new compiler pass to identify and retag such fusions for cuDNN compilation, extends autotuner and compiler integration to handle these fusions, and provides a new flag to control this feature, along with comprehensive tests covering various operations and edge cases.

• URL: pull/47839

• Associated Commits: 2eb88, 40224, 8ab71, d371b

2. [XLA:GPU] Support MultiGpuBarrierKernel with more than 32 peers: This pull request enhances the MultiGpuBarrierKernel in the XLA GPU backend to support synchronization across more than 32 peers by increasing the maximum peer capacity to 128 while maintaining a fixed launch size of 32 threads to prevent race conditions, thereby fixing crashes occurring with over 32 GPUs.

• URL: pull/47889

• Associated Commits: 54aa7, 6557e

3. Add specialized interleave kernels for transposes with a tiny stride-1 dimension: This pull request adds specialized interleave and deinterleave microkernels optimized for transposes with a stride-1 dimension of size 3, such as RGB channels, enabling vectorized code generation that significantly improves performance on relevant shapes without regressions in the benchmark suite.

• URL: pull/47974

• Associated Commits: 3ad1a, c9fcf

Other Open Pull Requests

• Removal of obsolete CPU and fusion emission code: These pull requests remove legacy and obsolete code paths in the CPU IrEmitter and elemental emission routes by eliminating the parallel fork-join call emission and retiring the ParallelLoopEmitter, ElementalKernelEmitter, and ThunkEmitter dispatch paths. This simplifies the call handling logic and enforces routing through MLIR fusion emitters, converting fallback cases into error returns to fully retire old emission routes.
  • pull/47976, pull/48001

• Fixes for dynamic dimension and slicing bugs: Multiple pull requests address bugs related to dynamic dimension sizes and slicing operations by clamping out-of-bound sizes, initializing dynamic dimension sizes to their bounds, and fixing miscompiles in nested dynamic-slice and dynamic-update-slice fold operations. These changes prevent runtime aborts, uninitialized memory usage, and incorrect tensor slicing or updating, ensuring compliance with HLO specifications.
  • pull/47800, pull/47804, pull/47805, pull/48024

• SYCL and oneMKL utility additions: These pull requests add new utility files and helper functions for SYCL convolution operations using oneDNN and for invoking oneMKL calls. These additions lay groundwork for future support of oneMKL-based implementations such as FFT, solvers, and complex GEMMs on the SYCL platform.
  • pull/47813, pull/47829

• Improvements and fixes in GPU backend and kernel logic: Several pull requests improve GPU backend functionality by fixing tile propagation logic to accept negative but phase-aligned offsets, enhancing ragged all-to-all communication kernel load balancing, and fixing LLVM IR builder positioning to prevent compilation crashes. These changes improve correctness, performance, and stability of GPU kernels and tiling.
  • pull/47925, pull/47928, pull/48089

• ROCm and hardware support updates: These pull requests update ROCm test scripts to use the amd-smi command instead of deprecated rocm-smi, enable hwloc embed-mode renaming to fix PCI count issues on AWS EFA, and add a predicate to gate collective kernels on devices supporting peer-visible atomics to prevent hangs on gfx90a hardware. They also temporarily disable oneAPI command buffer conversion to improve test pass rates.
  • pull/47876, pull/47936, pull/48043, pull/47975

• Enhancements to XLA partitioning and fusion analysis: These pull requests improve the SPMD partitioner by enabling batch sharding for grouped convolutions to avoid redundant all-gathers and fix dynamic-slice fusion boundaries in the GPU backend to correctly handle collective memory analyses. These changes enhance performance and prevent allocation errors in fused collective operations.
  • pull/48069, pull/47992

• Bug fixes in compiler and backend code: These pull requests fix a compiler crash caused by multi-output fusion of scatter operations, resolve an ambiguous conversion error in ExpandIntegerPower on GCC, and improve while loop range analysis by enforcing constant induction-variable steps and adding regression assertions. These fixes improve compiler robustness and correctness.
  • pull/48091, pull/48107, pull/47846

• Miscellaneous updates and tests: This includes moving oneDNN tests out of restricted folders to run in public CI, adding HLO code for the Keras vision transformer model for benchmarking, and a test pull request for CPU isolation in ROCm CI.
  • pull/47989, pull/47978, pull/48043

3.2 Closed Pull Requests

This section provides a summary of pull requests that were closed in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Closed This Week: 23

Key Closed Pull Requests

1. [ROCm] Move CI to rocm 7.14 take two: This pull request updates the continuous integration (CI) setup to use ROCm version 7.14, including various fixes and improvements such as resolving LLVM symbol clashes, optimizing test job configurations, and enforcing dynamic linking for GPU and CPU test correctness to enhance the build and testing process for ROCm workloads.

• URL: pull/47522

• Associated Commits: 5127b, d5288, 83c58, a5389, 6179b, f905f, 3fa7e, 60a7e, 6e18d

2. [XLA:GPU][oneAPI] Add an HLO pass to resize matmul workspace size with oneDNN: This pull request introduces the SyclGemmWorkspacePass, an Intel-GPU-specific HLO pass that adjusts the matmul workspace size from the default cuBLASLt configuration to the optimal size required by oneDNN's matmul primitive scratchpad, improving efficiency for oneDNN-backed matrix multiplications.

• URL: pull/47696

• Associated Commits: c6ac5, 0c8b7, d8191

3. [XLA:GPU] Add GPU collectives FFI extension on the backend-agnostic API: This pull request implements a new GPU collectives FFI extension within the backend-agnostic API for XLA:GPU, enabling the reuse of XLA's native NCCL communication and clique management to facilitate host-side NCCL collectives via custom calls and laying groundwork for symmetric-memory registration.

• URL: pull/46911

• Associated Commits: c6c91, f597b

Other Closed Pull Requests

• ROCm and GPU Profiling Enhancements: Multiple pull requests improve ROCm and GPU support by enabling the SoL latency estimator for gfx942 GPUs with native FP8 support, enhancing the ROCm XLA Profiler with detailed VGPR and LDS kernel statistics, and implementing a listener for application-emitted ROCTX markers in the ROCm profiler timeline. These changes collectively advance profiling accuracy and GPU runtime integration for AMD hardware.
  [pull/47757, pull/47917, pull/47999, pull/48044]

• Collective Operations and Backend Support: Several pull requests focus on collective operations by registering the oneCCL collective backend for GPU and oneAPI platforms, extracting collective_rendezvous into a separate library for better runtime isolation, and improving ROCm runfiles directory selection for configuration flexibility. These updates enhance collective communication support and maintainability across platforms.
  [pull/47694, pull/47701, pull/48136]

• XLA CPU and Architecture-Specific Fixes: Pull requests address architecture-specific issues by adding support for __truncsfhf2 and __extendhfsf2 symbols on s390x to fix JIT symbol resolution failures, explicitly declaring direct dependencies in the xla/service/cpu directory to fix violations, and removing an unused BW pool cache from the ROCm DNN component. These changes improve CPU backend stability and codebase hygiene.
  [pull/46823, pull/47627, pull/47973]

• Shape and Function Behavior Corrections in XLA: Fixes to XLA operations include correcting the Digamma function's pole behavior to align with TensorFlow CPU kernels and resolving false shape-mismatch errors in several special functions by replacing strict shape equality checks with compatibility checks. These fixes ensure consistent and correct mathematical operation behavior.
  [pull/47754, pull/47755]

• Build System and Configuration Improvements: Updates to the build system include upgrading Bazel to version 9.2.0, fixing a broken hwloc system build by removing an unused with_numa_support condition, and ensuring proper linking to ROCm libraries with bzlmod and legacy_external_runfiles=false to eliminate CI workarounds. These changes enhance build reliability and configuration correctness.
  [pull/47820, pull/47858, pull/47999]

• XLA Compiler Pass and Custom Call Handling: A pull request fixes bugs in the TopkDecomposer, TopKSplitter, and TopkSpecializer compiler passes by skipping "TopK" custom calls without comparators, preventing fatal errors on interpreter and GPU backends and aligning behavior with the CPU backend. This ensures graceful handling of comparator-less custom calls during compilation.
  [pull/47874]

• Performance Model and Device Info Updates: Improvements to performance model tests include adding accurate AMD device descriptions and correcting compute capability assignments for MI210 and RTX A6000 devices, ensuring tests run with correct hardware parameters. This enhances the fidelity of performance modeling.
  [pull/48007]

• Code Cleanup and Test Management: One pull request separates lit tests that use only FileCheck by tagging them with a GPU tag to enable targeted runs during the rocm_cpu CI step, and another introduces an HLO module pass to remove integer add-zero operations with comprehensive testing and a development container wrapper. These efforts improve test organization and code optimization.
  [pull/47502, pull/47973]

3.3 Pull Request Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed pull requests that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed pull requests from the past week.

IV. Contributors

4.1 Contributors

Active Contributors:

We consider an active contributor in this project to be any contributor who has made at least 1 commit, opened at least 1 issue, created at least 1 pull request, or made more than 2 comments in the last month.

If there are more than 10 active contributors, the list is truncated to the top 10 based on contribution metrics for better clarity.

Access Last Week's Newsletter:

• Link