Weekly GitHub Report for Pytorch

Thank you for subscribing to our weekly newsletter! Each week, we deliver a comprehensive summary of your GitHub project's latest activity right to your inbox, including an overview of your project's issues, pull requests, contributors, and commit activity.

---

Table of Contents

• I. News
  • 1.1. Recent Version Releases
  • 1.2. Other Noteworthy Updates
• II. Issues
  • 2.1. Top 5 Active Issues
  • 2.2. Top 5 Stale Issues
  • 2.3. Open Issues
  • 2.4. Closed Issues
  • 2.5. Issue Discussion Insights

• III. Pull Requests
  • 3.1. Open Pull Requests
  • 3.2. Closed Pull Requests
  • 3.3. Pull Request Discussion Insights

• IV. Contributors
  • 4.1. Contributors

I. News

1.1 Recent Version Releases:

The current version of this repository is v2.6.0

1.2 Version Information:

Released on January 29, 2025, PyTorch 2.6 introduces significant enhancements including torch.compile support for Python 3.13, a new dynamic compilation control API torch.compiler.set_stance, and improved AOTInductor packaging and ABI compatibility. Notable highlights also include FP16 support on X86 CPUs, expanded Intel GPU support, FlexAttention for X86 CPUs targeting LLMs, and a backward-incompatible security improvement flipping the default of torch.load to weights_only=True, alongside the deprecation of official Conda package publishing.

II. Issues

2.1 Top 5 Active Issues:

We consider active issues to be issues that that have been commented on most frequently within the last week. Bot comments are omitted.

1. [TRIAGED] [ONCALL: PT2] [ONCALL: EXPORT] [MODULE: AOTINDUCTOR] [aoti] [xpu] [null-pointer-deference] potential npt issue in sycl_runtime_wrappers.h: This issue addresses a potential null-pointer dereference in the sycl_runtime_wrappers.h file where memory allocated by malloc may fail, resulting in a nullptr that could cause a crash when accessed. The reporter suggests adding a null-pointer check to safely handle allocation failure and prevent the driver from writing into a null buffer, and the discussion leads to a consensus on implementing a non-throwing guard around the allocation to preserve error reporting.

   • The comments confirm the need for a null-pointer check, with one contributor offering to draft a PR without a test, which is encouraged by others. Further discussion reveals a detailed fix involving a guarded allocation and error handling that avoids throwing exceptions, and the fix is eventually incorporated into a refactoring PR after verification against a real driver scenario.
   • Number of comments this week: 6

2. [FEATURE] [TRIAGED] [NEEDS RESEARCH] [MODULE: VULKAN] Offer official Pytorch Vulkan backend on pytorch.org: This issue requests official support for a Vulkan backend in PyTorch to enable machine learning workloads on a wide range of hardware that currently lacks adequate support, such as AMD GPUs with poor ROCm support, Intel GPUs, Qualcomm chips, and potentially even macOS devices via MoltenVK. The motivation is to increase consumer choice, reduce NVIDIA's dominance in ML hardware, and leverage Vulkan's cross-platform capabilities and competitive performance, as demonstrated by benchmarks where Vulkan often outperforms ROCm, especially on AMD hardware.

   • The comment discussion extensively covers the current state and challenges of Vulkan support, including comparisons of Vulkan performance to ROCm and CUDA, the incomplete and inference-only nature of PyTorch’s existing Vulkan backend, and ongoing community efforts to build a full Vulkan backend by porting BLAS libraries and kernels. Contributors highlight Vulkan’s broad hardware compatibility and stability advantages, note the difficulty of implementing all required kernels, and share progress on developing a Vulkan backend that supports training and inference, while also discussing alternatives and related projects.
   • Number of comments this week: 5

3. [ONCALL: DISTRIBUTED] [NEEDS DESIGN] [BOT-TRIAGED] [BOT-MISLABELED] [ONCALL: DISTRIBUTED INFRA] [PTD-BOT-TRIAGED] nanosleep() call in socket.cpp delay method cannot handle signal interrupts (e.g. from sampling-based profilers): This issue addresses a bug in the nanosleep() call within the delay() method of socket.cpp that fails to properly handle signal interrupts such as SIGPROF, causing threads to exit prematurely and leading to deadlocks or hangs when profiling distributed PyTorch applications. The problem arises because the current error handling does not retry the nanosleep() call upon interruption, and a proposed fix involves distinguishing SIGPROF interrupts from others by using a signal handler that sets a flag to control whether to throw an error or continue sleeping.

   • The comments include a detailed reproducer script and build instructions for a thread profiler that triggers the issue, a discussion about the nature of the interrupt handling problem, suggestions for a fix involving signal flags, confirmation of a local patch that addresses the issue, and a request to mark the issue as actionable to allow opening a pull request.
   • Number of comments this week: 5

4. [ONCALL: DISTRIBUTED] [BOT-TRIAGED] [ONCALL: DISTRIBUTED INFRA] Update NVSHMEM to Resolve RoCE Failures: This issue requests updating NVSHMEM to the latest version to resolve failures with RDMA over Converged Ethernet (RoCE) encountered when using new methods in DeepEP within vLLM and llm-d, as current versions lack necessary fixes introduced after 3.5.19. The update aims to avoid patching PyTorch directly and improve compatibility by incorporating fixes such as zero-initialization of a critical struct in NVSHMEM's IBGDA transport plugin, which addresses the all2all backend failure on RoCE.

   • The comments discuss the need for more details on the failing methods and provide references to related patches and pull requests for context. There is agreement on the update's necessity, with offers to test the update on local systems and clarifications on the specific fix in NVSHMEM that resolves the issue.
   • Number of comments this week: 5

5. [MODULE: DOCS] [TRIAGED] [ONCALL: PT2] [BOT-TRIAGED] Tensor.is_contiguous is false for expanded tensors: This issue discusses the ambiguity in the definition of the is_contiguous property for expanded tensors in PyTorch, particularly when strides are zero, which leads to unexpected behavior in memory layout and affects operations like matmul. The user requests clearer documentation on what "contiguous" means in this context, highlighting that expanded tensors with zero strides are not considered contiguous despite occupying a contiguous memory chunk, causing discrepancies in Ahead-of-Time Autograd and eager execution paths.

   • The comments clarify that memory being contiguous does not imply the tensor is contiguous, emphasizing that contiguous tensors must have strictly decreasing strides without overlap, which expanded tensors violate. Participants discuss the nuances of memory format definitions, the implications of zero strides, and suggest improvements to the documentation to better explain these concepts.
   • Number of comments this week: 5

2.2 Top 5 Stale Issues:

We consider stale issues to be issues that has had no activity within the last 30 days. The team should work together to get these issues resolved and closed as soon as possible.

As of our latest update, there are no stale issues for the project this week.

2.3 Open Issues

This section lists, groups, and then summarizes issues that were created within the last week in the repository.

Issues Opened This Week: 98

Summarized Issues:

• Inductor Backend Numerical and Semantic Bugs: Multiple issues in the Inductor backend of torch.compile cause incorrect numerical results and IEEE 754 semantic violations, including mishandling of negative zero, incorrect rounding of ELU outputs, catastrophic cancellation in reciprocal computations, and producing negative NaN bit patterns. These bugs lead to silent discrepancies between eager and compiled execution modes, affecting accuracy and downstream computations.
• issues/189787, issues/189799, issues/189801, issues/189803, issues/189804, issues/189808

• Inductor Backend Compilation and Runtime Failures: Several Inductor backend issues cause runtime crashes or compilation errors, including unhandled Triton CompilationError during autotuning, NaN symbolic bound errors in layer norm computations, overflow errors when summing infinite values, and crashes due to broadcasting shape mismatches or unsupported complex dtypes. These failures disrupt model compilation and execution despite successful eager mode runs.
• issues/190002, issues/190363, issues/190417, issues/190418, issues/190421, issues/190275

• Inductor Backend Performance Regressions: The Inductor compiler exhibits significant slowdowns when compiling models with symbolic dynamic dimensions, particularly for torch.cat operations, due to inefficient runtime integer division and modulo operations. A fallback to the ATen kernel is suggested to mitigate this 2–2.5× slowdown compared to eager execution.
• issues/189940

• MPS Backend Bugs and Memory Corruption: The MPS backend contains multiple critical bugs including silent out-of-bounds writes in index_copy_, index_fill_, and index_reduce_ functions due to lack of bounds checking, off-by-one errors in embedding_bag kernel causing out-of-range reads and writes, and silent ignoring of scale_grad_by_freq flag in gradients. These issues lead to data corruption and incorrect outputs compared to CPU and CUDA backends.
• issues/189968, issues/189969, issues/189970, issues/189971, issues/190061, issues/190060

• MPS Backend Functional Incorrectness: The MPS implementation of certain functions like adaptive_max_pool2d and LayerNorm produce incorrect outputs due to improper kernel implementations and type reinterpretation errors, causing backward pass failures and silent errors in forward computations.
• issues/190064, issues/190054

• MPS Backend Tensor Concatenation and Copy Bugs: Bugs in the MPS backend cause silent data corruption during tensor concatenation with torch.cat when output sizes exceed 2^31 elements, and incorrect element copying with non-dense strides during view-to-view copies, leading to out-of-bounds writes and incomplete updates.
• issues/189960, issues/189961

• CUDA Kernel Illegal Memory Accesses: Several CUDA kernels suffer from illegal memory reads due to shape mismatches or invalid input values, including batch norm statistics reduction, int4 packed matrix multiplication, and native batch norm backward kernels. These out-of-bounds accesses cause crashes or undefined behavior during execution.
• issues/189828, issues/189829, issues/189831

• Distributed Deadlocks and Process Group Issues: Deadlocks occur in distributed tests involving expandable segments IPC and during destroy_process_group() calls due to inconsistent process group naming and fabric-handle stalls, causing test hangs and requiring disabling until resolved.
• issues/189878, issues/189879, issues/190396

• Test Failures and Skips in CI: Numerous tests across platforms and features are currently skipped or disabled in the main branch to maintain CI stability, including tests for dropout alignment, ROCm GPU operations, DTensor scaling, and various backend-specific tests. These skips await fixes to restore full test coverage.
• issues/190232, issues/190233, issues/190234, issues/190235, issues/190236, issues/190237, issues/190238, issues/190239, issues/190240, issues/190241, issues/190242, issues/190295

• ROCm Platform Accuracy and Performance Issues: ROCm platform users experience intermittent accuracy flakiness in model training and performance regressions in scaled dot product attention, with some issues attributed to ROCm version bugs rather than PyTorch regressions.
• issues/189893, issues/190154

• Distributed Communication and Backend Enhancements: Proposals include building ROCm preview nightly wheels, refactoring distributed tests for multi-backend support, and developing a low-latency RDMA-based collective communication library for Apple Silicon MPS tensors to enable native distributed workloads.
• issues/189756, issues/189809, issues/189853, issues/190299

• Documentation and Code Style Improvements: Issues highlight incorrect or ambiguous documentation, including invalid comments in torch.profiler examples, ambiguous definitions of tensor contiguity, and proposals for consistent formatting and linting rules to improve documentation quality.
• issues/189832, issues/190202, issues/190203, issues/190204, issues/190038, issues/190367

• PyTorch ONNX Exporter Security and Correctness Issues: The new ONNX exporter embeds non-tensor arguments as hard-coded constants, potentially allowing silent injection of attacker-controlled values and reducing model portability.
• issues/190199

• Sparse Tensor and Memory Safety Bugs: Malformed sparse tensors with invalid metadata cause memory write crashes and floating point exceptions during conversion to dense tensors, indicating a need for better input validation before kernel execution.
• issues/190046, issues/190048

• Gradcheck and Autograd Edge Case Failures: Forward-mode automatic differentiation crashes on zero-element differentiable inputs due to assumptions about non-empty Jacobians, requiring gradcheck to skip such cases gracefully.
• issues/190436

• Memory Leak and Resource Management Issues: CUDA memory leaks occur in tests involving torch.compile with real tensor propagation and data-dependent operations, and warnings are proposed to detect missing calls to destroy_process_group to prevent difficult-to-debug crashes.
• issues/190093, issues/190489

• Build and CI Infrastructure Updates: Proposals include promoting CUDA 13.2 as default for PyTorch 2.14, deprecating CUDA 12.6 builds for PyTorch 2.15, and enabling Windows Python 3.15 XPU nightly wheel builds, alongside fixes for build failures caused by environment variable misconfiguration.
• issues/190355, issues/190385, issues/190542, issues/190070

• Linear Algebra and Memory Efficiency Issues: Lack of early validation for 32-bit integer workspace size overflows in CPU LAPACK operations can cause silent overflows and cryptic errors, and NCCL symmetric memory backend suffers from inefficient O(n) lookups due to hash map usage, suggesting improvements to ordered maps for O(log n) performance.
• issues/190509, issues/190541

• Benchmark Failures and Runtime Errors: The vLLM benchmark consistently fails on certain Linux systems due to invalid tensor shapes and CUDA initialization errors, preventing benchmark execution and result generation.
• issues/190215, issues/190216, issues/190225

• New Feature Proposals: Requests include adding a native Log-Cosh Loss implementation, loading AOT Inductor model packages from in-memory buffers, and enhancing AOTInductor runtime support for PrivateUse1 backends to unify backend integration.
• issues/190037, issues/189995, issues/189998

• Tensor Creation and Scalar Handling Inefficiencies: Creating scalar tensors on CUDA devices causes unnecessary host-to-device synchronization, unlike torch.full, leading to blocking behavior that could be optimized.
• issues/190217

• Regression in Float8 Scaling on CUDA: Starting from PyTorch 2.10, the scale_result parameter is silently ignored in _scaled_mm producing float8 outputs on CUDA, causing incorrect scaling and potential downstream corruption.
• issues/190449

• Sparse Tensor Multiprocessing Memory Leak Uncertainty: CUDA sparse tensors shared across processes via forkserver cause allocator memory to be flagged as leaked during slow-gradcheck, with uncertainty whether this is a true leak or a false positive from the leak checker.
• issues/190268

2.4 Closed Issues

This section lists, groups, and then summarizes issues that were closed within the last week in the repository. This section also links the associated pull requests if applicable.

Issues Closed This Week: 155

Summarized Issues:

• Depthwise3x3 Convolution Test Failures: Multiple issues report failures in the depthwise3x3 convolution tests within the Caffe2 component of PyTorch, where relative error thresholds were exceeded causing test failures. These issues were eventually closed due to the removal of Caffe2 and its tests from PyTorch, marking the problem as obsolete.
  • issues/19708, issues/35553

• CUDA Runtime and Memory Errors: Users experience intermittent CUDA runtime errors such as "CUBLAS_STATUS_EXECUTION_FAILED" and illegal memory access during gradient computations on CUDA 10.1, indicating instability in CUDA operations during model training.
  • issues/43325

• ModuleDict Ordering Issue: The ModuleDict in PyTorch does not preserve the order of the initializing dictionary, contrary to expected behavior where insertion order should be maintained, leading to potential issues in model reproducibility and debugging.
  • issues/43402

• ImportError Due to Version Mismatch: Running Detectron2 causes an ImportError related to a missing shared library libtorch_cpu.so because of a version mismatch between the prebuilt Detectron2 extension and the installed PyTorch version, requiring version alignment to fix.
  • issues/43604

• GPU Installation Problems: Users report difficulties installing the GPU-enabled version of PyTorch despite having CUDA installed, with tests indicating that the CUDA version of PyTorch was not properly installed on their machines.
  • issues/66359

• Non-ASCII Path Loading Failure: The torch.jit.load function fails to load models when the file path contains non-ASCII characters on Windows, resulting in runtime errors and preventing model loading in such cases.
  • issues/75171

• AdaptiveMaxPool Overflow and Segmentation Faults: Extremely large output sizes or input dimensions in torch.nn.AdaptiveMaxPool2d and AdaptiveMaxPool3d cause integer overflow errors or segmentation faults, crashing the program instead of raising controlled errors, though some issues have been fixed in later versions.
  • issues/91552, issues/91553, issues/91556, issues/91557

• AdaptiveAvgPool Integer Overflow: Using excessively large output sizes with torch.nn.AdaptiveAvgPool2d results in integer multiplication overflow errors, causing tensor sizes to exceed representable limits and leading to operation failures, identified as usage limitations rather than bugs.
  • issues/107543, issues/107556

• Torch Compile Dynamic Mode Runtime Errors: Running .sample() in diffusers VAE code with Torch Compile's dynamic mode triggers runtime errors in the CUDA backend due to symbolic shape inputs, which can be temporarily avoided by disabling dynamic compilation at the cost of slower performance.
  • issues/111385

• Heap Buffer Overflows in Tensor Flip Operations: The torch.Tensor.rot90, fliplr, and flipud functions cause heap buffer overflows when used on specific quantized tensor inputs, leading to memory corruption errors during flip kernel operations.
  • issues/116250, issues/116252, issues/116253

• Incorrect Documentation for dsplit(): The documented indices_or_sections parameter for the dsplit() function does not work as described, while the sections or indices parameters function correctly, indicating a need to update the documentation.
  • issues/127009

• Operator Tag Expansion Proposal: A proposal to add reduction and view tags to PyTorch operator tags aims to better categorize operators by behavior, improving consistency and maintainability.
  • issues/129020

• Zig C++ Compiler Support Request: There is a request to support using the Zig C++ compiler invoked as zig c++ with torch.compile, highlighting compatibility challenges due to spaces in the compiler command and assumptions about version string formats.
  • issues/133078

• Inductor Compiler Redundant Assignments: The Inductor compiler generates redundant tensor assignments during code lowering, complicating test logic without affecting performance, suggesting a need for code cleanup.
  • issues/136012

• Sparse-Dense BFloat16 Multiplication Not Implemented: The sparse-dense matrix multiplication function bmm_sparse_cuda lacks implementation for the BFloat16 data type, causing runtime errors when attempting half precision sparse tensor operations on CUDA.
  • issues/136186

• Complex Tensor Norm Returns Float: The linalg.norm() and linalg.vector_norm() functions return float tensors instead of complex tensors when applied to complex inputs with dtype=torch.complex64, which is expected and documented behavior.
  • issues/136452, issues/136568

• Min/Max Expression Simplification Limitations: Expressions involving Min/Max functions do not simplify correctly when value range information is available due to lack of access to shape environment data within Sympy simplification rules, complicating implementation of such optimizations.
  • issues/137096

• Outdated Torch Logs Tutorial Documentation: The PyTorch tutorial for torch logs contains outdated expected output indicating lack of torch.compile support, suggesting the need to update documentation by running the tutorial to reflect current behavior.
  • issues/137285

• CPU vs GPU Float32 and BFloat16 Rounding Differences: Functions like torch.special.polygamma, xlog1py, xlogy, and logcumsumexp show inconsistent results between CPU and GPU executions due to expected rounding differences in float32 and bfloat16 precisions, not indicating correctness bugs.
  • issues/140701, issues/140796, issues/140892, issues/141802

• slow_conv_transpose3d Crash with Large Dilation: The slow_conv_transpose3d function crashes with a double free or corruption error when given inputs with extremely large dilation values, aborting the program.
  • issues/142457

• Quantization Export Input Node Ordering Bug: In PyTorch 2.0 export quantization, get_source_partitions() returns inconsistent ordering of input nodes for the same graph, causing errors in quantization backend code due to unpredictable input node order.
  • issues/147170

• Logging Error on Pytest Shutdown: The remote_cache.py module attempts to write log messages after pytest has closed its logging system, causing a ValueError due to I/O on a closed file during test teardown.
  • issues/147486

• Add keepdim Parameter to Cosine Similarity: A feature request proposes adding a keepdim parameter to torch.nn.functional.cosine_similarity to maintain output dimensionality after aggregation, aligning it with similar functions.
  • issues/149120

• Dynamic Shapes with Keyword Arguments Export Bug: Using dynamic shapes with keyword arguments in model export causes assertion errors due to missing required tree names or mismatched keys, leading to export failures and confusion.
  • issues/150022, issues/150371

• AMD EPYC CPU Test Failure Due to Tolerance: The test_vmap_autograd_grad_nn_functional_conv2d_cpu_float32 test fails on AMD EPYC 7702 CPUs due to assertion errors from tensor values not being close enough, suggesting increasing tolerance to match CUDA thresholds.
  • issues/151113

• JIT Compiled Instance Norm Output Discrepancy: The F.instance_norm function produces different results when compiled with JIT compared to eager mode, especially when passing input as running statistics, causing inconsistencies.
  • issues/153315

• Cherry-Pick Management for PyTorch 2.8.0: The process and criteria for cherry-picking low-risk and critical fixes into the PyTorch 2.8.0 release branch are outlined to ensure stability before final release.
  • issues/156745

• FSDP Argument Mutation Bug: The Fully Sharded Data Parallel implementation copies arguments instead of passing them as-is, causing mutations inside the forward method to not reflect outside and leading to inconsistencies between single-device and FSDP models.
  • issues/157832

• torchrun NUMA Binding Documentation Request: A request to add documentation for torchrun NUMA binding to improve information on running elastic jobs on the PyTorch website.
  • issues/158775

• Compile Regression with Forward Pre-Hook: Registering a forward pre-hook with prepend=True on a parallelized module causes a TypeError during compilation in PyTorch 2.7.1 and 2.8.0, which did not occur in 2.6.0, with a workaround involving custom hook registration.
  • issues/160083

• Inductor Test Suite Disabled on CPU: The Inductor test suite test_torchinductor_opinfo was mistakenly disabled on all devices except CUDA, leading to skipped CI jobs and CPU job timeouts when reverting the change.
  • issues/161132

• Multiple Test Failures on B200 CUDA Runner: The test_max_autotune.py suite fails on the B200 CUDA runner with mismatched elements, unpacking errors, and assertion errors related to autotuning splits.
  • issues/163429

• torch.compile Graph Break on CUDA Memory Pool: torch.compile causes graph breaks when compiling torch.cuda.use_mem_pool, which is necessary for symmetric memory allocation in compiled CUDA operations.
  • issues/167026

• Serialization Failure with String Type Annotations: Serialization of fx.GraphModule fails with an AttributeError when function type annotations are strings, breaking torch.save due to improper import block formatting.
  • issues/167117

• ncclDevComm Kernel Access Implemented: The proposal to provide kernel access to ncclDevComm similar to nvshmem_team was successfully implemented with a manager, feature gates, kernel, and tests.
  • issues/167684

• Inductor Cache Freshness Bug: The torch._inductor.utils.fresh_cache function fails due to persistent subprocesses retaining outdated cache directory paths after cache deletion, causing FileNotFoundError during compilation.
  • issues/167725

• ROCm Autotuning Test Case: The test_autotune_int64_user_defined_triton_kernel test case relates to ROCm platform autotuning for user-defined Triton kernels.
  • issues/168566

• torch.compile NotImplementedError on MXFP8 MoE Training: Using the latest PyTorch nightly with torch.compile causes a NotImplementedError related to make_reindexer on DtypeView during MXFP8 MoE training on local B200 GPUs.
  • issues/168995

• torch.compile TVM Backend SymInt Tracing Failure: Using torch.compile with the TVM backend on PyTorch 2.9.0 fails due to the JIT tracer not handling symbolic integer types (SymInt), causing runtime errors before code generation.
  • issues/169111

• torch.compile TVM Backend Incorrect Output: Compiling a simple model with torch.compile and the TVM backend produces incorrect output (0.0) on CUDA, whereas eager mode returns the correct result (2.0).
  • issues/169188

• xpu Test Disabled Due to Failure: The test_buffer_mutation_and_force_mmap_weights_xpu test was disabled due to consistent failure on the main branch for the xpu platform.
  • issues/170007

• vLLM CI Failure Due to Inductor Import Error: The vLLM continuous integration tests fail because lazy_imports.py cannot handle importing torch._inductor.async_compile in PyTorch 2.9.0 and 2.10.0 rc1, causing an AssertionError.
  • issues/170345

• Inductor Meta Kernel Shape Mismatch on ROCm: The Inductor meta kernel for flash and efficient attention on CDNA architectures fails due to shape mismatches caused by assumptions of CUDA output shapes differing from ROCm's, leading to assertion failures.
  • issues/171568

• vLLM Draft Model KeyError Crash: The test_draft_model_correctness test fails with a KeyError due to missing attention layer keys when using the same model architecture for target and draft models, causing compiled graph execution errors.
  • issues/173546

• vLLM Engine Core Initialization Failures: vLLM end-to-end tests fail during engine core initialization in release 2.10, with workarounds suggested for compatibility with specific PyTorch versions.
  • issues/174081

• cpu_inductor_torchbench Test Instability Resolved: Instability and accuracy failures in the cpu_inductor_torchbench test were fixed, restoring test stability and coverage in the blocking set.
  • issues/174929

• make_fx Symbolic Tracing Incorrect Gradients: Using make_fx with symbolic tracing produces incorrect second-order gradients involving torch.sqrt, causing significant divergence from eager execution gradients.
  • issues/175477

• vLLM HF vs vLLM Model Output Logit Differences: Tests show output logit differences beyond tolerance between Hugging Face and vLLM models for certain language models, suspected to be related to the Inductor backend, prompting tolerance updates and fixes.
  • issues/175928

• vLLM Extended Generation Test Failure: The vLLM extended generation test for the bigcode/starcoder2-3b model fails due to mismatched token log probabilities compared to Hugging Face runner outputs.
  • issues/177393

• torch.compile Shape Mismatch Regression: A regression causes torch.compile to raise a RuntimeError on a valid torch.addmm call with beta=0.0 due to incorrect bias shape validation, whereas eager mode succeeds.
  • issues/178040

• Triton Heuristic Refactoring Proposal: A proposal to refactor Triton heuristic selection in Inductor introduces a unified device-scoped class hierarchy and shared registry to improve maintainability and reduce conditional branching.
  • issues/178153

• Case Mismatch Prevents CUSTOM Backend Setting: A case mismatch between uppercase plugin keys and lowercase backend config values prevents setting the default backend type to CUSTOM, causing failures in backend hook lookups during process group destruction.
  • issues/178756

• E8M0 Quantization Numerical Discrepancy: Numerical discrepancies occur between eager and torch.compile (Inductor) modes for E8M0 bit-manipulation quantization due to subtle float precision differences affecting classification logits.
  • issues/178880

• Scaled_mm Test Disabled Due to Failure: The test_scaled_mm test in the DistMatrixOpsTestWithLocalTensor suite was disabled due to failures on ROCm and CUDA platforms, with indications it only works correctly on a single GPU.
  • issues/179951

• RecursionError from Custom getattr in torch.compile: Using torch.compile on an nn.Module subclass with a custom __getattr__ that calls hasattr causes infinite recursion and a RecursionError, unlike eager execution.
  • issues/180824, issues/183890

• test_emulate_precision_casts_preserves_explicit_precision_cast_float16 Disabled: This test was disabled due to failures on the main branch, particularly on xpu platforms.
  • issues/180948

• torch.compile Inductor C++ Compilation Failure: Compiling a model using torch.gather with a self-referential dynamic index causes C++ kernel compilation failure due to out-of-scope variable references.
  • issues/181621

• Triton Kernel Signature Generation Crash: Compiling CUDA code with complex number operations using torch.compile causes a KeyError for 'complex64' during Triton kernel signature generation, leading to internal errors.
  • issues/185396

• Incorrect Multilabel Margin Loss in torch.compile: torch.compile produces incorrect forward loss and backward gradients for F.multilabel_margin_loss when target tensors contain -1 padding markers, silently corrupting training results.
  • issues/185464

• torch.compile Runtime Crash with torch.cond and scatter_add: Compiling a function with torch.compile involving torch.cond and scatter_add causes runtime crashes due to non-tensor input errors in fused kernels.
  • issues/185482

• Invalid Shape RuntimeError in torch.compile Dynamic Mode: Using torch.compile(dynamic=True) generates invalid "garbage" shapes during forward passes, causing crashes in alias tracking not seen in eager mode.
  • issues/185630

• Internal Assertion Failure with torch.autograd.grad and torch.pow: An internal assertion failure occurs in the C++ engine when using torch.compile(dynamic=True) with torch.autograd.grad after torch.pow if the exponent is a Python integer argument.
  • issues/185715

• Inefficient torch.mm for Small Matrices: torch.mm is slower than explicit pointwise multiplication plus sum for small matrix dimensions, suggesting decomposition into unsqueeze + mul + sum for better performance and optimization.
  • issues/186348

• Dynamo Graph Break on torch.matmul in Distributed Training: Dynamo compiler encounters graph breaks when tracing distributed training workloads involving torch.matmul due to untraceable calls in the eager router.
  • issues/186354

• ROCm Device Count Bug with amdsmi Package: Installing the standalone amdsmi Python package causes torch.cuda.device_count() to incorrectly return 0 on ROCm due to duplicate libamd_smi libraries causing symbol clashes.
  • issues/186516

• swap_tensors Fails on Tensors with Weak References: torch.utils.swap_tensors() fails on tensors with weak references, which occur in live torch.compiled modules due to persistent weakrefs, causing runtime errors during group offloading.
  • issues/186796

• test_combo_kernel_no_bench_persistent_reduction Disabled and Fixed: This test was disabled due to failures on xpu but later fixed and re-enabled.
  • issues/187019

• Inductor Missing Autotune Candidate for baddbmm with Broadcast Bias: Lack of autotune candidate for baddbmm with broadcast bias prevents fusion and performance improvements in MoE expert-GEMM shapes.
  • issues/187093

• Inductor TMA Persistent Template Crash: The TMA persistent template crashes with an AssertionError due to an unregistered range symbol when fusing epilogue operations with tma_store enabled.
  • issues/187158

• torch.compile Drops Forward-Mode AD Tangents: Compiling with torch.compile using the Inductor backend silently drops forward-mode automatic differentiation tangents, returning None instead of expected tangent tensors.
  • issues/187284

• ROCm expandable_segments Causes hipErrorIllegalAddress Crash: Enabling expandable_segments on ROCm causes illegal memory access crashes on AMD Radeon AI PRO R9700 hardware, likely due to kernel-level virtual memory bugs.
  • issues/187343

• Dynamo Avoid Unnecessary Guards Proposal: Proposal to optimize Dynamo by avoiding creation of guards on unused function inputs while preserving essential guards to prevent incorrect behavior.
  • issues/187593

• torch.compile Reduce-Overhead Mode Causes ROCm Training Degradation: Using torch.compile(mode='reduce-overhead') on ROCm silently degrades training convergence due to missing HIP guards in cudagraph implementation.
  • issues/187715

• torch.compile Causes Memory Growth on ROCm: Enabling torch.compile in PyTorch 2.9 on ROCm causes persistent host memory growth during repeated inference, unlike earlier versions or disabled compile mode.
  • issues/188127

• test_fallback_kernel_with_symexpr_output_xpu Disabled: This test was disabled due to failures on the xpu platform.
  • issues/188131

• torch.cond Compilation Failure with Identical Symbolic Shapes: torch.cond fails to compile under @torch.compile(fullgraph=True, dynamic=True) when both branches have identical symbolic shapes and strides, causing runtime errors related to non-contiguous strides.
  • issues/188151

• Undocumented Determinism of Module.parameters() Ordering: Lack of documentation on whether torch.nn.Module.parameters() returns parameters in a deterministic order, raising questions about consistency across calls.
  • issues/188389

• DTensor Stable Hashing False Cache Hits: DTensor stable hashing generates identical AOTAutograd hash keys across ranks by omitting FakeTensor device_id metadata, causing incorrect autograd cache reuse.
  • issues/188390

• Transformers sdpa_can_compile_dynamic Test Failure: The transformers library test fails with PyTorch 2.13 RC due to Inductor errors involving negative exponents during symbolic shape evaluation.
  • issues/188423

• Triton 3.8.0 Upgrade Causes Inference Accuracy Regression: Upgrading Triton to 3.8.0 causes numerical divergence in bf16 inference on H100 GPUs for the swin_base_patch4_window7_224 model, affecting all Triton-codegen Inductor backends.
  • issues/188492

• PyProcessGroup Missing Collective Method Overrides: The PyProcessGroup class lacks overrides for several collective communication methods, causing failures in Python-defined process groups.
  • issues/188542

• torch.special.bessel_y0 Output Discrepancy on CUDA: Differences occur between eager and torch.compile (Inductor) modes for torch.special.bessel_y0 on CUDA when input contains positive infinity, affecting NaN handling.
  • issues/188545

• torch._dynamo int() on pybind11 Enum Causes Recursion: Calling int() on a pybind11 enum returned from a compiled function causes infinite recursion and maximum recursion depth exceeded errors.
  • issues/188547

• test_stream_invalid_device_index Disabled: Disabled due to consistent failure on main branch.
  • issues/188691

• test_vjp_scalar_logging_tensor Disabled: Disabled due to consistent failure on main branch.
  • issues/188693

• test_cow_inputs_accepted_by_override Disabled on xpu: Disabled due to consistent failure on main branch for xpu platform.
  • issues/188773

• Triton Version Pinning Causes Regression: Updating pinned Triton version to 3.8 causes regressions in compilation and float8 training numerics, resolved by reverting to previous version.
  • issues/188841

• test_optimizer_non_static_param Disabled on xpu: Disabled due to consistent failure on main branch for xpu platform.
  • issues/188987

• Nightly update-metamates-rule Job Fails Due to Token Permissions: The job fails repeatedly because the UPDATEBOT_TOKEN lacks read:org permission and SSO authorization for private team membership.
  • issues/189032

• Build Profiling Instrumentation Proposal: Proposal to add build profiling instrumentation to PyTorch's CMake setup to monitor build times and performance impact.
  • issues/189078

• Inductor Autotuner Cache Creation Bug: Single-config reduction kernels extended by _dynamic_scale_rblock fail to create or save autotune cache, causing repeated benchmarking and nondeterministic performance.
  • issues/189121

• B200 Runner Maintenance Notification: Notification about ongoing maintenance causing job failures requiring restarts.
  • issues/189164

• Periodic-rocm-mi300 Test Disabled Due to Cluster Outages: Disabled due to MI300 cluster outages causing reduced capacity and queueing.
  • issues/189261

• rocm-nightly Test Disabled Due to Cluster Outages: Disabled due to MI300 cluster outages causing reduced capacity and queueing.
  • issues/189262

• vLLM Multimodal Pooling Tests Fail After Triton 3.8 Update: EngineCore fails to initialize CUDA driver due to unsupported driver version on test hardware, causing test failures.
  • issues/189287

• Missing @onlyCUDA Decorator Causes CPU Test Failure: Missing decorator on CUDA-specific test causes CPU variant to fail due to profiling errors.
  • issues/189409

• test_create_with_external_constants_xpu Disabled: Disabled due to consistent failure on main branch for xpu platform.
  • issues/189511, issues/189515

• torch.cond Branch Output Mismatch Causes NameError: Using torch.cond with branch outputs mismatching in non-leading dimension causes undefined variable errors and crashes in Inductor backend.
  • issues/189528

• MPS to CPU Copy Overwrites Source Tensor: Dtype-converting copy from MPS to CPU with nonzero buffer offset and smaller element size silently overwrites source MPS tensor, causing corruption.
  • issues/189563

• Error Uploading File 19efb3181397a6288861.mp4:
  • issues/189715

• PyTorch Nightly RoCM 7.2 CI Download/Install Issues: Downstream projects cannot properly download or install latest nightly wheels due to invalid package formats and Python 3.12 compatibility issues.
  • issues/189748

• FlexAttention Unbacked SymInt Checks Cause Recompilation Failures: Adding torch._checks for unbacked symbolic integer support causes repeated recompilation failures due to pinning query sequence dimension.
  • issues/189786

• CI Jobs Not Running Due to Quota Exceeded: No CI jobs run on pull requests due to exceeding max queued jobs across enterprise account, caused by internal repo consuming quota.
  • issues/189789

• XPU Backend Missing memory_allocated Implementation: Lack of memory_allocated function for Intel Arc GPUs causes runtime errors and prevents model loading on Linux with Intel Arc Pro B70 hardware.
  • issues/189811

• test_unused_cpu_input_cudagraphs Disabled on xpu: Disabled due to consistent failure on main branch for xpu platform.
  • issues/189815

• CUDA multi_tensor_apply_kernel Illegal Memory Read: Division on large complex double tensor list causes out-of-bounds access due to metadata over-read in kernel chunk-count accounting.
  • issues/189827

• MPS linear Function Slow on 3D Inputs in Half Precision: torch.nn.functional.linear on MPS with 3D inputs in bf16/fp16 is up to 8.5x slower than 2D or matmul equivalents due to per-batch GEMV execution.
  • issues/189847

• ROCm Manywheel Python 3.15 Build Failure: Building numpy 2.5.1 from source causes pkg-config conflict and Cython sanity check failure due to system Python 3.6 detection.
  • issues/189852

• Inductor constant_pad_nd Fusion Bug: Fusing constant_pad_nd into split reduction kernel drops pad boundary selection, corrupting outputs and causing intermittent garbage or NaNs in flash attention models.
  • issues/189895

• _io.text_encoding Causes Graph Break in torch.compile: The C builtin _io.text_encoding lacks a traceable Python implementation, causing graph breaks and fallback to eager execution.
  • issues/189925

• test_buffer_mutation_and_force_mmap_weights_xpu Disabled and Unintentionally Unskipped: Disabled due to failure on xpu platform, but was unintentionally unskipped after a recent PR.
  • issues/189963

• PGTransport Tests Device Binding Bug: Per-rank devices for registered accelerator backends were not properly bound, causing multiple ranks to use the same device and communication failures, fixed by setting current device index before communication init.
  • issues/189989

• share_memory=True, pin_memory=True Option Fails on Non-CUDA Accelerators: The combined option fails due to lack of accelerator-generic equivalent to CUDA's cudaHostRegister, raising questions about intended usage.
  • issues/189991

• AOTInductor Mixed CPU/CUDA Model Segmentation Fault: Segmentation fault occurs when CPU fused embedding kernel crashes due to embedding parameter pointer being non-CPU-readable despite passing index bounds checks.
  • issues/190000

• torch.fmod Integer Overflow Crash: Calling torch.fmod with int32 tensor containing minimum signed integer and divisor -1 triggers floating-point exception due to unsafe native kernel handling.
  • issues/190041

• torch.floor_divide Integer Overflow Crash: Dividing minimum 32-bit integer by -1 on CPU causes crash due to unhandled signed integer overflow in native kernel.
  • issues/190042

• torch.sparse.mm Segmentation Fault on Malformed CSR Matrix: Multiplying malformed CSR sparse matrix with zero columns by zero-column dense tensor causes invalid memory read and crash due to missing bounds checks.
  • issues/190045

• Malformed sparse_csc_tensor and sparse_bsc_tensor Crash: Invalid compressed column metadata causes segmentation fault and crash under AddressSanitizer when calling to_dense().
  • issues/190047

• F.linear with in_features=0 on macOS MPS Aborts: Forward pass aborts due to uncatchable MetalPerformanceShaders assertion, and backward pass fails due to improper gradient checks and reshaping.
  • issues/190050

• torch.abs on MPS Integer Tensors Causes Rounding Errors: torch.abs converts integers to float32 and back, causing silent rounding errors and value corruption for int32 and int64 tensors on Apple MPS devices.
  • issues/190052

• MPS LSTM Dropout Gradient Bug: Multilayer LSTM with dropout enabled on MPS computes incorrect gradients due to dropout mask not being reapplied, silently corrupting training.
  • issues/190056

• MPS LSTM Dropout=1.0 Returns NaNs: Multilayer LSTM with dropout set to 1.0 on MPS returns all NaN outputs due to incorrect scaling factor, unlike CPU backend.
  • issues/190057

• Embedding_bag Dense Backward Gradient Scaling Bug: CPU implementation incorrectly indexes counts array when scaling gradients by frequency, causing wrong gradient values during backpropagation.
  • issues/190063

• Backwards_compat CI Job Fails Due to CMake Version: Build breaks under CMake 4.x because NNPACK requires minimum CMake version below 3.5, causing schema diff not to run.
  • issues/190066

• B200 Runner Intermittent "No Devices Found" Errors: Incorrect cgroup and eBPF device filter configurations cause GPU visibility issues, resolved by adjusting DeviceAllow settings and node reboot.
  • issues/190132

• NLLLoss2d CUDA Backward Kernel Integer Overflow: 32-bit integer tensor offsets overflow when processing inputs with more than INT_MAX elements, causing illegal GPU memory access and crashes.
  • issues/190139

• SDPA Integration Incompatible with flash-attn >=4.0.0.beta19: Change in _flash_attn_fwd forward return signature breaks SDPA integration.
  • issues/190212

• FlopCounterMode Causes Dynamo to Skip Compilation: Dynamo permanently skips compiling frames after detecting active TorchDispatchMode, even outside that context.
  • issues/190286

• MPS Backend Lacks aten::_int_mm Support: Missing support for INT8 matrix multiplication on Apple Silicon MPS backend causes NotImplementedError and CPU fallback.
  • issues/190337

• ONNX Export fp16 LayerNorm Mixed Types Bug: Exporting Hugging Face BertModel in fp16 with dynamo disabled results in LayerNormalization nodes with mixed float32 and float16 types, causing ONNX Runtime load failures.
  • issues/190338

• hf-cache OOM Causes Kubernetes Pod Failures: Out-of-memory issues in hf-cache cause pods to fail during container initialization, leading to long CI job runtimes without execution.
  • issues/190347

• CI Runner Maintenance Advanced Due to Related Issue: Scheduled maintenance on CI runners was moved earlier, causing long queue times and job cancellations requiring retries.
  • issues/190364

• Broken Link in PyTorch 2.10 Documentation: A 404 broken link was fixed in main branch but remains in archived versioned docs.
  • issues/190387

• ROCm Backend Test Failure Due to Invalid Argument: The test_register_hook_on_intermediate_autograd_cache test fails due to invalid argument combination in torch.randn() in device-agnostic test modification.
  • issues/190414

• torch.nextafter on MPS Does Not Advance bfloat16: torch.nextafter returns input unchanged for bfloat16 on MPS, causing mismatches with CPU results due to missing bfloat16 overload.
  • issues/190476

• Withdrawn Issue: A previously submitted concern was withdrawn without further details.
  • issues/190484

• MPS LayerNorm Numerical Stability Bug: Variance computation in MPS layer_norm forward kernel causes divergence from CPU reference due to rounding noise and clamping, leading to incorrect outputs; fix proposed with two-pass variance computation.
  • issues/190491

• NVSHMEM Signal Pad Allocation Proposal: Proposal to allocate signal pad within same memory allocation as data buffer to reduce collective allocations and synchronization barriers, aligning with CUDA backend behavior.
  • issues/190501

• check_binary_symbols.py Requires C++20 Compiler: Smoke test probe program compiles with C++17 but updated headers require C++20, causing build failures until compile flags updated.
  • issues/190527

2.5 Issue Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed issues that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed issues from the past week.

---

III. Pull Requests

3.1 Open Pull Requests

This section provides a summary of pull requests that were opened in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Opened This Week: 336

Key Open Pull Requests

1. Add Autograd Support for Low-Level Bessel Functions: This pull request adds autograd support for eight low-level Bessel special functions by making them differentiable in both forward and backward modes, updating derivative formulas, implementing special backward handling for singularities, enabling comprehensive OpInfo testing, and including new autograd tests to ensure correctness.

• URL: pull/189872

• Associated Commits: 63a28, 15da7, 49dd1, 7034c, 145b1, 016d7, 59a5d, b7fa1, 2225e, 272c5, df2d6, 499ac, 976ac, 3e8c5, 5b59f, fc8f6

2. [precompile] Author the artifact driver as real code, emit via getsource: This pull request refactors the artifact driver code in torch.compiler.precompile by moving the previously hard-coded runtime driver strings into real, type-checked Python functions within torch/_precompile_driver.py and emitting them via inspect.getsource during code generation, thereby improving code visibility, maintainability, and type checking without changing runtime behavior or introducing version skew.

• URL: pull/190226

• Associated Commits: 6b2b0, 1535e, de8c8, 3ebe5, 3df30, 45610, feec8, 9610d, 7a16e, 7f3f5, 2411a, 18ade, 411c5

3. [distributed] Generalize logical RNG replay and add uniform support: This pull request generalizes the logical random number generator (RNG) replay mechanism by replacing the normal-specific backend with a private distribution-kind operator that supports both normal and uniform distributions, introduces a unified Python op-spec table and Meta implementation for scalar distributions, and includes validation and testing to ensure correct generator reservation and replay behavior.

• URL: pull/190444

• Associated Commits: ca15a, eedb6, a4bfb, f1908, f7851, f1911, 4359a, 74b26, 3ce15, 056ab, 78a7c, 67c4d, c6a63

Other Open Pull Requests

• Test Disabling for Failing Cases: Multiple pull requests disable or skip failing tests in continuous integration to maintain stability. One disables the parametrized test_main_loop_scaling in the TestFP8LoweringCUDA suite, while another disables five specific GPU tests related to cumsum, reduction, boolean operations, and vmap on ROCm, also affecting CPU variants.
  • pull/190103, pull/190106

• ROCm and GPU Performance Optimizations: Several pull requests focus on improving ROCm support and GPU performance. These include layout optimizations for convolution operations on CDNA architectures, validation and CI updates for ROCm 7.14 wheels, and disabling failing ROCm tests to handle known issues.
  • pull/190278, pull/190399, pull/190106

• Random Number Generation Enhancements: Pull requests introduce new protocols and APIs to improve stateful RNG behavior. They add a StatefulRNGTensor protocol for accurate normal distribution replay in distributed CUDA kernels and unify device-specific RNG offset functions into accelerator-agnostic APIs.
  • pull/190289, pull/190004

• AOTInductor and Autograd Runtime Improvements: Multiple pull requests enhance the AOTInductor runtime and Autograd performance. Changes include fixing segmentation faults in JIT autotuning, moving save_from_forward bookkeeping to C++ for performance gains, and making output wrapping and forward AD input checks more efficient.
  • pull/190250, pull/189790, pull/189865, pull/189866

• Windows ARM64 Build Support: One pull request adds support for building CUDA-enabled PyTorch and MAGMA libraries on Windows ARM64. It introduces ARM64-specific build configurations, scripts, and CI jobs while improving environment setup and removing fragile header patching.
  • pull/189949

• Profiler and Metadata Handling Enhancements: A pull request improves the PyTorch profiler by enabling direct passing of typed NCCL metadata to Kineto. This avoids inefficient string casting and JSON roundtrips, improving metadata handling efficiency and future extensibility.
  • pull/190371

• Inductor Compilation and Kernel Optimizations: Pull requests optimize Inductor compilation and kernel performance. They add a fallback for torch.cat to avoid costly operations with symbolic dimensions and modify the combo compile-time autotune process to be safe in deterministic mode by using heuristics instead of timing benchmarks.
  • pull/190034, [pull/189724](https://github.com/pytorch/pytorch/pull/189724]

• NVGEMM Backend Fixes and Enhancements: Several pull requests address NVGEMM backend stability and performance. Fixes include resolving a reentrant self-deadlock by replacing a threading.Lock with RLock, making addmm epilogue cudagraph-safe, enabling scaled-GEMM subprocess precompilation, and lowering subprocess precompile thresholds to improve compile times.
  • pull/189779, pull/189780, pull/189817, pull/189781

• Pybind11 Backend Subclassing Support: One pull request adds a PyBackend trampoline using pybind11 to allow Python classes to subclass c10d::Backend. This enables overriding collectives, lifecycle methods, and properties with improved dispatching and exposes missing Backend bindings.
  • pull/189817

• CheckpointableTensor and Safetensors Integration: A pull request adds support for CheckpointableTensor in Hugging Face safetensors storage. It introduces metadata mapping physical shard keys to logical names and stores global shapes, fixing tensor size reconstruction bugs and ensuring compatibility with various checkpoints.
  • pull/189945

• Python 3.15 Windows Wheel Support: One pull request updates CI workflows to build and upload Python 3.15 and 3.15t nightly binary wheels for Windows. It mirrors existing Linux support by adjusting build matrices, pinning dependencies, and modifying installation scripts for the pre-release Python environment.
  • pull/190360

• Autograd Function Call Optimization: A pull request optimizes the autograd.Function forward method by using PyObject_Vectorcall. This avoids creating temporary call tuples and reduces reference count overhead, improving instruction count performance in benchmarks.
  • pull/189897

3.2 Closed Pull Requests

This section provides a summary of pull requests that were closed in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Closed This Week: 672

Key Closed Pull Requests

1. [functorch][dim] Refactor test_split to use instantiate_device_type_tests for multi-device compatibility: This pull request refactors the test_split test cases in the functorch module to use instantiate_device_type_tests, thereby unifying multi-device test logic and enabling compatibility with multiple device backends without modifying core runtime code or changing original CPU/CUDA test behaviors.

• URL: pull/189833

• Associated Commits: 75e26, 153b4, d8689, 37ce4, 9cf93, d9a84, 1f91b, 0cb48, a5f77, 8aba9, 8b26a, 94891, 485b0, 8390d, a7992, 8752a, 0a64e, 9f591, b6f4a, dee41, 2a0c3, 53cde, e9c59, fef7a, 999bf, 3a069, c91f0, e4e91, e9478, ecf94, 4dc6a, 9afc2, febbc, b0269, b7fa5, f51bb, 039b4, 5d42d, 39989, 3129c, e7cb8, c2adb, 1c1d5, bf590, e9120, 6cb19, 01a35, fd203, 59e1e, 97046, f13a6, 5ca91, d0505, 9cbf2, 03985, d654c, ddb0b, ecdfc, a4322, 6f255, d4473, efc2c, e9c79, f3255, 1a5a8, 9d2d6, 5adb4, 8ce7e, 9f230, 0f57e, dbd09, 772a7, 51ea7, d2c35, 4b8b1, 0e678, c0de5, a8a58, a1fa4, 910d9, 7b369, 6c22b, ea52e, 645a3, d3075, 3eaf6, ec204, 65ab6, 7dd46, 6cc54, 7cce7, fb000, 8fe83, 76310, 41f63, 3cd77, 1cf17, a6fca, d4ea0, 8b6a6, dd0e3, 35600, f92e9, 6f4c9, d6581, 5b9b0, e99c6, 9c1e0, d5640, 828c2, bf9ce, 98a14, 3583d, 7106d, 34921, c6bf4, d9fcf, 71f5a, ba311, 15320, 6a637, e4840, e1580, c4173, 88f22, 17ad7, 32538, acbf8, d8be6, a6b08, ba9df, 7e1ef, a58f8, 7ce9e, e307e, 4b0cf, 864e1, ccb29, 63902, a16f5, c2ed8, 0c394, 1b98c, 107af, 9b4e0, 5497b, 9e201, aac38, f1f8b, b2c3b, 3519e, 7123e, 54cf9, 5963d, b335e, 4b969, 23745, 6baf8, 9602d, 8728c, 7cf2e, 754f6, fb98f, fec8e, c0e08, 6420e, d8575, 631c1, 9bb35, f5697, 9a389, 379fb, f9640, 50ea5, 6afc6, ac10d, a36b9, ac81f, cb2d1, 77809, 48f01, a69c3, 884ec, 573b4, e6fdf, eac32, a2d67, f72b8, c4268, f1ba0, 22899, 9d32d, 9c69b, 9ad3f, 1e740, 63f43, 2406f, 873ba, 6b58c, 69879, 5863d, 72b0c, 3aa99, f6ab3, b2951, 7a73d, 6ba6b, 82807, a8180, f8124, 7e672, f6245, 0ec52, 925d8, 47ea0, 2f802, f7515, 54e77, 82313, b076a, 365b3, bfb56, 00d96, e78ab, 11f9f, e5d6e, f3db6, 20cdb, 86ad4, 8bec5, 58342, c78df, 4db08, 61a63, 345ba, efde2, bce5a, a0bf2, 15a37, 969bf, 6e319, 5b49d, 0fc11, cd4bd, 2a6ce, 326b8, 8f407, 76587, 2d75d, 7fc56

• Associated Commits: 75e26, 153b4, d8689, 37ce4, 9cf93, d9a84, 1f91b, 0cb48, a5f77, 8aba9, 8b26a, 94891, 485b0, 8390d, a7992, 8752a, 0a64e, 9f591, b6f4a, dee41, 2a0c3, 53cde, e9c59, fef7a, 999bf, 3a069, c91f0, e4e91, e9478, ecf94, 4dc6a, 9afc2, febbc, b0269, b7fa5, f51bb, 039b4, 5d42d, 39989, 3129c, e7cb8, c2adb, 1c1d5, bf590, e9120, 6cb19, 01a35, fd203, 59e1e, 97046, f13a6, 5ca91, d0505, 9cbf2, 03985, d654c, ddb0b, ecdfc, a4322, 6f255, d4473, efc2c, e9c79, f3255, 1a5a8, 9d2d6, 5adb4, 8ce7e, 9f230, 0f57e, dbd09, 772a7, 51ea7, d2c35, 4b8b1, 0e678, c0de5, a8a58, a1fa4, 910d9, 7b369, 6c22b, ea52e, 645a3, d3075, 3eaf6, ec204, 65ab6, 7dd46, 6cc54, 7cce7, fb000, 8fe83, 76310, 41f63, 3cd77, 1cf17, a6fca, d4ea0, 8b6a6, dd0e3, 35600, f92e9, 6f4c9, d6581, 5b9b0, e99c6, 9c1e0, d5640, 828c2, bf9ce, 98a14, 3583d, 7106d, 34921, c6bf4, d9fcf, 71f5a, ba311, 15320, 6a637, e4840, e1580, c4173, 88f22, 17ad7, 32538, acbf8, d8be6, a6b08, ba9df, 7e1ef, a58f8, 7ce9e, e307e, 4b0cf, 864e1, ccb29, 63902, a16f5, c2ed8, 0c394, 1b98c, 107af, 9b4e0, 5497b, 9e201, aac38, f1f8b, b2c3b, 3519e, 7123e, 54cf9, 5963d, b335e, 4b969, 23745, 6baf8, 9602d, 8728c, 7cf2e, 754f6, fb98f, fec8e, c0e08, 6420e, d8575, 631c1, 9bb35, f5697, 9a389, 379fb, f9640, 50ea5, 6afc6, ac10d, a36b9, ac81f, cb2d1, 77809, 48f01, a69c3, 884ec, 573b4, e6fdf, eac32, a2d67, f72b8, c4268, f1ba0, 22899, 9d32d, 9c69b, 9ad3f, 1e740, 63f43, 2406f, 873ba, 6b58c, 69879, 5863d, 72b0c, 3aa99, f6ab3, b2951, 7a73d, 6ba6b, 82807, a8180, f8124, 7e672, f6245, 0ec52, 925d8, 47ea0, 2f802, f7515, 54e77, 82313, b076a, 365b3, bfb56, 00d96, e78ab, 11f9f, e5d6e, f3db6, 20cdb, 86ad4, 8bec5, 58342, c78df, 4db08, 61a63, 345ba, efde2, bce5a, a0bf2, 15a37, 969bf, 6e319, 5b49d, 0fc11, cd4bd, 2a6ce, 326b8, 8f407, 76587, 2d75d, 7fc56

2. [DO NOT MERGE] Wrong PR Opened: This pull request was mistakenly opened and not merged, as indicated by its title "[DO NOT MERGE] Wrong PR Opened" and the absence of a body or merge status.

• URL: pull/190408

• Associated Commits: 2ceed, d80a5, c649f, 4b8a5, 5decb, a2813, 98e35, ef5c6, 013fd, d70d8, 883a7, 9bb65, 8fcce, 6fe4b, 57614, c67ec, 9e94d, 195f9, f99ab, f95d7, dc12b, 63fcb, 5d919, 3c404, f31ba, 052ff, bac7b, e04dd, e2fa2, 41f8e, 036b2, fa384, 76fd0, 7f2cd, 483b5, db741, 3e05c, 70d99, 8ecd5, e0fbd, ccba6, 7a10d, f5a3a, a5c71, 8f496, 4e323, 0446f, 85430, 52064, 141ba, 50bfd, 0320c, 3aaa9, 48211, e16e3, 1a6ad, 9413e, 7d37b, 345ca, 44360, 52236, 96bfe, 0c210, 5f7b0, 6c683, 53679, 515e8, a1f91, 98563, 9022c, 484cd, dadd2, d0c8b, 05b2a, 975e8, da1ed, 17dcd, 17c69, 858f2, ffdb7, e2aeb, 15ced, 52331, a5cae, 39218, 3b959, 9c287, 22869, 86083, aef1f, 8f317, ce208, dc843, 77070, 2a81d, 3aaed, d614a, 53056, f92d7, eb687, 92e6d, 32f4d, e8e6f, 8a06d, ea2ec, 4a964, 3de38, 993ac, 71258, 9df77, fccd3, 07102, e2cfd, 68519, 08bd5, 67d66, b2b54, 590e2, 98f89, 6bc66, 8dfb7, 6f686, 44d32, 450be

• Associated Commits: 2ceed, d80a5, c649f, 4b8a5, 5decb, a2813, 98e35, ef5c6, 013fd, d70d8, 883a7, 9bb65, 8fcce, 6fe4b, 57614, c67ec, 9e94d, 195f9, f99ab, f95d7, dc12b, 63fcb, 5d919, 3c404, f31ba, 052ff, bac7b, e04dd, e2fa2, 41f8e, 036b2, fa384, 76fd0, 7f2cd, 483b5, db741, 3e05c, 70d99, 8ecd5, e0fbd, ccba6, 7a10d, f5a3a, a5c71, 8f496, 4e323, 0446f, 85430, 52064, 141ba, 50bfd, 0320c, 3aaa9, 48211, e16e3, 1a6ad, 9413e, 7d37b, 345ca, 44360, 52236, 96bfe, 0c210, 5f7b0, 6c683, 53679, 515e8, a1f91, 98563, 9022c, 484cd, dadd2, d0c8b, 05b2a, 975e8, da1ed, 17dcd, 17c69, 858f2, ffdb7, e2aeb, 15ced, 52331, a5cae, 39218, 3b959, 9c287, 22869, 86083, aef1f, 8f317, ce208, dc843, 77070, 2a81d, 3aaed, d614a, 53056, f92d7, eb687, 92e6d, 32f4d, e8e6f, 8a06d, ea2ec, 4a964, 3de38, 993ac, 71258, 9df77, fccd3, 07102, e2cfd, 68519, 08bd5, 67d66, b2b54, 590e2, 98f89, 6bc66, 8dfb7, 6f686, 44d32, 450be

3. Enable kineto XPU scope profiler extending existing XPU profiler plugin: This pull request introduces the XPU scope profiler by extending the existing XPU profiler plugin in PyTorch to enable gathering of XPU hardware metrics, incorporates a build-time option to enable or disable the profiler, enhances the kineto_shim.cpp tracing path, and adds corresponding tests to support profiling for the newly integrated PyTorch built-in XPU device.

• URL: pull/165766

• Associated Commits: 37cfe, 8b906, e5cd8, 80aab, 03334, eba2f, 62af1, 55bb8, b7cfe, f15f7, 93d55, 9eaa2, f0bf4, 810e0, d8ff9, 3f495, 87bf9, 19490, 2048a, af7a5, 6d762, 84ebc, 5c92d, 6f0f1, ab5e8, 35c9b, 79657, f1b4d, 4336c, a2ea8, 91de6, 6e8fd, b84f2, 02e86, 075ab, 47402, f6836, fa626, ce806, fce38, 02396, 5c322, 875d9, d2a1d, 28864, 24684, 10183, 01e76, 43997, b9879, d096c, d8997, be167, d7e2f, 8cb79, 316e3, 2021b, 8a0ce

• Associated Commits: 37cfe, 8b906, e5cd8, 80aab, 03334, eba2f, 62af1, 55bb8, b7cfe, f15f7, 93d55, 9eaa2, f0bf4, 810e0, d8ff9, 3f495, 87bf9, 19490, 2048a, af7a5, 6d762, 84ebc, 5c92d, 6f0f1, ab5e8, 35c9b, 79657, f1b4d, 4336c, a2ea8, 91de6, 6e8fd, b84f2, 02e86, 075ab, 47402, f6836, fa626, ce806, fce38, 02396, 5c322, 875d9, d2a1d, 28864, 24684, 10183, 01e76, 43997, b9879, d096c, d8997, be167, d7e2f, 8cb79, 316e3, 2021b, 8a0ce

Other Closed Pull Requests

• Generator Finalization and Exception Handling: Two pull requests focus on improving resource cleanup and exception handling in PyTorch. One explicitly closes all open generators in compile_subgraph to ensure proper execution of finally blocks using CPython's tp_finalize mechanism, while the other rewrites exception handling to better align with CPython's behavior and style.
  • [pull/157149, pull/188004]

• AOTAutograd and Inductor Compile to Python Enhancements: Multiple pull requests add and improve functions that compile AOTAutograd and Inductor graphs into standalone Python modules. These changes include adding torch._functorch.aot_autograd.compile_to_python and torch._inductor.compile_to_python to produce auditable, source-expressible Python code capturing backend contracts, with comprehensive testing across CPU and CUDA targets.
  • [pull/187859, pull/187858]

• Native Group Norm Backward Improvements: Several pull requests extend and add support for multiple gradients in the native_group_norm_backward function. These include adding a new overload to support gradients from all outputs, fixing precision issues for float16, and proposing MPS backend kernel support, although some remain unmerged pending prerequisite reviews.
  • [pull/183980, pull/186946, pull/186947, pull/174893]

• FlexGEMM and Local Reduction Enhancements: A set of pull requests enhance FlexGEMM epilogues by enabling grouped local reductions, supporting compressed local reduction auxiliary outputs, and extending local reduce auxiliary output groups with physical combine and finalize callbacks. These improvements ensure correctness, configurability, and performance optimizations across various grouped geometries and tile capabilities.
  • [pull/188112, pull/188469, pull/188470]

• Inductor Reduction and Pointwise Heuristics Refactoring: Two pull requests refactor Inductor heuristics by moving reduction configuration and pointwise autotuning logic into modular, class-based systems within triton_codegen. This enables device-scoped registries and cleaner inheritance patterns for CUDA, ROCm, and XPU devices without changing existing behavior.
  • [pull/183278, pull/183277]

• DTensor Redistribution and _StridedShard Support: Multiple pull requests improve DTensor redistribution by introducing CollectivePaddingContext to centralize padding logic, adding support for converting Shard to _StridedShard with all-to-all communication, enabling partial tensor operations for _StridedShard, and proposing uneven _StridedShard redistribution support respecting device order, though one was not merged.
  • [pull/171178, pull/171023, pull/171219, pull/171338]

• Test Suite Skips Migration to @skipOps Decorator: Two pull requests migrate device-agnostic test skips in the TestFwdGradients and TestMeta suites to use the @skipOps decorator. This improves test management and addresses related PyTorch issues.
  • [pull/184356, pull/184678]

• FakeTensor Migration and Lint Enforcement: One pull request updates the codebase by replacing all isinstance(t, FakeTensor) checks with the is_fake_tensor(t) function, adds support for C++ FakeTensor objects, introduces helper functions for attribute access, and enforces a lint rule to prevent future direct isinstance checks, preparing for the upcoming C++ FakeTensor migration.
  • [pull/188978]

• Portable RNG Flag for Performance Optimization: A pull request introduces a portable flag allowing users to opt out of certain stateless RNG API properties to maximize performance by using existing torch.rand() and torch.randn() kernels instead of new kernels with enhanced properties.
  • [pull/177882]

• Inductor NVGEMM Epilogue Fusion Proposal: A pull request proposes enabling epilogue fusions in the Inductor NVGEMM backend to optimize performance, but it was ultimately not merged.
  • [pull/174893]

• AlmaLinux Image Build Migration: One pull request migrates the build process for AlmaLinux images to use the egress-exempt mt-rel-* OSDC release runner with remote BuildKit mode, introducing a REMOTE_BUILDKIT path while maintaining the existing local docker-build path.
  • [pull/188626]

3.3 Pull Request Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed pull requests that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed pull requests from the past week.

---

IV. Contributors

4.1 Contributors

Active Contributors:

We consider an active contributor in this project to be any contributor who has made at least 1 commit, opened at least 1 issue, created at least 1 pull request, or made more than 2 comments in the last month.

If there are more than 10 active contributors, the list is truncated to the top 10 based on contribution metrics for better clarity.

Contributor	Commits	Pull Requests	Issues	Comments
guilhermeleobas	361	36	1	5
jansel	384	5	0	4
bobrenjc93	328	59	1	1
williamwen42	353	15	0	0
frgossen	199	48	44	7
drisspg	176	10	0	16
huydhn	122	19	0	7
d4l3k	123	17	1	5
ezyang	113	20	0	10
dolpm	135	7	0	0

Access Last Week's Newsletter:

• Link