Weekly GitHub Report for Llama.cpp

Thank you for subscribing to our weekly newsletter! Each week, we deliver a comprehensive summary of your GitHub project's latest activity right to your inbox, including an overview of your project's issues, pull requests, contributors, and commit activity.

Table of Contents

• I. News
  • 1.1. Recent Version Releases
  • 1.2. Other Noteworthy Updates
• II. Issues
  • 2.1. Top 5 Active Issues
  • 2.2. Top 5 Stale Issues
  • 2.3. Open Issues
  • 2.4. Closed Issues
  • 2.5. Issue Discussion Insights

• III. Pull Requests
  • 3.1. Open Pull Requests
  • 3.2. Closed Pull Requests
  • 3.3. Pull Request Discussion Insights

• IV. Contributors
  • 4.1. Contributors

I. News

1.1 Recent Version Releases:

The current version of this repository is b4991

1.2 Version Information:

The version released on March 29, 2025, introduces key updates that enhance overall performance and stability, with notable improvements in user interface responsiveness and security features. This release reflects a continued focus on optimizing user experience and safeguarding data integrity.

II. Issues

2.1 Top 5 Active Issues:

We consider active issues to be issues that that have been commented on most frequently within the last week. Bot comments are omitted.

1. [BUG-UNCONFIRMED] Eval bug: gemma4 thinking starts to emit progressivly long trailing garbage: This issue describes a problem with the gemma4 model emitting progressively longer trailing garbage output during its reasoning phase, sometimes causing a model error. The user reports that this occurs when the main agent calls a tool with incremental information, and the log shows unparsed output with trailing UTF-8 characters, which started appearing in builds around the 9800 range.

   • The comments discuss attempts to reproduce and diagnose the issue, with suggestions to trim whitespace in the chat template to prevent injected newline characters causing the model to spiral into generating garbage output; after applying this fix, the user confirms the problem is significantly improved and no longer produces nonsense or excessive token consumption.
   • Number of comments this week: 8

2. [CI / PACKAGING] Update vulkan CI runners to NVIDIA r615 driver: This issue requests updating the Vulkan CI runners to use the newly released NVIDIA r615 Linux driver, which includes a compiler fix for intermittent coopmat1 failures observed in continuous integration. The update aims to allow disabling older driver-specific workarounds once the new driver is deployed, improving CI stability.

   • The comments discuss compatibility concerns of the r615 driver with DGX Spark hardware, note that the T4 runners in Azure will be updated to r615, and confirm that the DGX Spark runner will soon be retired, allowing removal of legacy workarounds.
   • Number of comments this week: 7

3. [BUG-UNCONFIRMED] Misc. bug: ggml crash - ggml_backend_sched_alloc_splits: unexpected graph reallocation: This issue reports a crash occurring in the test-recurrent-state-rollback test of the llama.cpp project, triggered by an unexpected graph reallocation error inside the ggml backend scheduler during memory allocation. The user provides detailed environment info, logs, and test commands showing the failure happens with certain GGUF models and across different backends, and the root cause appears related to ggml's graph/memory allocation logic but remains unknown.

   • The comments discuss attempts to reproduce and isolate the issue across various backends (SYCL, CPU, Vulkan), testing different models, and build configurations; it is noted that the crash occurs in common ggml code rather than backend-specific code. Suggestions include testing with different parameters and disabling SYCL features, and a later comment indicates the issue may have been resolved in a newer commit, recommending rebuilding and retesting with updated code.
   • Number of comments this week: 7

4. [BUG-UNCONFIRMED] Eval bug: OpenVINO backend crashes with STATUS_ILLEGAL_INSTRUCTION due to AVX-512: This issue reports a crash occurring with the OpenVINO backend on an Intel Core Ultra 7 265K CPU due to the use of AVX-512 instructions, which the CPU does not support, resulting in a STATUS_ILLEGAL_INSTRUCTION error. The user describes attempts to run various models on Windows 11 with the OpenVINO backend, all failing silently or crashing, and a workaround involving replacing the OpenVINO build's CPU DLL with one from a normal llama.cpp build is suggested.

   • The comments confirm the crash is caused by unsupported AVX-512 instructions on the user's CPU, provide a detailed explanation of the error, suggest a practical workaround by swapping DLLs, and discuss which models work or fail with this approach, while also expressing frustration about the state of the OpenVINO backend development.
   • Number of comments this week: 6

5. [BUG-UNCONFIRMED] Misc. bug: Severe drop in prompt processing speed after b10780 on Vulkan, RDNA3: This issue reports a significant decrease in prompt processing speed on Vulkan with RDNA3 GPUs starting from build b10781 compared to b10780, with performance dropping by nearly half or more in some tests. The problem appears linked to a specific commit and is exacerbated by GPU memory pressure and graphical effects, causing lag and slower token processing, while reverting to b10780 restores the original performance levels.

   • The comments include requests for additional testing on different prompt sizes and configurations, detailed benchmark results confirming the slowdown on multiple RDNA3 GPUs, observations about increased GPU load and desktop lag during processing, and comparisons showing that neither batch size nor KV cache quantization fully explain the performance regression.
   • Number of comments this week: 6

2.2 Top 5 Stale Issues:

We consider stale issues to be issues that has had no activity within the last 30 days. The team should work together to get these issues resolved and closed as soon as possible.

As of our latest update, there are no stale issues for the project this week.

2.3 Open Issues

This section lists, groups, and then summarizes issues that were created within the last week in the repository.

Issues Opened This Week: 78

Summarized Issues:

• GPU Backend Crashes and Memory Errors: Multiple issues report crashes and memory errors across various GPU backends including Metal, CUDA, ROCm/HIP, Vulkan, and SYCL. These problems range from segmentation faults caused by missing kernels or illegal memory accesses to device resets and driver timeouts, often triggered by specific model configurations, large images, or unsupported hardware features.
• [issues/28565, issues/28581, issues/28582, issues/28602, issues/28608, issues/28624, issues/28652, issues/28661, issues/28676, issues/28684, issues/28722, issues/28726, issues/28728, issues/28760, issues/28765, issues/28768, issues/28805, issues/28807, issues/28808, issues/28812, issues/28826, issues/28840, issues/28844]

• Vulkan Backend Performance and Stability Issues: Several issues highlight performance regressions, GPU hangs, and crashes specific to the Vulkan backend on various GPUs including Intel Arc, AMD RDNA3, and Qualcomm Adreno. Problems include throughput collapses with increasing context length, device loss errors, and driver-specific shader compilation failures, often requiring driver updates or backend tuning.
• [issues/28635, issues/28637, issues/28648, issues/28659, issues/28721, issues/28807, issues/28808, issues/28812, issues/28840, issues/28841, issues/28843]

• CUDA Backend Bugs and Performance Degradations: Issues report CUDA-specific problems such as illegal memory accesses, nondeterministic embeddings, kernel launch misconfigurations, and performance drops due to fallback to CPU or inefficient kernel parameters. Some crashes are linked to specific GPU architectures or CUDA versions, with workarounds involving flag changes or driver upgrades.
• [issues/28581, issues/28624, issues/28633, issues/28652, issues/28656, issues/28661, issues/28734, issues/28756, issues/28760, issues/28761, issues/28790, issues/28877]

• Model Loading, Graph Scheduling, and Router Mode Failures: Several issues describe crashes or deadlocks during model loading, graph reservation, or router scheduling, often caused by assertion failures, race conditions, or improper atomic operations. These problems lead to failed requests, server errors, or permanent unresponsiveness requiring restarts.
• [issues/28614, issues/28698, issues/28774, issues/28777, issues/28829]

• Speculative Decoding and Slot Save/Restore Issues: Problems with speculative decoding include severe slowdowns due to inefficient CUDA kernels and slot save/restore functionality that fails to persist draft-model contexts, causing loss of KV cache and undermining performance benefits.
• [issues/28619, issues/28790]

• Quantization and Model Output Bugs: Issues report incorrect or garbage outputs from quantized models, bugs in LoRA adapter initialization, and unexpected reasoning output artifacts such as self-referential logic or trailing garbage characters. Some bugs involve token detection errors causing premature termination of reasoning blocks.
• [issues/28581, issues/28674, issues/28675, issues/28679, issues/28801, issues/28827]

• Threading and CPU Affinity Problems: Some issues describe incorrect default thread counts on hybrid CPUs and tools ignoring CPU thread pinning parameters, leading to suboptimal CPU utilization during server or benchmark runs.
• [issues/28625, issues/28650]

• Build, Compilation, and Linker Errors: Multiple issues report build failures due to undeclared identifiers, undefined symbols, or linker errors related to OpenMP and HIP, often triggered by specific compiler versions or environment configurations.
• [issues/28718, issues/28814]

• User Interface and Usability Requests: Requests include adding UI features for model selection in chat title summarization, displaying build/version numbers in interfaces and logs, and improving prompt size handling by truncation rather than hard failure.
• [issues/28703, issues/28773, issues/28848]

• Backend-Specific Memory and Resource Management Issues: Problems include excessive memory allocations in SYCL backend, GPU driver resets due to feature enabling, and memory corruption or invalidation in CUDA graph caches, often causing crashes or performance degradation.
• [issues/28608, issues/28778, issues/28860]

• Router and API Key Forwarding Bugs: Bugs in router mode cause premature eviction of requested models and failure to forward API keys to child server instances, resulting in failed client requests despite successful model listing.
• [issues/28698, issues/28820]

• Multi-GPU and Cross-Backend Inference Issues: Issues describe silent corruption or incoherent output when using multi-GPU setups or running multiple backends (e.g., Vulkan and Hexagon) on the same physical device, indicating untested or unsupported code paths.
• [issues/28676, issues/28891]

• Tokenization and Preprocessing Bugs: The UGM tokenizer incorrectly ignores byte fallback in SentencePiece vocabularies, and BPE pre-tokenizer patterns with Unicode escapes cause model loading failures on Windows builds.
• [issues/28788, issues/28862]

• Deadlocks and Server Unresponsiveness: The MCP server can deadlock on large payloads, requiring process respawn, and router deadlocks occur when requesting non-resident models with limited model slots, causing hanging or failed requests.
• [issues/28723, issues/28829]

• Performance Regression and Optimization Proposals: Reports include performance regressions in speculative decoding and prompt processing speed, with suggestions for adaptive kernel tuning, tile size adjustments, and skipping redundant operations to improve throughput and reduce decode time.
• [issues/28734, issues/28752, issues/28841, issues/28843, issues/28867]

• Crash and Error Handling in Specific Models and Features: Crashes occur with certain models or features such as DFlash2 tensor splitting, DeepSeek V4 Flash, and MTP speculative decoding, often requiring fallback or patching to avoid failures.
• [issues/28777, issues/28826, issues/28790]

• Inference and Generation Control Issues: The backend sometimes fails to halt GPU usage after frontend inference stops, causing high GPU utilization, and prompt queueing can accept inputs exceeding context limits leading to hard failures.
• [issues/28838, issues/28848]

• Hardware-Specific Bugs and Driver Issues: Problems include Vulkan driver bugs on Qualcomm Adreno, GPU hangs on AMD Radeon RX 5600 XT with large batch sizes, and illegal instruction crashes on AMD Zen 5 CPUs due to incorrect instruction set usage.
• [issues/28635, issues/28807, issues/28808, issues/28756]

• Segmentation Faults and Race Conditions on ARM and Android Devices: Segfaults occur in CPU compute paths on Android/aarch64 devices and in flash attention kernels on Android with long prompts, often related to buffer size mismatches or threading race conditions.
• [issues/28798, issues/28878]

• Model Modularization and Codebase Maintainability: A proposal suggests decentralizing model registration and making models more self-contained to reduce frequent modifications to central source files and improve maintainability.
• [issues/28772]

2.4 Closed Issues

This section lists, groups, and then summarizes issues that were closed within the last week in the repository. This section also links the associated pull requests if applicable.

Issues Closed This Week: 110

Summarized Issues:

• Model Loading and Compatibility Issues: Several issues report failures or errors when loading specific models due to missing tensors, unsupported quantization types, or backend incompatibilities. These problems cause crashes, failed initializations, or require workarounds such as using forks or disabling features to successfully load and run models.
  • issues/23033, issues/25809, issues/26073, issues/26339, issues/28831

• Backend and GPU Inference Bugs: Multiple issues describe crashes, corrupted output, or performance regressions related to Vulkan, CUDA, HIP, SYCL, and Metal backends on various GPUs. These include device lost errors, illegal memory accesses, assertion failures, and output corruption caused by backend-specific bugs or hardware limitations.
  • issues/24177, issues/24734, issues/26188, issues/26195, issues/26197, issues/26205, issues/26208, issues/26213, issues/26246, issues/26314, issues/26363, issues/26366, issues/26384, issues/26845, issues/27849, issues/28285, issues/28256, issues/28296, issues/28745, issues/28247

• Server and API Functionality Bugs: Several issues highlight bugs in the llama-server related to API endpoints, model management, file uploads, and server modes. Problems include incorrect API key handling, failure to reload or delete models, improper handling of multiple servers or tools, and dynamic timestamp regeneration causing caching issues.
  • issues/20475, issues/21779, issues/23460, issues/23948, issues/24761, issues/26169, issues/26266

• Tool Calling and Agent Mode Issues: Multiple bugs affect tool calling workflows and agent modes, including parsing failures with natural language before tool calls, tool functionality breaking in router mode, and malformed or hanging outputs due to parameter handling errors. These issues disrupt multi-turn workflows and tool integrations.
  • issues/20260, issues/24992, issues/25887, issues/28522

• Performance and Throughput Regressions: Several reports describe performance degradations in decoding, model loading, and inference throughput caused by inefficient CPU or GPU utilization, synchronization overhead, or pipeline regressions. These issues result in slower processing times and reduced hardware efficiency.
  • issues/25804, issues/25890, issues/25908, issues/26200, issues/26285, issues/28863

• Security Vulnerabilities: There are critical security issues including instruction injection via query parameters and symlink traversal in slot save APIs, which could allow arbitrary command execution or file overwrites outside designated directories, posing significant risks to server integrity.
  • issues/25790, issues/26315

• Accessibility and UI Bugs: Problems affecting user accessibility and UI behavior include dropdown menus inaccessible to screen readers, incorrect scaling of images in Chromium browsers, and missing UI options when API keys are set, which degrade user experience and accessibility compliance.
  • issues/24680, issues/24701, issues/28794

• Compilation and Build Failures: Several issues report build errors due to missing includes, incompatible compiler flags, or linker problems across different platforms and toolchains, causing failed builds or crashes during compilation.
  • issues/26186, issues/26340, issues/28557, issues/28559, issues/28560, issues/28561, issues/28562, issues/28564, issues/28491, issues/28758

• Quantization and Tokenization Bugs: Issues include corrupted quantization results due to in-place math errors, invalid embeddings from multi-output batch decoding, and tokenization inconsistencies in API endpoints, which affect model accuracy and output correctness.
  • issues/24888, issues/26142, issues/26309, issues/28438

• Feature Requests for Model and Backend Support: Requests include adding support for new large models, quantization formats, tokenizer improvements, and Vulkan backend optimizations to enhance performance and expand hardware compatibility.
  • issues/22477, issues/26115, issues/26125, issues/26139, issues/26305, issues/26335, issues/26360, issues/26365, issues/27194

• Crash and Stability Issues: Various crashes occur due to assertion failures, illegal memory accesses, division by zero, or silent failures during server initialization and runtime, leading to unrecoverable errors and server termination.
  • issues/24932, issues/25061, issues/26106, issues/26128, issues/26268, issues/26282, issues/26366, issues/26246, issues/26384, issues/28537, issues/26205, issues/28744

• Model Reasoning and Output Issues: Problems include premature deactivation of reasoning features, reasoning models not exhibiting expected behavior, and output corruption or silent errors during generation, which degrade model functionality and user experience.
  • issues/25067, issues/25511, issues/25724, issues/26312

• Cache and Memory Management Problems: Issues with key-value cache configurations, video frame caching, and memory allocation on GPUs cause degraded performance, corrupted outputs, or incorrect reuse of data, impacting inference quality and efficiency.
  • issues/26356, issues/28580, issues/26208, issues/28296

• Miscellaneous Bugs and Requests: Other issues include bugs in template engines causing capability detection failures, requests for private security reporting channels, and enhancements to benchmarking tools for better error handling.
  • issues/28786, issues/26355, issues/26236

2.5 Issue Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed issues that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed issues from the past week.

III. Pull Requests

3.1 Open Pull Requests

This section provides a summary of pull requests that were opened in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Opened This Week: 105

Key Open Pull Requests

1. OpenVINO: optimize stateful decode and GPU MoE inference: This pull request optimizes stateful key-value cache handling with improved sliding-window and per-layer head layouts, adds support for cacheless encoder models on NPU, enhances GPU Mixture of Experts (MoE) inference through compressed expert fusion and grouped 8-bit re-quantization, introduces OpenVINO weight spilling and configurable K-quant re-quantization, and extends support for CPY, RoPE, normalization, and pooling operations to improve performance and compatibility in the OpenVINO backend.

• URL: pull/28638

• Associated Commits: a6f29, 47ac7, 06bf5, 09cd2, 9fa36, d548e, a5651, 952a0, 0a47a, 91676, 33078, 04651, 85d9b, c0c2d, a3e8b, e7aee, e61e2, 39ad5, 0fe1a, aa5c8, 9d047, 41d3c, 8d3d6

2. mtmd : support CPU_REPACK extra buffer types for encoder weights: This pull request introduces support for CPU_REPACK extra buffer types in the encoder weights of the mtmd module, enabling the creation of encoder graphs that utilize these extra buffers to improve CPU performance by repacking tensors, while including fallback mechanisms, exclusions for certain models due to current kernel limitations, and new command-line options to control repacking behavior.

• URL: pull/28563

• Associated Commits: 154c0, a6419, 67106, 9a551, 48a46, 17c32, a44e4, d5f34, 4c739, f4e81, f9326, b4014, 6c257, 37712, 8468e, 80d07, cbe81, efd07, ea689

3. server: Add support for binding to multiple addresses: This pull request adds support for binding the llama-server to multiple IP addresses and Unix socket paths simultaneously, allowing the server to remain accessible through both VPN and localhost by accepting a comma-separated list of addresses via the --host option, creating one listener per address that shares the same API and model through a shared HTTP worker pool, while ensuring all addresses bind successfully or the startup fails.

• URL: pull/28690

• Associated Commits: 5a1ff, 6f130, 8a710, a0de4, cb990, 90a0a, 0ba1e, f99f9

Other Open Pull Requests

• Continuous Integration and Testing Improvements: Multiple pull requests enhance the continuous integration pipeline by adding new test cases such as nrc = 2 with KleidiAI self-hosted runners, enabling Vulkan and WebGPU support in Hugging Face jobs, and allowing a subset of CUDA server pytest tests to run on pull requests. These changes improve test coverage, stability, and efficiency of the CI process.
  • pull/28850, pull/28712, pull/28714

• OpenCL and GPU Backend Enhancements: Several pull requests add and optimize OpenCL and GPU backends, including initial OpenCL support for q2_K and q3_K matrix multiply types, fixing OpenCL GEMV kernel bugs, improving HIP kernel performance with fp32 accumulation, and optimizing Vulkan backend performance on RDNA3 integrated GPUs. These updates improve hardware support and computational efficiency across various GPU architectures.
  • pull/28577, pull/28572, pull/28573, pull/28576

• Quantization and FP8/NVFP4 Support: Pull requests introduce support for quantization scales in GGML for FP8 and NVFP4 data types, add Vulkan validation for these types, and enable conversion of compressed-tensor mixed-precision NVFP4 checkpoints to the gguf format. These changes facilitate simpler backend implementations and broader compatibility with new quantization formats.
  • pull/28898, pull/28611

• Text-to-Speech (TTS) Model Integration: Multiple pull requests add support for new TTS models, including KaniTTS-2 based on LFM2 and NeMo Nano Codec, and Soprano audio generation with the Soprano-1.1-80M model. These contributions include model integration, speaker reference audio encoding, and documentation updates to enhance TTS capabilities.
  • pull/28609, pull/28607

• MetaX GPU Backend and CUDA Optimizations: Pull requests add a runnable base inference backend for MetaX GPUs with fallback mechanisms and disable unvalidated NVIDIA features, and implement CUDA kernel fusion combining dense Q4_K Gate and Up projections with SwiGLU activation to optimize performance on NVIDIA Ampere-or-newer GPUs. These efforts improve GPU backend support and accelerate inference.
  • pull/28694, pull/28702

• Server and Model Handling Improvements: Several pull requests fix server behavior by correctly applying the --lora-init-without-apply flag, improve model file selection by preferring GGUF files matching directory names, add a new small-test preset with a pruned Qwen3.5-0.8B multimodal model for CI, and optimize causal attention handling to avoid unnecessary scheduler re-reserves. These changes enhance server reliability, model management, and performance.
  • pull/28707, pull/28809, pull/28751, pull/28746

• RDNA3.5 and NVIDIA Blackwell GPU Performance Optimizations: Pull requests implement batched WMMA operations and MMQ tile size adjustments for RDNA3.5 GPUs to improve matrix multiplication performance, and add pipelined NVFP4 MMQ tile loads using cp.async and TMA on NVIDIA Blackwell GPUs, achieving significant speedups and reduced stalls. These optimizations target specific GPU architectures for enhanced throughput.
  • pull/28714, pull/28815

• Build and Configuration Fixes: Pull requests update CMake configurations to correctly use CMAKE_CURRENT_SOURCE_DIR for precompiled headers, enforce OpenSSL as a required package when enabled to prevent silent failures, and modify CUDA backend tensor initialization to use per-thread streams to fix ROCm parallel stream capture issues. These fixes improve build robustness and backend stability.
  • pull/28725, pull/28685, pull/28771

• Model Conversion and Architecture Support: A pull request adds conversion support for the DeepSeek V4.1 model by subclassing the V4 architecture and handling key differences such as nested text parameters and new tensor naming conventions, preparing for future runtime support. This extends model compatibility within the project.
  • pull/28696

• SYCL Port of CUDA Graph Functionality: One pull request ports CUDA graph recording and replay to SYCL, addressing SYCL-specific challenges like data reordering and GEMM limitations, resulting in significant decode performance improvements on the A770 platform while keeping the feature off by default on other hardware.
  • pull/28725

• RPC Buffer and Cache Optimization: A pull request modifies the RPC buffer tensor hashing to restrict hash caching to weight tensors only, preventing activations and other large data from being cached, which significantly reduces disk usage and cache growth during server inference.
  • pull/28789

3.2 Closed Pull Requests

This section provides a summary of pull requests that were closed in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Closed This Week: 181

Key Closed Pull Requests

1. hexagon: support for multi-device model split (aka row-split): This pull request adds support for a new multi-device model parallelism mode called row-split in the Hexagon backend, enabling internal splitting of model tensors across multiple NPUs without requiring ALLREDUCE operations, while also improving async backend APIs and pipeline-parallel execution, fixing related bugs, and updating user and developer documentation accordingly.

• URL: pull/28589

• Associated Commits: 71fcf, 0abdc, e1345, dd241, f7a6d, 89aa7, fbd86, cd3f0, 20c1f, 35c9f, 9bdfc, 60244, 428e3, 165a8, 5ddb1, 01ace, 1ad87, d2995, a48a6, 165fb, 26b76, 446c7, ca234, b0c23, 7c742, c5ea8, 78cd3, e4df2, 007b1, 7c22c, 20707, 3c10d, 93091, 2dbb1, 2dea4, 03a5f, bd6cc, 50064, b08a8, 295f7, 132aa, 64761, 94eea, ca94e, fa1f7, 0e458, 01ee3, 8c983, 08054, e40f2, 90050, d57cb, d896f, 97d0a, ecc59, 11638, 3c3ca, eb202, b752c, b70b1, 1966a, a7624, d1194, 2833f, f733d, 250e3, 2a8cb, 9b9bd

2. Pure macro 3c826: This pull request introduces advanced recurrent layer features including custom routing and harmonic step scaling, implements a universal autotuner script, optimizes memory management and compute overlap for MoE models, fixes various compilation and runtime issues, and updates documentation to support these enhancements in the llama.cpp project.

• URL: pull/28663

• Associated Commits: 470b1, bea2e, 0c73a, 56820, dcc88, 82d78, 8e297, 09e66, 57d05, 26d51, 07f7e, a3572, c1506, 0c35d, eca00, ba04a, 4bcc0, 60164, 3a4fe, fae7c, e52dd, cc453, 99b6f, e4be9, e0f92, 6f4af, e7aa9, fe08e, 4320c, 927c2, 0dcca, 717dd, d1a8f, dce77, ec261, ac6bd, e2f7d, 4e6db, 3c826, 5f0e6

3. model : fix k-pool indexer softmax gridDim.y overflow: This pull request addresses a CUDA limitation by reshaping the k-pool gate logits before applying the softmax operation to prevent the gridDim.y overflow when n_new exceeds the CUDA cap of 65535, thereby fixing prompt processing aborts at large key-value sizes (n_kv >= 262144) in the k-pool indexer.

• URL: pull/28730

• Associated Commits: 74bb0, a7716, 5728a, c35bd, 3bdb2, 81f95, 3498c, 611e4, 1eca2, 7de5a, 9fe9f, a386c, 5b05f, 120eb, d0c55, 8c289, a2f1d, fe318, 2b533, ff6be, ad74e, 1b564, 99fda, db547, 5c4bd, cc05b, 81341, 05f7c, ae989, 3f238, 26ace, de253, 79de1, 798d6

Other Closed Pull Requests

• Metal backend fusion and cache optimization: Multiple pull requests improve the Metal backend by consolidating fusion patterns into a single fixed pattern table and implementing gated_delta_net cache fusion that routes the KV cache pointer directly into the kernel. These changes eliminate redundant copying, fix output indexing bugs, and improve performance on Apple M3 CPU models while adding comprehensive fusion regression tests.
• [pull/28164, pull/25788]

• JSON schema handling refactor: A pull request refactors JSON schema handling by introducing a new internal representation called common_schema to improve type resolution and streamline the json-schema-to-grammar process. This removes redundant Python implementations and enhances parser efficiency and flexibility.
• [pull/28736]

• Build system improvements: Updates to the CMake build system add precompiled headers for the Parsing/Frontend compiler stage, significantly improving build times with minimal and full build scripts. Additional build fixes include adding missing headers to fix clang and musl compiler errors and enhancing the Nix flake for llama-cpp-rocm with support for ROCm and CUDA builds.
• [pull/28091, pull/28566, pull/28583, pull/28566, pull/28555]

• Kernel and shader optimizations: Several pull requests enhance kernel performance and compatibility by adding Vulkan specialization constants for matrix multiplication shaders, consolidating shared memory tables, and fixing CUDA grid dimension overflow issues. These changes reduce the number of required shaders, improve tuning flexibility, and prevent launch aborts during long context rebuilds.
• [pull/25773, pull/28729]

• Continuous integration and testing enhancements: Continuous integration tests are extended to cover i8mm kernels with specific row chunk scenarios, and sanitizer builds are improved by ensuring actual test runs and removing slow or irrelevant tests. These updates make testing more comprehensive and efficient.
• [pull/16234, pull/28583]

• Multi-GPU and memory optimizations: The Qwen4 next flash implementation is optimized with chunked QSA prefill, fixes for QSA corruption, improved memory residency, and CUDA kernel improvements to reduce buffer overhead and prevent out-of-memory errors during long-context attention.
• [pull/28623]

• Hexagon RoPE optimizations: Comprehensive updates to the hexagon RoPE include vectorization of components, in-place RoPE for all scenarios, enforcement of kernel parameters, and removal of legacy timers. These changes result in significant speedups across multiple modes while maintaining full test coverage.
• [pull/28628]

• Ternary quantization support on CUDA: Support for the TQ2_0 ternary quantization format is added to the CUDA dequantize path and MMVQ vector dot product implementation, enabling correct GPU execution of ternary models. The update also includes HIP compatibility checks and depends on prior Maple architecture additions.
• [pull/28769]

• Ubuntu release packaging with CUDA support: A release job is proposed for Ubuntu 22.04 that packages the llama.cpp library with CUDA 12.4 support, including a standard CUDA-enabled build and a standalone version that removes the need for large third-party dependencies, streamlining installation and compilation.
• [pull/16773]

• Tokenizer and parser updates for MiniCPM5: Support for the MiniCPM5 BPE tokenizer is added to fix garbage output and enable handling of MiniCPM5 XML tool calls by updating tokenizer logic, chat templates, and parser to correctly process attribute-keyed XML formats.
• [pull/24743]

• Vulkan and documentation updates: Multiple commits update the master branch with Vulkan top-k radix implementation, README enhancements for clarity and formatting, synchronization with upstream changes, and alignment of the Halo Box section with related forks, though this has not been merged.
• [pull/28629]

• OpenCL backend bug fixes for Adreno GPUs: Several bugs are fixed including device version checks to prevent aborts on OpenCL 2.0 devices, skipping kernel dispatches with zero global work sizes to avoid crashes, declining non-contiguous CONV_2D operands to prevent incorrect results, and improving memory allocation robustness with retries and fallbacks.
• [pull/27630]

• Server child process handling refactor: The server's child process management is refactored from a multi-threaded approach to a single-threaded poll()/select() model to improve IO-bound child server efficiency, after discarding a C++20 coroutine proof of concept.
• [pull/28555]

• Flash Attention implementation for SYCL backend: A basic Flash Attention implementation is introduced for the SYCL backend, enabling more efficient attention computation on Intel GPUs with a new kernel, block-wise forward pass, and integration with existing SYCL infrastructure supporting F32 data types.
• [pull/16969]

• Scheduler graph error handling improvement: The llama_params_fit function is modified to propagate scheduler graph allocation or splitting failures as a return value instead of aborting the process, allowing callers to handle errors gracefully.
• [pull/26292]

• Maple 20B-A1B ternary MoE architecture addition: The Maple 20B-A1B ternary Mixture of Experts architecture is added as a CPU-only model with 24 layers, 256 experts, sliding-window and global attention, ternary quantization support, and includes converters, constants, and tests ported from DeepGrove's implementation.
• [pull/27000]

• Meta backend context allocation fix: An issue where using --spec-draft-device CUDA0 with -sm tensor unnecessarily creates a Meta backend context consuming VRAM is fixed by adding a special case to prevent redundant allocation.
• [pull/28390]

• Bounded recurrent-state rollback for Kimi-K3: Support for bounded recurrent-state rollback during speculative decoding is added for the Kimi-K3 model by saving convolution windows and KDA state snapshots, adding Kimi-K3 to the rollback allowlist, and validating with extensive testing and performance measurements.
• [pull/28466]

3.3 Pull Request Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed pull requests that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed pull requests from the past week.

IV. Contributors

4.1 Contributors

Active Contributors:

We consider an active contributor in this project to be any contributor who has made at least 1 commit, opened at least 1 issue, created at least 1 pull request, or made more than 2 comments in the last month.

If there are more than 10 active contributors, the list is truncated to the top 10 based on contribution metrics for better clarity.