Weekly GitHub Report for Llama.cpp

Thank you for subscribing to our weekly newsletter! Each week, we deliver a comprehensive summary of your GitHub project's latest activity right to your inbox, including an overview of your project's issues, pull requests, contributors, and commit activity.

Table of Contents

• I. News
  • 1.1. Recent Version Releases
  • 1.2. Other Noteworthy Updates
• II. Issues
  • 2.1. Top 5 Active Issues
  • 2.2. Top 5 Stale Issues
  • 2.3. Open Issues
  • 2.4. Closed Issues
  • 2.5. Issue Discussion Insights

• III. Pull Requests
  • 3.1. Open Pull Requests
  • 3.2. Closed Pull Requests
  • 3.3. Pull Request Discussion Insights

• IV. Contributors
  • 4.1. Contributors

I. News

1.1 Recent Version Releases:

The current version of this repository is b4991

1.2 Version Information:

The version released on March 29, 2025, introduces key updates that enhance overall performance and user experience, focusing on improved stability and new feature integrations. Notable highlights include streamlined workflows and expanded compatibility with emerging technologies.

II. Issues

2.1 Top 5 Active Issues:

We consider active issues to be issues that that have been commented on most frequently within the last week. Bot comments are omitted.

1. [BUG-UNCONFIRMED] Eval bug: [Vulkan] GGML_ASSERT(wg0 <= ctx->device->properties.limits.maxComputeWorkGroupCount on Intel Arc A770 when running Qwen 3.8 flash next: This issue reports a crash caused by a failed assertion in the Vulkan backend of the GGML library when running inference with the Qwen 3.8 Flash Next model on Intel Arc A770 hardware. The assertion failure occurs because the dispatch size for a fill operation exceeds the maximum allowed compute workgroup count on Intel GPUs, which is triggered by large batch and context sizes leading to oversized tensor fills in the model's sparse attention mask.

   • The comments confirm the bug on multiple Intel Arc GPUs and characterize the failure as a limit on the maximum compute workgroup count in Vulkan dispatches, specifically in the ggml_vk_fill operation used for the top-k sparse attention mask. A patch was developed to split the dispatch into a 2D grid to stay within hardware limits, which successfully fixes the crash without performance loss. Users verified the fix on different hardware and workloads, and also discussed performance tuning considerations related to batch size, context length, and VRAM usage on Intel GPUs.
   • Number of comments this week: 13

2. Eval bug: SYCL backend aborts at model load on Lunar Lake iGPU (Arc 140V) — device memory query fails with both Level Zero and SYCL APIs: This issue reports a crash occurring when loading models using the SYCL backend on an Intel Lunar Lake iGPU (Arc 140V) due to failure in querying device memory size via both Level Zero and SYCL APIs, causing the backend to abort without fallback. The user provides a detailed investigation including environment setup, reproduction steps, and a workaround patch that falls back to system RAM when device memory queries fail, enabling successful model loading and execution.

   • The comments reveal multiple users experiencing similar SYCL backend memory query failures on various Intel GPUs and driver versions, with attempts to use legacy environment variables and driver updates proving partially effective; a regression was identified in a specific commit, and ultimately a fix was merged to address the memory query issue on iGPUs.
   • Number of comments this week: 12

3. [HIGH PRIORITY] docker: no SemVer tags / release builds missing: This issue addresses the lack of semantic versioning (SemVer) tags for Docker images in the GitHub Container Registry (GHCR), which complicates pulling images for specific stable releases since only build number tags are available. It also highlights that not all builds, including stable releases like version 0.3.0, have corresponding container images due to the daily cron-based build schedule, and suggests workflow improvements to ensure stable release builds always have published images.

   • The comments discuss the inconvenience caused by the current tagging system and propose solutions including modifying workflows to automatically tag Docker images with SemVer tags upon release, adding new workflows triggered by tag pushes, and potentially adding cron jobs to verify image availability; there is consensus on testing these approaches and some users are preparing to implement and try out the proposed changes.
   • Number of comments this week: 9

4. [BUG-UNCONFIRMED] Eval bug: unpack8() corrupts MAT_MUL + CPY on Qualcomm Snapdragon X Elite: This issue reports a problem with the unpack8() function corrupting matrix multiplication and copy operations on the Qualcomm Snapdragon X Elite platform, resulting in garbage outputs and occasional hangs during inference with quantized models. The user has identified that the problem likely stems from numerical inaccuracies in Vulkan backend operations, particularly affecting MUL_MAT and CPY operators, and suspects it to be a driver bug rather than an issue within the ggml library itself.

   • The comments discuss detailed error measurements confirming significant numerical errors, note that a previous AMD workaround does not resolve the issue on Qualcomm hardware, and suggest that this is likely a driver bug that should be reported to Qualcomm; maintainers express willingness to investigate further and reproduce the issue, while the original reporter plans to contact Qualcomm and consider submitting a PR if needed.
   • Number of comments this week: 9

5. CUDA: qwen35 MTP (draft 4) gives ~1.06x on RTX 5090 (sm_120) vs ~1.8x on RTX 4090, plain decode on native Linux is bandwidth-normal: This issue investigates the performance discrepancy of token generation for the qwen35 architecture on NVIDIA RTX 5090 (sm_120) GPUs running Windows compared to RTX 4090 (sm_89) on Linux, where the 5090 achieves only about 28–45% of the memory-bandwidth bound versus 86% on the 4090. The discussion reveals that much of the deficit on the 5090 under Windows is due to host-side overhead and degraded PCIe link conditions rather than CUDA kernel inefficiency, while the multi-token prediction (MTP) path shows limited speedup on the 5090 compared to the 4090, suggesting a bottleneck in the MTP verify step or its overlap on sm_120 hardware.

   • The comments detail extensive benchmarking across different hardware, OS environments, and PCIe configurations, identifying that restoring a stable Gen5 x16 link improves but does not fully close the performance gap; running under Linux userland (WSL2) recovers much of the Windows overhead; native Linux on sm_120 hardware shows bandwidth-normal decode efficiency; and the MTP path on sm_120 yields only a modest speedup compared to the 4090, with kernel traces attributing the bottleneck to the verify step’s matrix-vector multiplications at higher column widths.
   • Number of comments this week: 8

2.2 Top 5 Stale Issues:

We consider stale issues to be issues that has had no activity within the last 30 days. The team should work together to get these issues resolved and closed as soon as possible.

As of our latest update, there are no stale issues for the project this week.

2.3 Open Issues

This section lists, groups, and then summarizes issues that were created within the last week in the repository.

Issues Opened This Week: 105

Summarized Issues:

• Backend and Hardware Compatibility Issues: Multiple issues report crashes, incorrect outputs, or failures related to specific hardware backends such as CUDA, Vulkan, SYCL, Metal, and HIP on various GPUs and platforms. These problems include memory access violations, unsupported operations, driver bugs, and backend-specific kernel errors that cause instability or degraded performance across different devices and operating systems.
• [issues/28111, issues/28112, issues/28124, issues/28126, issues/28132, issues/28134, issues/28142, issues/28144, issues/28158, issues/28160, issues/28165, issues/28166, issues/28180, issues/28185, issues/28193, issues/28214, issues/28239, issues/28241, issues/28247, issues/28251, issues/28252, issues/28282, issues/28312, issues/28314, issues/28328, issues/28338, issues/28342, issues/28343, issues/28353, issues/28357, issues/28360, issues/28361, issues/28370, issues/28376, issues/28384, issues/28416, issues/28425, issues/28433, issues/28438, issues/28441, issues/28448, issues/28484, issues/28487, issues/28513, issues/28515, issues/28519, issues/28524, issues/28527, issues/28537]

• Prompt Cache and Slot Management Bugs: Several issues describe problems with prompt cache usage and slot restoration in llama-server, including skipped cache hits due to NaN calculations, failure to reuse cached conversation states, livelocks during context checkpoint erasure, and inconsistent outputs caused by cache reuse or improper cache sizing. These bugs lead to redundant computation, slower prefill times, or corrupted prompt handling.
• [issues/28139, issues/28194, issues/28276, issues/28280, issues/28368, issues/28495]

• Model Loading and Build System Errors: There are multiple reports of build failures, compile errors, and model loading problems caused by hardcoded paths, missing dependencies, incorrect version reporting, or unsupported model architectures. These issues affect various platforms and complicate development and deployment workflows.
• [issues/28114, issues/28291, issues/28295, issues/28397, issues/28400, issues/28505, issues/28546]

• Performance Regressions and Bottlenecks: Some issues highlight significant slowdowns or reduced throughput in specific configurations, such as Vulkan backend on AMD GPUs, CUDA on Windows, or speculative decoding with tensor splits. These performance degradations are linked to recent code changes, inefficient memory usage, or scheduling problems.
• [issues/28160, issues/28196, issues/28218, issues/28256, issues/28495]

• Memory Management and Allocation Failures: Several issues report out-of-memory errors, illegal memory accesses, heap corruption, or memory query failures across different backends and hardware. These problems cause crashes, process aborts, or degraded inference quality and often require workarounds or fallback mechanisms.
• [issues/28112, issues/28142, issues/28239, issues/28338, issues/28357, issues/28448, issues/28495]

• Quantization and Numerical Accuracy Problems: Issues include silent numerical corruption, sign loss during quantization, out-of-bounds token IDs during decoding, and non-reproducible results due to race conditions or uninitialized memory in quantization shaders. These bugs affect model output correctness and stability.
• [issues/28111, issues/28124, issues/28158, issues/28343, issues/28438]

• Speculative Decoding and Mixture of Experts (MoE) Issues: Problems reported include output degeneration with tensor splits, cross-slot content contamination with draft-mtp parallel execution, CUDA crashes with MoE models, and requests for more granular control of active experts per layer. These affect model correctness and efficiency during complex decoding scenarios.
• [issues/28185, issues/28286, issues/28251, issues/28380]

• Server and RPC System Bugs: Issues include deadlocks with multiple RPC endpoints, tool call validation errors rejecting optional parameters, and bugs in chat format parsing causing parameter collisions. These affect server stability and correct handling of client requests.
• [issues/28340, issues/28429, issues/28487, issues/28522]

• User Interface and Tooling Problems: Reports include Linux web UI adding extra prompt tokens, web UI failing to apply config on first launch, repeated 403 errors causing IP bans, and localhost UI persisting after process exit. These issues degrade user experience and require manual workarounds.
• [issues/28167, issues/28260, issues/28299, issues/28444]

• Feature Requests and Enhancements: Requests include adding support for new models (XingChen4-29b, Kimi-K3), deferred tool loading, bounded recurrent-state rollback, new tensor types for reproducible dot products, and native media generation support. These aim to extend functionality and improve flexibility.
• [issues/28153, issues/28179, issues/28461, issues/28470, issues/28541]

• Shader and Kernel Compilation Failures: Some issues describe shader compilation errors on specific GPUs or platforms, including internal shaderc errors and unsupported Vulkan features causing crashes or hangs. These prevent successful model execution on affected devices.
• [issues/28234, issues/28342, issues/28502]

• Cross-Platform and OS-Specific Bugs: Problems include missing DLLs on Windows, AVX-VNNI feature detection failures, illegal instruction errors due to OS support, and GPU driver incompatibilities on ARM or Apple Silicon platforms. These affect portability and performance across environments.
• [issues/28290, issues/28291, issues/28295, issues/28328, issues/28441, issues/28505]

• Context and Batch Size Handling Errors: Bugs include batch size zero causing assertion failures, draft-mtp context sizing errors, and batch-dependent CUDA crashes, leading to server termination or incorrect decoding behavior.
• [issues/28433, issues/28525, issues/28241]

• Tensor Splitting and Load Balancing Issues: The tensor split implementation floors split points causing imbalance and out-of-memory errors, impacting multi-GPU model fitting and inference stability.
• [issues/28506]

• Miscellaneous Bugs: Other issues include URL parsing errors dropping query parameters, SVG image export missing elements, and incorrect tool parameter limits causing JSON output corruption.
• [issues/28400, issues/28336, issues/28522]

2.4 Closed Issues

This section lists, groups, and then summarizes issues that were closed within the last week in the repository. This section also links the associated pull requests if applicable.

Issues Closed This Week: 103

Summarized Issues:

• GPU Backend Performance and Stability Issues: Multiple issues report performance regressions, crashes, and incorrect outputs related to GPU backends such as Vulkan, CUDA, HIP, and Metal on various hardware including AMD, NVIDIA, and Apple GPUs. These problems include degraded token generation speeds, memory bandwidth inefficiencies, kernel launch failures, and hardware-specific bugs causing crashes or corrupted outputs, often requiring workarounds or kernel tuning to mitigate.
• [issues/20029, issues/21284, issues/23774, issues/23909, issues/24438, issues/25582, issues/25652, issues/25866, issues/25870, issues/26220, issues/26462, issues/26746, issues/27021, issues/27189, issues/28113, issues/28135, issues/28151, issues/28201, issues/28377, issues/28404, issues/28436, issues/28453, issues/28454, issues/28455]

• Model Loading and Compatibility Bugs: Several issues describe crashes, assertion failures, or incorrect behavior during model loading due to metadata errors, unsupported architectures, tensor offset mismatches, or incompatible backend support. These include problems with specific models like Gemma 4, Qwen3.5, Spark-X2.5, Bonsai, and Internlm, often requiring fixes in model parsing, metadata handling, or backend support to enable successful loading and inference.
• [issues/24015, issues/24309, issues/25739, issues/25833, issues/25857, issues/25882, issues/25884, issues/28188, issues/28197, issues/28207, issues/28300, issues/28317, issues/28318, issues/28345]

• Tensor Parallelism and Multi-GPU Execution Issues: Problems related to tensor parallelism and multi-GPU setups include crashes during model warmup, assertion failures due to incompatible tensor splits, race conditions, and unstable multi-GPU execution with CUDA tensor splits. These issues highlight challenges in managing tensor views, buffer ownership, and synchronization across devices, leading to aborts or stalls in distributed inference scenarios.
• [issues/24015, issues/27750, issues/28047]

• Quantization and Kernel Implementation Bugs: Issues cover bugs and inefficiencies in quantization formats, kernel implementations, and fused attention paths, including buffer overflows, incorrect memory accesses, and fallback to slower CPU paths without warnings. These affect performance and correctness for quantized models and specialized kernels, especially on AMD and CUDA backends.
• [issues/18864, issues/25858, issues/28135, issues/28151, issues/28454, issues/28455]

• Inference Output and Tokenization Errors: Several bugs cause corrupted or incorrect inference outputs, such as garbage tokens, repeated punctuation, incorrect tokenization of special tokens, and fabricated reasoning outputs. These issues impact the quality and reliability of generated text and chat interactions, sometimes bypassing input sanitization or causing reasoning parser desynchronization.
• [issues/17459, issues/24382, issues/25889, issues/28113, issues/28182]

• Memory Management and Resource Utilization Problems: Reports include memory leaks, inefficient VRAM usage, inconsistent free VRAM calculations, and GPU memory not returning to idle states after inference. These problems degrade system stability and performance, especially on AMD hardware and HIP backends, and complicate resource management during long-running or multi-turn sessions.
• [issues/24438, issues/24684, issues/24836, issues/25939, issues/28436]

• Server Crashes and Parsing Failures: Multiple issues describe server crashes caused by stack overflows, infinite recursion in grammar parsing, denial-of-service vulnerabilities triggered by malformed requests, and failures in JSON-schema-to-grammar conversion due to duplicate or unparseable rules. These bugs affect server robustness and reliability during chat completions and tool call processing.
• [issues/25755, issues/25958, issues/25959, issues/25960, issues/25967]

• Web UI and Tool Call Interface Bugs: Problems include missing UI elements, broken image rendering in tool call results, incorrect tool call grammar generation, and unwanted tool calls causing errors or permission requests. These issues degrade user experience and complicate interaction with the llama-server WebUI and tool integration.
• [issues/22178, issues/25742, issues/25744, issues/25746, issues/25789, issues/28157]

• Build and Compilation Issues: Several issues report build failures, hangs during asset provisioning, environment variable misconfigurations, and backend-specific compile errors affecting Linux, Windows, and SYCL builds. These problems hinder development and deployment workflows across platforms and require updates to build scripts or toolchains.
• [issues/25429, issues/25808, issues/25985, issues/26060]

• Performance Bottlenecks on CPU Architectures: Issues highlight compute-bound decode and prefill operations on Arm Neoverse-N2 CPUs, with GEMV kernels dominating CPU cycles and limiting throughput despite high memory bandwidth, indicating kernel-level performance constraints.
• [issues/25976, issues/25977]

• Model-Specific Feature Requests and Enhancements: Requests include adding support for new quantization formats, model architectures, next-generation sampling methods, and options for optimized loading of large MoE models on low-end GPUs, aiming to improve compatibility and performance.
• [issues/23909, issues/25855, issues/26020, issues/27501]

• Security and False Positive Malware Detections: Reports of Windows Defender falsely flagging llama.cpp binaries as malware, causing quarantine and blocking execution, which affects user trust and software distribution.
• [issues/28224, issues/28315]

• Miscellaneous Bugs and Edge Cases: Various other issues include incorrect handling of audio input in reasoning mode, hardware-specific Vulkan extension requests causing failures, and bugs in specific vector dot product implementations leading to incorrect numerical results.
• [issues/25889, issues/28161, issues/28281]

2.5 Issue Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed issues that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed issues from the past week.

III. Pull Requests

3.1 Open Pull Requests

This section provides a summary of pull requests that were opened in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Opened This Week: 166

Key Open Pull Requests

1. metal : single-source fusion table + gated_delta_net cache fusion: This pull request reworks the Metal backend's operation fusion by consolidating all fusion patterns into a single fixed pattern table used consistently by both the graph optimizer and op encoders to prevent discrepancies, fixes output indexing bugs, introduces a new fused pattern that combines the gated_delta_net kernel with cache copy to optimize recurrent state snapshot writes by eliminating redundant copies, and adds a comprehensive fusion regression test tool that validates fusion correctness and performance against a committed per-device baseline.

• URL: pull/28164

• Associated Commits: 299d3, 9bd8b, 17dfd, 48c9b, 5f4d7, 8e3b9, 6b3a6, c92d7, 61e1a, 69845, 6cb9a, 8af25, ba232, 2d43f, 50f4f, 91aaa, 22905, 12d76, e451d, ad283, c2a79, eeba4, 6ef12, c5806, 98f29, 29991, 0bf10, 68700, 5fb4f, 05900, 2e8ff, a050b

2. CI : Add ubuntu-cuda builds to release.yml: This pull request adds Ubuntu CUDA build jobs to the release CI workflow in order to close the publication gap by publishing Ubuntu packages, including necessary dependent libraries like NCCL, while optimizing package size and ensuring proper licensing.

• URL: pull/28186

• Associated Commits: 84337, ed948, 1db1a, 7339d, bd101, 7d9dc, 140db, ed020, ef878, 237c1, ec92d, 26625, 75ab6

3. Enables Windows ARM64 build with MSVC cl.exe: This pull request enables building the project on Windows ARM64 systems using the MSVC cl.exe compiler by modifying the CMake configuration and source code to address MSVC-specific limitations and intrinsic support gaps, allowing developers to build natively without requiring clang, while noting some pending intrinsic support and performance differences compared to the clang build.

• URL: pull/28362

• Associated Commits: 43e49, 039dd, 050b9, 5fb9d, 5a05b, 537e7, 6267f, 620d5, 7d270, 77d81, 2047d, c518b, 18eaa

Other Open Pull Requests

• UI Reorganization and Component Modularization: This pull request reorganizes the UI by grouping the four model selector components into a dedicated models/ModelsSelector module with its own barrel, extracting shared elements like the reasoning panel and in-flight download row. It also relocates option-list helpers, integrates the selector hook with download and reasoning menu functionality, and updates searchable-dropdown features and related story imports to support this rework.
  • pull/28419

• Multi-Token Prediction (MTP) Enhancements: This pull request introduces enhanced MTP support for the Qwen3.8-Flash-Next model, enabling 1.3 to 2 times faster performance by reusing shared embedding tokens to save disk space and memory. It adds draft MTP head architecture, shared embedding borrowing, and improved loading mechanisms for draft-only exports, building upon prior work in the repository.
  • pull/28243

• Memory and Debugging Improvements on Windows WSL: This pull request fixes memory size detection issues on WSL for Windows by adding a fallback to the SYCL API when the Level Zero Sysman API is unsupported. It also improves debugging with a new GGML_SYCL_DEV_DEBUG feature and includes various code optimizations and build error fixes verified across multiple platforms.
  • pull/28227

• Batch Invariance Testing for Vulkan Backend: This pull request adds a comprehensive suite of tests to reproduce and isolate multiple batch invariance bugs reported in issue #25618, focusing on Vulkan backend failures compared to CPU. The goal is to enable incremental fixes while maintaining continuous integration stability.
  • pull/28488

• Peer-to-Peer Connection Mode for llama-server: This pull request introduces a new --connect mode enabling P2P connections between the web UI and server outside the local network using a privacy-focused WebRTC protocol with end-to-end encryption. It relies on a Rust-based pre-built binary for portability and minimal maintenance, deferring frontend implementation and macOS-x64 support to future updates.
  • pull/28277

• XingChen4 Model Support and Integration: This pull request adds support for the XingChen4 large language model by registering its architecture, implementing model weight conversion and quantization to GGUF format, and enabling CPU and GPU backend inference with specific operator implementations. It validates the integration through operator tests and performance evaluations.
  • pull/28156

• Memoization Cache for Grammar Parsing: This pull request implements a persistent, hand-tuned memoization cache for the reject_candidates_for_stack function in grammar parsing to eliminate branching explosion. This significantly improves performance and prevents hangs in complex grammar scenarios by caching results keyed on stack and candidate content across multiple token generations.
  • pull/28371

• Lazy Quantization for KV Cache: This pull request introduces a lazy quantization option for the KV cache to q8_0 format, where the cache initially uses a higher-precision fp16 view until it fills up, then quantizes down to q8_0. This balances precision and memory usage during model inference.
  • pull/28267

• MTP Processing with imatrix and Quantization Preferences: This pull request explores the impact of processing MTP with the imatrix, concluding minimal benefit. It recommends using Q4_K quantization when VRAM is limited and Q6_K otherwise, supported by performance and memory usage comparisons.
  • pull/28351

• CUDA0 Spec-Draft Device VRAM Optimization: This pull request fixes an issue where using --spec-draft-device CUDA0 with -sm tensor unnecessarily creates a Meta backend context consuming VRAM without use. It adds a special case to prevent this redundant allocation.
  • pull/28390

• Recurrent-State Rollback for Kimi-K3 Model: This pull request enables bounded recurrent-state rollback support for the Kimi-K3 model during speculative decoding by saving convolution windows and KDA state snapshots for each rollback position. It adds Kimi-K3 to the recurrent-state rollback allowlist and validates the implementation through extensive testing and performance measurements.
  • pull/28466

• Dependency and CI Configuration Updates: This pull request updates the ty dependency to version 0.0.78, includes several type-related fixes, and adjusts the CI configuration by adding the --exit-zero-on-warning flag.
  • pull/28548

• SVE 256-bit Vectorization for ggml Kernel: This pull request adds support for SVE 256-bit vectorization to the ggml_gemv_mxfp4_4x4_q8_0 kernel, improving performance in the GPT-OSS-20B MoE model on Graviton3e processors without affecting perplexity.
  • pull/28141

• Fixes for Undefined Behavior in String Handling: This pull request fixes undefined behavior in the string_strip function and related code by casting char values to unsigned char before passing them to ctype functions. This ensures correct handling of non-ASCII UTF-8 multibyte input and prevents out-of-bounds locale table access.
  • pull/28272

• CUDA MMQ Memory Access Fixes for MoE Models: This pull request fixes out-of-bounds memory access errors in the CUDA MMQ implementation for MoE models by clamping tile reads and buffer accesses to valid column ranges. It corrects padding calculations and adds tests to prevent illegal memory reads causing crashes on certain GPU configurations.
  • pull/28383

• Vulkan Backend MMVQ Path Selection Consistency: This pull request makes MMVQ path selection in the Vulkan backend independent of batch size by removing an early batch-size check. This ensures consistent computation paths for all batch sizes and prevents divergence in greedy speculative decoding results.
  • pull/28489

• Flash Attention Performance Improvements on AMD RDNA4: This pull request improves Flash Attention performance on AMD RDNA4 GPUs by tuning CUDA and HIP configurations, fixing a bug related to head size 256 in CUDA FA code, and optimizing kernel selection. These changes significantly enhance prefill throughput for large models like Qwen3.8 27B.
  • pull/28102

• Sparse Flash Attention Support in Vulkan Backend: This pull request adds support for sparse Flash Attention in the Vulkan backend, including implementation for DSV4/GLM, tuning, and tests.
  • pull/28105

• ROCm TOP_K Kernel Performance Optimizations: This pull request improves ROCm TOP_K kernel performance by adding a small-case kernel ported from Vulkan and optimizing various cases. These changes achieve significant speedups and time reductions across multiple test parameters.
  • pull/28313

• SynthID-Text Sampler for Watermarking: This pull request implements a SynthID-Text sampler enabled via --synthid-keys to embed a subtle, predictable distortion in model logits as a watermark for generated high-entropy text. It includes a script to verify the watermark's presence, aimed at legal or commercial use cases.
  • pull/28350

• Native Media Generation Support with LTX-2.3: This pull request introduces native support for image, video, and audio generation using the LTX-2.3 audio-video diffusion transformer. It adds a mediagen library and CLI tool enabling OpenAI-compatible endpoints, enhanced prompt conditioning, multi-backend support, server-side integration, and fixes for related ggml issues.
  • pull/28540

• Direct-Read Method for Lazy PLE Table: This pull request introduces a direct-read method for the lazy PLE table in the qwen4exp model, improving real-world prefill performance by 2 to 3 times on GB10 systems. It optimizes parallel pread() operations to reduce costly mmap over-reads, resulting in faster token processing without requiring PLE to be pinned in RAM.
  • pull/28136

• Crash Prevention via Allocation Failure Checks: This pull request addresses crashes in the ggml library by adding checks for allocation failures in ggml_gallocr_reserve_n and ggml_backend_sched_reserve. These checks ensure buffer allocation errors are properly handled to prevent SIGABRT and other crashes during graph allocation.
  • pull/28149

• Host Buffer Overrides and Memory-Mapped Model Loading: This pull request enables explicit host buffer overrides (e.g., CUDA_Host) to be preserved when using memory-mapped models, allowing CPU-resident expert layers in pinned memory for operator offload and increasing prefill throughput. It also fixes a loading performance issue by changing tensor loading from copying via mmap to direct file reads, drastically reducing cold load times on multi-GPU, multi-CPU systems.
  • pull/28223

• Verbosity Filter Fix in llama-bench Tool: This pull request fixes the verbosity filter in the llama-bench tool to ensure error-level logs (GGML_LOG_ERROR) are always shown regardless of verbosity settings. This addresses issue #28107 by allowing errors to bypass the filter.
  • pull/28229

3.2 Closed Pull Requests

This section provides a summary of pull requests that were closed in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Closed This Week: 199

Key Closed Pull Requests

1. Fix/sidecar runtime safety: This pull request addresses runtime safety improvements for the sidecar component by implementing validated fixes for sidecar n-gram verification width, safe multi-ubatch output materialization, and gated capability controls for GPU architectures, while also enhancing build workflows and documentation related to unified RDNA support.

• URL: pull/28359

• Associated Commits: fd940, 91797, 27457, becb8, 42e5a, c1050, f3597, 10355, a3a07, baf62, 31e6d, f97d2, 7346e, 53ad1, c1312, c38ac, 71c62, 9667e, b4129, 8d77a, 0a723, 7f813, 99679, 91221, 60f00, 0ebc3, c0d49, 135a9, 3f958, e3a9b, 2fa3b, 80919, 82a56, 92503, 7d0e6, 49fd7, af1ac, 08cee, fb144, 2d793, 00758, 844b1, d5295, 0a37f, fd8cd, 0ace3, 93e01, 14307, 95d09, 62e1c, 6631a, 30deb, 87b26, 7ba70, f1cdb, d6f43, 2f690, 4b23e, 47625, fecbe, 929bf, 46309, 0b5d7, 527a8, d74af, bf7fe, a6f7c, 56e71, 00b80

2. ui : Models Discover and Download (WIP): This pull request introduces a work-in-progress user interface layer for the "Discover & Download Models" feature, including a sidebar entry that opens a searchable two-pane dialog with curated and HuggingFace model catalogs, detailed model metadata and README views, a comprehensive download lifecycle with per-quant download controls and progress tracking, live integration with server-side download status, and a reorganized component structure to support these functionalities.

• URL: pull/28374

• Associated Commits: 2853c, 9a851, 4b544, c1cbd, 4c93c, f6f90, 4aa70, ddb8d, 7e63e, 548ac, af985, a0735, 0d229, 14ac9, 5b7a4, 5c648, 70f8d, b8f0e, 71a37, 8bf37, c43ce, c00a7, 41c67, 17375, 2043d, 46c1d, cc08b, 1ecfd, 7cbbd, b72af, 08f8b, c3839, 95bc4, 48aeb, fc682, 584b1, 0bbc2, cb2a9, b5ec1, 1d7dc, 8faff, 14e83, 57376, 258ae, 38358, 4ca4f, 1bc87, 66a71, 130b2, 8e7a9, 22476, 1bea3, 35ae8, 0c698, 03d45, e8704, ffbd6, 20769, 422b7, c4c30, 717db, 46f4d, 195be, 231cf, 9f070

3. [DRAFT] Support for Zaya1 8B model: This pull request introduces initial draft support for the Zaya1 8B hybrid recurrent/attention model into the llama.cpp project by porting and refactoring its architecture from the vLLM implementation, including convolution-based CCA layers to replace classic attention, adjustments for inference compatibility on GPUs like the RTX 4070 Ti, and multiple refinements to tensor constants and quantization handling to improve maintainability and backend support.

• URL: pull/23112

• Associated Commits: 1b785, eb8ea, 22db4, aa665, 5b8a2, 62977, 71f61, 3227d, 93d5a, 7fb1a, 313cb, b3a6a, b3eea, 59534, 3ace1, 42aff, c351c, 89676, 10403, e1f96, 98b4f, 3791a, d050e, e81be, ca65b, 57088, 50b2f, 64527, 2719c, 0aa6a, ac48c, 671c0, 84ed4, 3f3dc, 90487, 3750f

Other Closed Pull Requests

• OpenCL decode-side optimizations for Adreno GPUs: This topic covers multiple pull requests that implement advanced OpenCL optimizations for K-quantized dense models on Adreno GPUs, including tiled GEMV layouts, cooperative-K GEMM to reduce register pressure, multi-row f16 decode GEMV, and fused q4_K MUL_MAT+GLU decode operations. These changes are gated by per-device performance data and environment overrides to improve speculative decoding and multi-token prediction efficiency while maintaining correctness and compatibility.
  • pull/26477, pull/27632, pull/27633

• Bug fixes and correctness improvements in qwen4exp model: Several pull requests address critical correctness fixes for the qwen4exp model, including alignment of sparse-attention block selection, improved metadata validation, sequence copy indexing corrections, and fixes for CUDA aborts and image token position collapsing. These fixes ensure stable model behavior without changing perplexity and improve compatibility with CUDA and other backends.
  • pull/27879, pull/27941, pull/27977

• Vulkan backend enhancements and bug fixes: Multiple pull requests improve Vulkan backend operations by adding support for f16 B operands in coopmat1 quantized matrix multiplications, fixing GET_ROWS operation crashes due to misaligned tensor offsets, and resolving hard crashes on view tensors with non-zero offsets. These changes remove CPU fallbacks, add comprehensive tests, and optimize memory usage and performance on various GPUs.
  • pull/28348, pull/28253, pull/26854

• Mixture-of-Experts (MoE) model support and optimizations: This topic includes pull requests that add support for NVIDIA's Nemotron-3-Puzzle-75B-A9B model with per-layer heterogeneous MoE configurations, improve routed MoE prefill performance on RDNA3 GPUs by selecting MMQ N-tile sizes based on typical routed expert width, and add comprehensive support for the Tencent Hy 4 preview model with advanced MLA + MoE features.
  • pull/25444, pull/24546, pull/28127

• Sparse Flash Attention and CUDA kernel improvements: Pull requests in this group add sparse key-value index support in MMA flash attention kernels for CUDA, implement sparse Flash Attention support on Metal GPUs with optimized mask handling, and improve kernel fusion and buffer management for HMX and HVX kernels to enhance performance and reduce fragmentation.
  • pull/25917, pull/28098, pull/28202

• Quantization support and performance optimizations: This set of pull requests introduces support for TQ1_0 and TQ2_0 quantization types in the Vulkan backend, adds an AVX2-optimized GEMM panel for IQ-quantized models to speed up large batch prompt processing, and optimizes SYCL Q4_K multi-column matrix multiplication by reducing redundant weight reconstruction. These changes improve inference quality and performance across hardware.
  • pull/19743, pull/19743, pull/27062, pull/27765

• WebUI and server improvements: Pull requests here enhance the WebUI with configurable PDF handling, message history features, improved model selector interface, and optimized chat message rendering through lazy mounting and caching. Additionally, the server example is updated to start the HTTP listener and register health API endpoints before model loading to better report loading status and handle errors.
  • pull/28262, pull/28460, pull/6739

• Integer overflow and memory safety fixes: This topic covers fixes for integer overflow bugs in the ggml-metal backend that prevent deep-context corruption during large batched matrix multiplications, and fixes for out-of-bounds reads in OpenCL Adreno image kernels by clamping input fetch indices and enforcing strict tiling and layout requirements. These changes ensure safe memory access and prevent silent buffer overwrites.
  • pull/28210, pull/27632

• Automated code quality and testing: A pull request introduces a Pylint workflow for automated Python code analysis to improve code quality and maintainability across the project.
  • pull/19671

3.3 Pull Request Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed pull requests that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed pull requests from the past week.

IV. Contributors

4.1 Contributors

Active Contributors:

We consider an active contributor in this project to be any contributor who has made at least 1 commit, opened at least 1 issue, created at least 1 pull request, or made more than 2 comments in the last month.

If there are more than 10 active contributors, the list is truncated to the top 10 based on contribution metrics for better clarity.