Weekly GitHub Report for Llama.cpp: October 19, 2026 - October 26, 2026 (23:40:46)
Weekly GitHub Report for Llama.cpp
Thank you for subscribing to our weekly newsletter! Each week, we deliver a comprehensive summary of your GitHub project's latest activity right to your inbox, including an overview of your project's issues, pull requests, contributors, and commit activity.
Table of Contents
I. News
1.1 Recent Version Releases:
The current version of this repository is b4991
1.2 Version Information:
The version released on March 29, 2025, introduces key updates enhancing overall performance and stability, with notable improvements in user interface responsiveness and security features. These changes reflect a continued focus on optimizing user experience and safeguarding data integrity.
II. Issues
2.1 Top 5 Active Issues:
We consider active issues to be issues that that have been commented on most frequently within the last week. Bot comments are omitted.
-
[BUG-UNCONFIRMED] Compile bug:: This issue reports a compilation failure when building llama.cpp on a Linux server with AMD MI210 GPUs using the HIP backend, specifically encountering errors related to undefined identifiers and missing function matches in CUDA source files. The problem did not occur in version 0.6.0 but appears in later versions, with speculation that it might be related to ROCm version compatibility or required version checks for certain GPU features.
- The comments confirm the issue is reproducible and possibly linked to conflicting or missing HIP headers, with users discussing ROCm versions and suggesting that bumping the minimum ROCm version requirement might resolve the errors; one user also mentions having a patch generated by an AI assistant but defers to maintainers for the correct fix.
- Number of comments this week: 7
-
Eval bug: SYCL F16: NaN output on dense Qwen3.8-27B Q4_K_M since #29107 (IQ3 code reorder); fused GEMM off fixes it (Arc Pro B70): This issue reports a regression introduced by a specific commit that causes NaN or corrupted output when running the Qwen3.8-27B model with SYCL F16 enabled on Intel Arc GPUs, traced to a fused GEMM kernel reading reordered quantized weights incorrectly. The problem manifests as repeated or garbage tokens at any prompt length and is resolved either by disabling the fused GEMM path or applying a later fix that corrects the weight layout handling in the SYCL backend.
- Multiple users independently reproduced the bug across different Intel Arc GPUs and driver/compiler versions, confirming it is tied to the fused GEMM kernel and SYCL F16 mode; a workaround disabling XMX gather types restores correct output, and a subsequent patch (#30604) was verified to fix the issue without needing the workaround, with confirmations from several testers on various hardware and builds.
- Number of comments this week: 7
-
[SYCL] DeltaNet kernel produces token soup on long prompts (Qwen3.6, Qwen3.8, Nemotron): This issue reports that the DeltaNet SYCL kernel in llama.cpp produces incoherent and repetitive token outputs, described as "token soup," when processing long prompts (greater than approximately 2,000 tokens) on Intel Arc Pro B70 GPUs, while short prompts work correctly. The problem affects multiple hybrid DeltaNet models but not dense attention models, and the root cause is hypothesized to be related to precision and accumulation errors in the SYCL kernel's recurrent state computations, with a proposed fix involving fp32 accumulation and more precise math operations.
- The comments clarify that the long-prompt failure on SYCL with F16 off cannot be reproduced, and the known SYCL bug with F16 on causes garbage output at all prompt lengths; the long-prompt issue reported is actually observed on the Vulkan backend with a batch size dependency, suggesting two distinct bugs. Various users share detailed environment setups, test results, and hypotheses, concluding that the SYCL F16 bug (#30892) is separate from the Vulkan long-prompt failure, and further testing with different builds and batch sizes is recommended to isolate the Vulkan issue.
- Number of comments this week: 6
-
DeepSeek-V4.1-Flash (PR #30817): single-token output corruption when a cached prompt prefix is reused (Engram n-gram history): This issue describes a problem in the DeepSeek-V4.1-Flash model where reusing a cached prompt prefix causes single-token output corruption due to divergence in the Engram n-gram history, leading to incorrect token substitutions that accumulate with reused prefix length. The issue is reproducible with a specific test harness and can be temporarily mitigated by disabling prompt caching, but it affects both lazy and non-lazy modes and persists across different hardware and model configurations.
- Commenters confirmed the bug also exists in the main branch and discussed whether existing tests like
llama-contextshould catch it; detailed system specs and test logs were shared showing partial test failures related to state restoration and token generation mismatches, indicating the problem is complex and not fully resolved by current testing methods. - Number of comments this week: 5
- Commenters confirmed the bug also exists in the main branch and discussed whether existing tests like
-
[BUG-UNCONFIRMED] Eval bug: Significant performance difference between llama-prism and main llama.cpp builds (same base version): This issue reports a significant performance discrepancy between the llama-prism fork and the main llama.cpp repository when running the same model with identical parameters, where the prism fork achieves 2-3 times higher throughput. The user provides detailed benchmark results showing faster prompt evaluation and generation speeds in the prism fork compared to the main repo, suggesting either an optimization in the fork or a regression in the main codebase.
- The comments include a request for the llama-prism repository link, which was provided, clarification on the tested version, and recommendations to test with the latest llama.cpp release since the reported main repo version is outdated; another user reports opposite findings with the latest commit, indicating the prism fork performs worse, and suggests further testing with updated versions.
- Number of comments this week: 5
2.2 Top 5 Stale Issues:
We consider stale issues to be issues that has had no activity within the last 30 days. The team should work together to get these issues resolved and closed as soon as possible.
As of our latest update, there are no stale issues for the project this week.
2.3 Open Issues
This section lists, groups, and then summarizes issues that were created within the last week in the repository.
Issues Opened This Week: 76
Summarized Issues:
- Performance issues on AMD GPUs and Vulkan backend: Multiple issues report significant performance regressions and inefficiencies on AMD RDNA2 and RDNA4 GPUs using the Vulkan backend, including slowdowns due to strict shared memory limits, fallback to slower kernels for larger head sizes, and corrupted logits causing premature sequence termination. Workarounds such as enabling native Wave32 subgroup execution or flipping Vulkan device enumeration are mentioned to mitigate these problems.
- issues/30832, issues/30838, issues/30841, issues/30884
- Server crashes and assertion failures in llama-server: Several critical bugs cause llama-server crashes due to unchecked casts, unbounded recursion from nested JSON, assertion failures during model loading, and improper handling of unrecognized input types. These issues lead to process aborts, stack overflows, or request failures, severely impacting server stability and client experience.
- issues/30834, issues/30835, issues/30859, issues/30860
- Quantization and model conversion bugs: Problems during quantization and model conversion include crashes after quantizing to TQ2_0 format, incorrect token labeling in MoE converters, and fp16 overflow causing NaN logits in CUDA kernels. These bugs result in corrupted outputs, crashes, or malformed decoding, requiring fixes such as forcing fp32 accumulation or correcting token type assignments.
- issues/30844, issues/30879, issues/30950, issues/30951
- Backend-specific crashes and compilation failures: Multiple backends including HIP, SYCL, Metal, OpenCL, and CUDA experience crashes or compilation failures due to missing functions, unsupported instructions, or driver limitations. These issues prevent successful builds or cause runtime crashes on specific hardware or OS configurations, blocking usage on affected platforms.
- issues/30869, issues/30875, issues/30892, issues/30973, issues/30983, issues/30989, issues/31007, issues/31033, issues/31064
- Memory management and resource exhaustion problems: Issues include Vulkan buffers backed by system memory exhausting shared memory budgets, llama-server hanging during teardown without releasing VRAM, and memory leaks in OpenVINO backend causing growing private memory usage. These problems lead to crashes, degraded throughput, or system resource exhaustion during model execution or server operation.
- issues/30888, issues/30975, issues/31129
- Model output correctness and reproducibility bugs: Several reports describe corrupted or repetitive outputs caused by bugs in CPU backend recurrent state handling, mixing speculative decoding methods, or missing rescale conditions in model code. These defects cause incorrect token generation, duplicated code blocks, or repeated tokens, affecting model reliability and output quality.
- issues/30906, issues/31158, issues/31160, issues/30896
- GPU memory allocation and device compatibility issues: Problems such as ErrorOutOfDeviceMemory on Intel Iris Xe and Mali GPUs despite sufficient free memory, Vulkan crashes on Android Mali GPUs during queue submission, and OpenCL kernel compilation failures on Adreno GPUs highlight device compatibility and memory allocation challenges. These issues cause crashes or degraded performance on affected devices.
- issues/30920, issues/31052, issues/30983
- Performance regressions and optimization requests: Reports include a 7% prefill performance regression on Intel Arc GPUs, 35% slower MoE prompt processing due to SYCL backend changes, and requests for data layout modifications and IQ quantization optimizations to improve GPU memory access and inference speed. These highlight ongoing performance tuning needs.
- issues/30939, issues/31068, issues/31194, issues/31195
- Model architecture support and feature requests: Several issues request adding support for new architectures such as EAGLE-4, metis16, KailA, Raon-1, and Phonon TTS model integration, emphasizing the need for updated model code, conversion tools, and inference support to enable usage of emerging models and features.
- issues/30925, issues/31043, issues/31072, issues/31078, issues/31106, issues/31112
- Build and installation issues: Problems include missing PATH updates after installation scripts, Windows executables failing to load long-path GGUF files due to manifest issues, and outdated prebuilt CUDA binaries not reflecting recent commits. These hinder user setup and deployment.
- issues/30935, issues/30998, issues/31085
- Concurrency and request handling bugs: Bugs causing silent exits on concurrent POST requests, llama-server prompt restore failures due to cache slot handling, and stale batch reuse in image embedding tools lead to crashes, increased processing times, or incorrect results under concurrent or repeated usage scenarios.
- issues/30961, issues/31150, issues/31155
- Backend-specific numerical correctness issues: The Metal backend produces incorrect bf16 matrix multiplication results on older Intel GPUs, and the SYCL backend's CPY operation inconsistently matches CPU references for certain quantization formats, causing output errors and test failures. These correctness bugs affect model output fidelity.
- issues/31091, issues/31140
- Shader and kernel inefficiencies: A Vulkan shader loops inefficiently over expert IDs causing slow prompt processing, and SYCL backend fused-SDPA flash-attention optimization applies only to the first context, reducing performance for subsequent contexts. Fixes are proposed to improve throughput without affecting accuracy.
- issues/31123, issues/31163
- Compilation and linkage errors on Linux: Older glibc versions cause missing pthread linkage errors, and Level Zero SDK detection fails with newer oneAPI setups, requiring CMake fixes to enable successful builds on affected Linux systems.
- issues/31007, issues/31074
- JSON and grammar parsing bugs: Raw control characters in JSON Schema patterns are copied verbatim into grammar, causing parser failures and HTTP 400 errors when processing requests with such patterns, indicating a need for proper escaping or validation.
- issues/31199
2.4 Closed Issues
This section lists, groups, and then summarizes issues that were closed within the last week in the repository. This section also links the associated pull requests if applicable.
Issues Closed This Week: 100
Summarized Issues:
- Model Metadata and Token Handling: The omission of
eos_tokenandbos_tokenfrom Hugging Facetokenizer_config.jsonin GGUF metadata causes incorrect chat model templating and generation stopping behavior, leading to generation continuing past the end of assistant responses. This issue highlights the need to include these tokens as metadata keys to ensure proper generation control.
- Model Management API and UI Enhancements: There is a request for a model management API with endpoints for reloading, downloading, tracking, and deleting models, including handling cancellation and cleanup of partial downloads. Additionally, the ability to pre-select models in the WebUI using aliases instead of IDs is requested to improve user experience by enabling bookmarks and direct model selection.
- Model Loading and Backend Compatibility Issues: Several models fail to load or run correctly on specific backends or hardware, including Gemma-4-12B on OpenVINO due to shape inference errors, Gemma 4 on AMD gfx1151 with ROCm/HIP backend producing infinite reserved tokens, and Nemotron models failing due to incorrect tensor shape checks and RoPE dimension computations. These issues cause crashes, unstable outputs, or loading failures across platforms.
- GPU Backend Crashes and Performance Problems: Multiple crashes and performance degradations occur on GPU backends including CUDA, Vulkan, HIP, and SYCL. Examples include softmax kernel crashes on NVIDIA RTX 5090 due to shared memory bugs, Vega 8 Vulkan GPU context loss causing crashes, AMD Radeon RX 9070 XT Vulkan backend suffering severe token generation slowdown, and Vulkan backend crashes on AMD GPUs with fused kernels. These issues often involve driver mismatches, kernel launch failures, or resource limits.
- CUDA Kernel and Memory Errors: There are several CUDA-specific errors including illegal memory access crashes during flash-attention with Qwen3.6-35B MoE, invalid configuration argument errors in RMS-norm kernel during concurrent connections, and out-of-memory errors causing hard aborts in top-k operations. These errors cause server crashes or degraded stability under certain workloads or configurations.
- issues/26609, issues/27835, issues/28338, [issues/27911](https://github.com/issues/27911]
- Multi-Turn Prompt and State Management Bugs: Enabling Multi-Token Prediction (MTP) or multi-turn prompt features causes state leakage, corruption, or non-deterministic outputs, leading to progressive degradation of model quality or repeated incorrect outputs. These bugs affect multi-turn tool-calling workflows and require server restarts or workarounds to mitigate.
- Prompt Caching and Adapter Handling Issues: The server incorrectly reuses cached prompt key-value pairs across requests with different LoRA adapters, contaminating outputs silently. Additionally, pinning requests to empty slots causes prompt cache skipping, leading to unnecessary full prefill processing and degraded performance.
- Grammar and Parsing Failures: Bugs in grammar generation and parsing include invalid grammar from empty object schemas or large maxLength values causing tool-call grammar failures, incorrect copying of regex shorthand classes causing unparseable grammars, and unbounded mutual recursion in GBNF grammar-text parser causing stack overflows and server crashes.
- Memory Management and Leak Issues: Memory margin miscalculations cause overestimation of free GPU memory leading to offloading and performance degradation. Speculative decoding with DSpark causes VRAM memory leaks due to improper cache compaction, and unbounded dry_sequence_breakers arrays cause excessive CPU and memory usage during sampler initialization.
- Distributed and Multi-GPU Inference Bugs: Multi-GPU inference with SYCL backend crashes due to unimplemented peer-to-peer features, and multi-GPU tensor-split configurations cause hard GPU kernel deadlocks with MTP speculative decoding enabled. These issues prevent stable distributed or multi-GPU model serving.
- Model Output and Tool Call Failures: Some models fail to produce expected tool calls, either generating plain text instead of structured calls or silently losing calls due to parser incompatibilities. Additionally, certain chat templates do not enforce required tool calls, causing unconstrained prose generation and token budget waste.
- Backend Loading and DLL Issues: Backend DLL loading fails on Windows due to LoadLibraryW not searching DLL directories for dependencies, causing GPU acceleration to be disabled. Similarly, symbol lookup errors on Linux prevent model loading due to missing symbols in shared libraries.
- Quantization and Model Preset Handling: Quantization tool failures occur when converting 3D convolutional tensors due to size mismatches, and model presets do not treat different quantizations or tags independently, requiring manual modifications to section headers and identifiers.
- Tokenization and Detokenization Bugs: Tokenizers for GPT-4o and Tekken models incorrectly split combining marks, causing token ID mismatches with Hugging Face. The detokenizer removes spaces before punctuation incorrectly for some vocabularies, leading to output differences from Hugging Face tokenizers.
- Build and Compilation Failures: Compilation errors occur on various platforms including SYCL backend failures due to missing device arguments, CUDA object compilation errors on DGX Spark, and Clang/ROCm type mismatches requiring workarounds. Build system enhancements are also requested for SYCL to improve compiler detection and fix warnings.
- Server Stability and Deadlock Issues: The RPC system deadlocks when using three or more endpoints with the CPU backend, causing client hangs without errors. Additionally, recurrent state corruption in Qwen4exp architecture due to incomplete snapshot coverage blocks performance improvements and causes logits mismatches.
- Performance Regression and Optimization Requests: A regression increased first inference time from ~10s to over 60s on large CPU-resident MoE weights due to memory locking changes. Optimization requests include adding a new tensor type BPOSIT8 for reproducible dot products and improving ARM BF16 dot product performance using native instructions.
- WebUI and User Interface Bugs: The WebUI shows duplicate clear buttons in the sidebar search input due to overlapping native and custom buttons, and users cannot scroll horizontally to view the start of very long attachment text lines, limiting usability.
- Embedding and Interpretation Features: A feature request proposes adding a public API and tools for extracting per-layer hidden states during decoding to facilitate interpretability and analysis without private forks.
- Miscellaneous Bugs and Issues: Other issues include Vulkan performance degradation on MacPro6,1 with dual GPUs, silent EOS generation failures in Qwen3.8-27B with large prompts, and llama-gguf-split tool lacking input validation causing crashes or silent failures.
2.5 Issue Discussion Insights
This section will analyze the tone and sentiment of discussions within this project's open and closed issues that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.
Based on our analysis, there are no instances of toxic discussions in the project's open or closed issues from the past week.
III. Pull Requests
3.1 Open Pull Requests
This section provides a summary of pull requests that were opened in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.
Pull Requests Opened This Week: 127
Key Open Pull Requests
1. cuda : gather batched tensor reads into mapped host memory: This pull request improves the performance of batched tensor reads in the CUDA backend by introducing a gather kernel that consolidates multiple scattered GPU-to-CPU memory copies into a single efficient operation using mapped host memory, significantly reducing the number of cudaMemcpyAsync calls and stream synchronizations required during KV cache snapshot saves, thereby decreasing latency and increasing throughput in concurrent streaming scenarios.
- URL: pull/30986
- Associated Commits: e39a9, 2f72d, 00408, ffa5a, 4bd80, 5a51f, 9020f, 5acbc, 40238, 97a07, cc19c, 0c3c5, d6067
2. ggml-cpu: add SVE-optimized q5_0 and q5_1 dot-product kernels: This pull request adds SVE-optimized implementations for the q5_0 x q8_0 and q5_1 x q8_1 dot-product kernels in the ggml CPU backend, improving performance by using vector-length-specific code paths for efficient unpacking of quantization bits while retaining NEON implementations as fallbacks for CPUs without SVE support.
- URL: pull/30867
- Associated Commits: 9dd72, 7d8f5, ae65c, 445a8, 5c05d, e7a21, dda6b, fe18f, 81821, 7150f, 09d15, 971b0
3. hexagon: add conv2d support: This pull request adds support for f16 and f32 CONV2D operations to the Hexagon backend, including various optimizations such as double-buffered staging, implicit GEMM, and overflow prevention, and has been tested successfully on multiple Android and Windows on Snapdragon devices.
- URL: pull/30995
- Associated Commits: 873d7, 17d0a, 40c24, 292bd, fa9fa, 9c50d, b7fbd, 9b1e2, bba04, f5ab9, 99f95, 27d5a
Other Open Pull Requests
- Hexagon backend support and optimizations: This pull request adds support for the DSV4_HC_PRE and DSV4_HC_POST operations to the Hexagon backend, including optimizations such as double-buffered DMA, L2 prefetching, and kernel parameter precomputation. All related backend operation tests pass successfully, ensuring robust integration.
- Batched tensor copy API and backend-specific batching: This pull request introduces a generic batched tensor copy API replacing the 2D-specific API with support for arbitrary tensor batches using parallel arrays. It implements backend-specific batching for Metal and CUDA, updates context save/load operations, and includes comprehensive testing for various batch scenarios and buffer types.
- Vectorized and optimized reordered-layout dequantization kernels: This pull request expands vectorization and optimization of reordered-layout dequantization kernels for multiple quantization types by applying flat launches and wider vectorized memory stores. The changes yield bit-identical outputs with significant performance improvements across various models and quant types without affecting decode accuracy or perplexity.
- Mixture of Experts (MoE) cache CPU fallback and tuning: This pull request introduces the ability to run missed MoE cache computations on the CPU based on a configurable CPU share parameter and improves cache miss handling to optimize performance. It also adds a tuning script that benchmarks and recommends ideal cache size and CPU share settings to enhance decoding speeds.
- OpenCL support for q3_k format and Adreno GPU optimizations: This pull request introduces initial OpenCL support for the q3_k format with various kernel implementations and optimizations targeting Adreno GPUs. It references prior related work and includes AI-assisted contributions.
- SYCL communication and buffer management optimizations: This pull request optimizes the ggml_sycl_comm_exchange() function to increase prompt evaluation throughput by approximately 19% on Intel Arc Pro B70 GPUs. Improvements include better tensor split performance, replacing wait() calls with device-side barriers, and complementary buffer management enhancements without affecting perplexity.
- Model saving refactor and deterministic tensor generation: This pull request refactors model saving to eliminate unnecessary context creation and replaces per-tensor normal distribution draws with a shared pool of pre-generated draws for deterministic and faster tensor data generation. It also enhances logging by reporting saved model file sizes, resulting in significant speed improvements in tests.
- WebGPU backend throttling mechanism for prefill phases: This pull request implements throttling in the WebGPU backend to limit in-flight batch submissions to one during prefill phases, identified by specific graph characteristics, to prevent GPU queue congestion and desktop compositing blockage. Decoding phases remain unrestricted to maintain token throughput.
- MiniCPM-OCR-Jev model integration and decision model support: This pull request adds support for the MiniCPM-OCR-Jev model, a Qwen3.5-VL trunk variant designed to score OCR text against rendered images using a value head read at the last token. It integrates the model as a decision model accessible via the
POST /v1/systemoneendpoint with specific handling for scoring and rejection of text generation, updating related configurations and tests.
- GRAPH_COMPUTE RPC security enhancements: This pull request enhances the security of the GRAPH_COMPUTE RPC by adding rigorous validation and bounds checking to tensor descriptors. It rejects invalid buffer and view configurations, prevents overflow in tensor size calculations, and enforces shape contracts to mitigate remote code execution vulnerabilities.
- Mixed signedness integer dot product support on Vulkan: This pull request enables mixed signedness integer dot product support for MMQ on Vulkan devices by adding specialized
_sudotshader variants usingOpSUDot. It improves prompt processing speed by approximately 2.2x without affecting token generation and maintains compatibility with existing signed dot shaders and benchmarks.
- KV-cache performance optimization by retaining rotation constants on GPU: This pull request optimizes the kv-cache by retaining rotation constants (Hadamard static matrices) on the GPU to reduce host-to-GPU data transfers. This results in improved throughput on certain workloads and is verified through benchmarks and tests.
- Metal backend performance improvements: This pull request improves the Metal backend by optimizing the MXFP4 matrix-vector multiplication kernel using an fp16 bit decoding trick. It increases threadgroup utilization for contiguous unary and binary operations and parallelizes GLU row computations across multiple threadgroups to enhance single-token decode efficiency.
- SYCL DPAS flash attention kernel for Xe2 GPUs: This pull request adds a SYCL DPAS flash attention kernel performing efficient attention computations directly on quantized KV caches (q4_0, q8_0) for Xe2 GPUs. It significantly improves decode and small batch performance by leveraging inline vISA DPAS instructions, enabling up to 12.8x speedups while maintaining compatibility and providing an environment flag to disable the feature.
- UI enhancement for model details overlay: This pull request improves user experience by enabling model details to open in a global drawer overlay without navigating away from the current page. It ensures seamless interaction with the models selector and maintains focus and tooltip behavior across UI components.
- ggml-cuda build time optimization: This pull request optimizes ggml-cuda build time by reducing parser passes on headers in the nvcc front-end through batching translation units and pruning unused template instances. These changes result in over 20% faster builds on Linux and Windows without regressions.
- External PLE file loading and GGUF conversion support: This pull request introduces the capability to load external PLE files from separate GGUF files and adds conversion support to build both PLE-less and PLE-only GGUFs. This enhances compatibility with models such as Gemma4, Qwen3.8Next, and GLM5.3Flash.
- Small MoE cache handling improvements: This pull request improves handling of small mixture-of-experts caches by warning when caches are too small to hold a single token and switching from segmented LRU to plain LRU caching below approximately 1.8 times a token's size. It also implements a gradual ramp-up of the protected cache set to optimize performance and avoid regressions reported in issue #30637.
- Fix for zero divided by zero error in pinned slot handling: This pull request fixes an arithmetic comparison issue causing a zero divided by zero error when handling empty pinned slots. It ensures the cache update flag is correctly set to prevent reprocessing from cold instead of using cached data, resolving eviction and cache retrieval logic issues described in issue #28139.
- Delta_net_base convolution state refactor for speed improvement: This pull request refactors convolution state handling in the delta_net_base module to reduce copying, resulting in a 1-2% speed improvement.
- Paligemma model support with llama_batch_ext interface: This pull request adds support for the Paligemma model by leveraging the llama_batch_ext interface, enabling handling of both text and vision inputs within the same non-causal batch. This benefits applications such as medical fine-tuned variants and prepares for future FalconOCR integration.
- Sampling penalties optimization: This pull request optimizes sampling penalties computation by calculating direct-path penalty coefficients on the host to reduce the operation graph and fusing CUDA penalty kernels into a single kernel. This significantly decreases kernel launches per token and improves decoding speed, especially for smaller models, while maintaining correctness.
- BPE rank loading refactor: This pull request refactors code by extracting BPE rank loading logic into a separate function named
load_bpe_ranksto eliminate duplication across multiple tokenizer models and improve code readability and maintainability.
- --slot-attach feature for shared prompt prefix cells: This pull request introduces the --slot-attach feature enabling multiple slots to share prompt prefix cells via a unified KV cache. It improves efficiency by avoiding redundant copies, includes automatic fallback and compatibility checks for various memory types and models, and results in significant performance gains and reduced memory usage during concurrent sessions.
- CUDA graph shape keying and bounded caching: This pull request implements graph shape keying to support multiple CUDA graphs with different first node shapes and introduces bounded caching with frequency-based eviction to limit memory usage. These changes achieve measurable improvements in decoding throughput and resource consumption across tested devices.
3.2 Closed Pull Requests
This section provides a summary of pull requests that were closed in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.
Pull Requests Closed This Week: 221
Key Closed Pull Requests
1. hexagon: Op Fusion refactoring, optimizations and new fusions like BINARY_NX (back-to-back ADDs, etc): This pull request refactors and optimizes the ggml-hexagon operations by consolidating duplicate logic, enhancing the op fusion infrastructure with new fusion patterns like BINARY_NX for back-to-back ADDs and MULs, streamlining kernel parameter handling, simplifying operation dispatch and batching, improving GLU and UNARY operation plumbing, and fixing Flash Attention memory management, resulting in overall performance improvements of 20-30% across multiple supported Hexagon NPU platforms.
- URL: pull/30843
- Associated Commits: d76b2, 1b313, 129ba, 9b54f, 041ec, c8e0f, 56055, 8d6c7, 4a719, d71f2, a5c90, c0839, f6ca0, 95e20, 316b7, 6a20c, 5fe33, b8578, c5ae1, d9458, 0c247, 5bc4f, 8765b, b69a6, 545e3, 7b21c, fc23d, 60af4, d7638, 28e3a, 7d962, be751, f8aee, 32680, 05e91
2. Multi-column on Intel XMX matrix engines refactored: This pull request refactors the multi-column matrix-vector multiplication kernels for reordered K-quant and Q8_0 weights on Intel XMX matrix engines using the SYCL backend, optimizing performance for speculative decoding verification and short prompt batches on Qwen3.8 27B models by reducing redundant computations and improving throughput across various quantization types while maintaining numerical accuracy and passing extensive tests.
- URL: pull/29864
- Associated Commits: 6459c, 9bf0f, f383a, a09eb, 71831, c7268, cf4b0, c8db7, a6558, ea80e, 77f83, 10d73, 5e94c, e9949, 3e072, ed90f, 786dc, ad7dc, 1165a, b65df, 6a6c0, 1e3b0, 433ce, 97c9c, 9ac78, 34ca7, 6ce20, 5b3e0, 09f7b
3. server : add POST /tts endpoint for TTS models: This pull request adds a new POST /tts endpoint to the server that supports text-to-speech models with features including streaming audio output, multipart form-data uploads for speaker reference audio files, error handling for incompatible models, updated documentation, and routing enhancements to reflect TTS capability.
- URL: pull/26603
- Associated Commits: 499e2, 6ac1a, 83880, ae521, cf5d7, 4daa3, c450c, 3278c, d9d74, 2072d, 14cc5, 2e267, 9f26d, 435d4, b95fc, 00e37, da4ae, 575c9, 5fa9b
Other Closed Pull Requests
- P2P Connection Mode in llama-server: This pull request introduces a new
--connectmode enabling peer-to-peer connections between the web UI and the server outside the local network using a privacy-focused WebRTC protocol with end-to-end encryption. It relies on a pre-built Rust binary to maintain portability and minimize dependencies.
pull/28277
- SenseNova U1 Model Support: This pull request adds comprehensive support for the SenseNova U1 model's understanding/text-generation branch, including HF-to-GGUF conversion and integration of dynamic-resolution image preprocessing features like 2D RoPE and patch embedding. It also implements temporal/spatial Q/K normalization with 4-axis RoPE and a CPU mtmd projector graph for vision encoding, explicitly excluding the image generation branch.
pull/28919
- Expert Mask Optimization in ggml: This pull request enables the
mul_mat_idfunction to interpret an index of-1as a signal to skip computation, implementing an "expert mask" mechanism similar to PyTorch's Mixture of Experts. The change is applied across multiple backends (except CUDA) to optimize model execution by conditionally bypassing matrix multiplication for certain experts.
pull/26631
- Vulkan Backend Performance and Precision Enhancements: These pull requests optimize the RMS Norm Vulkan shader by replacing workgroup reduction with a two-stage subgroup reduction for better GPU performance and add support for handling src1 precision in matrix multiplication by scaling values into the fp16 range. They also introduce helper functions, improve fp32 overflow management, and align Vulkan's behavior with the Metal backend.
pull/29882, pull/30656
- Hybrid DNA Tokenizer Improvements: This pull request enhances the hybrid DNA tokenizer by adding automatic tagging of DNA sequences in plain text, implementing N-padding for partial tail tokens to align with the vocabulary, and improving explicit tag handling to match Hugging Face tokenizer behavior.
pull/30825
- DeepSeek-V4.1 Model Optimizations and Fixes: These pull requests introduce the DeepSeek-V4.1-Flash model with various optimizations such as BF16 Engram table conversion and caching compressed-KV read layers, followed by fixes including tensor copying improvements, parameter refactoring, redundant request removal, comment cleanup, and backend operation updates to enhance performance and maintainability.
pull/31086, pull/31137
- Recursive Depth Limits and Stability Improvements: This pull request introduces mandatory recursive depth limits across all recursive functions with specific constants for different components to prevent excessive recursion and improve stability, alongside related grammar parsing and schema walking enhancements.
pull/31153
- Snapdragon Toolchain and Hexagon SDK Update: This pull request updates the project to use Snapdragon Toolchain v0.8 with Hexagon SDK v6.7.0.0, supporting Hexagon v85 in Snapdragon 8 Elite Gen 6 SoCs, enables building the
libggml-htp-v85backend, and enhances build/run scripts with automatic software image validation and Ubuntu target support via ADB transport.
pull/30854
- Qwen4exp Model Optimization: This pull request optimizes the
qwen4expmodel by building the QSA selection without allocating thekq_mask, resulting in significant GPU and host memory savings and improved performance during buffer computation and decoding.
pull/30965
- Chat Tabs UX Enhancements: This pull request implements multiple user experience improvements for chat tabs, making them behave like browser tabs with controlled creation and closing, ensuring conversations scroll to the bottom before first paint, enhancing visual transitions, preventing elastic overscroll bouncing, adjusting layout, and adding shared styles and unit tests.
pull/31030
- SYCL Backend Compiler Warning Fixes: This pull request addresses compiler warnings in the SYCL backend by marking file-local functions as static, including necessary headers, removing unreachable code and extra semicolons, and updating deprecated kernel invocation patterns to ensure clean compilation without changing runtime behavior.
pull/30271
- Hexagon Backend SGN Operation Support: This pull request adds support for the SGN (sign) operation to the Hexagon backend, initially for FP32 data types, with clean kernel checks and verification, motivated by the Qwen3.8-Flash-Next project, and includes optimizations for dimension collapsing and unary operations.
pull/30346
- Sampling Parameters Forwarding in Server Endpoint: This pull request enhances the
/v1/messagesendpoint to forward the full set of 24 sampling parameters defined inserver-schema.cpp, ensuring all llama.cpp sampling settings are passed through and aligned with other endpoints, fixing previous issues where some parameters were dropped.
pull/30856
- OpenCL Backend Topk and GEMM Improvements: These pull requests add initial support for the topk operation and lightning indexer in OpenCL, along with several new GEMM variants optimized for small batch sizes, fixes for out-of-bounds reads, removal of unnecessary CPU fallback conditions, and subgroup count capping to improve performance and correctness.
pull/31026, pull/30850
- Models Manager UI Enhancements: This pull request moves the Models Manager view to its own route and implements multiple UI improvements including better layout, navigation, clickable table rows, and a redesigned detail pane displayed as a drawer for improved usability across devices.
pull/30988
- Desktop Sidebar Drag-to-Resize Restoration: This pull request restores and enhances the desktop sidebar's drag-to-resize functionality by adding rubber-band resistance beyond width limits, enabling drag-to-hide below minimum width, implementing dynamic fade effects on labels and lists, and persisting sidebar width and open state across reloads.
pull/31119
- Embedding CLI GitHub Actions CI Workflow: This pull request adds a GitHub Actions CI workflow implementing fast, lightweight end-to-end tests for the embedding CLI, validating output dimensions and deterministic behavior using small GGUF models to ensure embedding correctness.
pull/16940
- Nemotron Model Tokenizer and Loader Fixes: This pull request adds support for the non-standard GQA, BPE tokenizer, and domyn-small pre-tokenizer type in the Nemotron model by fixing vocab decoding issues, correcting tensor shape mismatches in multi-head attention, improving converter handling of head dimensions, and adding fallback mechanisms for vocab setting.
pull/23343
- Soprano Audio Generation Integration: This pull request adds support for Soprano audio generation by integrating Soprano-80M and Soprano-1.1-80M models, including conversion scripts for Qwen3 backbone and Vocos decoder, a new pre-tokenizer, audio generation pipeline enhancements, and updated text-to-speech documentation.
pull/28607
- Long Prefix Handling Improvement: This pull request improves the algorithm handling long prefix entries in the server to reduce incidental loss of work by preventing the discarding of small chunks of context at the top of large contexts.
pull/30319
- UI Tool Call Visibility Enhancements: This pull request introduces a "Show tool call in progress" setting that automatically expands tool call blocks while arguments stream and execution occurs, then folds them back once results arrive, improving real-time visibility. It also includes enhancements to streaming command titles, file tool body rendering without paths, and comprehensive tests.
pull/30681
- OpenCL Backend q2_k Format Support: This pull request adds initial support for the q2_k format in the OpenCL backend, including multiple optimized kernel implementations and significant refactoring based on a prior pull request.
pull/30999
3.3 Pull Request Discussion Insights
This section will analyze the tone and sentiment of discussions within this project's open and closed pull requests that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.
Based on our analysis, there are no instances of toxic discussions in the project's open or closed pull requests from the past week.
IV. Contributors
4.1 Contributors
Active Contributors:
We consider an active contributor in this project to be any contributor who has made at least 1 commit, opened at least 1 issue, created at least 1 pull request, or made more than 2 comments in the last month.
If there are more than 10 active contributors, the list is truncated to the top 10 based on contribution metrics for better clarity.
| Contributor | Commits | Pull Requests | Issues | Comments |
|---|---|---|---|---|
| ggerganov | 140 | 29 | 0 | 38 |
| max-krasnyansky | 191 | 6 | 0 | 1 |
| ngxson | 89 | 10 | 0 | 16 |
| am17an | 59 | 17 | 0 | 26 |
| aldehir | 80 | 1 | 0 | 2 |
| ORippler | 79 | 0 | 0 | 4 |
| 0cc4m | 58 | 4 | 0 | 20 |
| ServeurpersoCom | 71 | 3 | 0 | 6 |
| CISC | 26 | 5 | 0 | 48 |
| lhez | 65 | 8 | 0 | 0 |