Weekly GitHub Report for Llama.cpp

Thank you for subscribing to our weekly newsletter! Each week, we deliver a comprehensive summary of your GitHub project's latest activity right to your inbox, including an overview of your project's issues, pull requests, contributors, and commit activity.

Table of Contents

• I. News
  • 1.1. Recent Version Releases
  • 1.2. Other Noteworthy Updates
• II. Issues
  • 2.1. Top 5 Active Issues
  • 2.2. Top 5 Stale Issues
  • 2.3. Open Issues
  • 2.4. Closed Issues
  • 2.5. Issue Discussion Insights

• III. Pull Requests
  • 3.1. Open Pull Requests
  • 3.2. Closed Pull Requests
  • 3.3. Pull Request Discussion Insights

• IV. Contributors
  • 4.1. Contributors

I. News

1.1 Recent Version Releases:

The current version of this repository is b4991

1.2 Version Information:

The version released on March 29, 2025, introduces key updates and improvements, focusing on enhanced performance and user experience. Notable highlights include optimized features and bug fixes that streamline functionality and increase stability.

II. Issues

2.1 Top 5 Active Issues:

We consider active issues to be issues that that have been commented on most frequently within the last week. Bot comments are omitted.

1. [ENHANCEMENT] [STALE] Feature Request: XDNA backend: This issue requests the addition of a ggml backend for AMD XDNA NPUs to enable llama.cpp to offload compute operations, starting with matrix multiplications, to the Neural Processing Unit found in AMD Ryzen AI processors. The proposed backend would leverage AMD's XRT C++ API and the IRON operator library to compile and dispatch NPU kernels, aiming to unlock NPU acceleration for all GGUF models and improve local LLM performance on Ryzen AI laptops.

   • The comments discuss the technical challenges and performance characteristics of XDNA NPUs, including transport bottlenecks dominating latency over compute, the importance of model geometry and tiling, and comparisons with existing solutions like FastFlowLM; contributors share experimental results, implementation plans, and insights on kernel fusion, INT4 vs INT8 quantization, and runtime architecture, while also offering to assist with testing and integration efforts.
   • Number of comments this week: 7

2. [BUG-UNCONFIRMED] Eval bug: SYCL run fails with The program was built for 1 devices (Iris(R) Xe Graphics): This issue reports a failure when running self-built llama binaries with the SYCL backend on Intel Iris Xe Graphics, resulting in an error stating "The program was built for 1 devices" and an exception related to an OP SCALE error. The user provides detailed environment information including hardware, software versions, and logs, and discusses attempts to resolve the problem by updating drivers and enabling the recommended xe kernel driver instead of i915.

   • The comments focus on troubleshooting the issue by verifying the operating system and driver versions, recommending enabling the xe driver for better compatibility, and suggesting the installation of the latest Intel GPU drivers from the official compute-runtime releases to potentially fix the problem.
   • Number of comments this week: 6

3. HIP/ROCm backend produces corrupted output on gfx1151: two dense architectures fail, Vulkan is correct with byte-identical flags: This issue reports that the HIP/ROCm backend on a gfx1151 (RDNA3.5) system produces corrupted output for two dense model architectures, failing to correctly emit tool calls in long-context scenarios, while the Vulkan backend with identical weights and flags produces correct results. The problem manifests as missing or incorrect tool calls, with failures starting at around 4,000 prompt tokens for one model and persisting at higher depths, and remains unresolved despite multiple mitigation attempts and extensive testing across configurations and models.

   • The comments detail several follow-up investigations including attempts at mitigation that failed, a depth sweep confirming one MoE model passes at all tested depths while dense models fail, a correction showing the failure onset is at a much lower token count than initially reported, and user reports of similar issues affecting ROCm and CUDA backends but not Vulkan, reinforcing the backend-specific nature of the defect.
   • Number of comments this week: 6

4. [STALE] Feature: Compact Conversation Action: This issue proposes adding a feature for compacting conversation actions in the UI, either on-demand via a button or command, or automatically when a setting is enabled and the context limit is nearly reached. The goal is to improve usability by reducing manual summarization and thread creation in conversations.

   • The comments show interest from contributors wanting to claim the issue, with the original poster planning to work on it soon, possibly with a collaborator. Discussion includes timing the work after certain refactor pull requests are merged, and enthusiasm for starting pair programming on the feature.
   • Number of comments this week: 5

5. Server: improve progress reporting: This issue focuses on improving the server's progress reporting by enabling the /models/sse endpoint to report both loading and downloading states, functioning correctly in both router and standalone modes. The discussion also covers the implementation details of inter-process communication, memory usage calculation in child processes, and plans to enhance progress visibility during model configuration and downloading.

   • The comments include feedback on progress reporting during model configuration, confirmation that download progress tracking is forthcoming, updates on the implementation of child process communication for memory calculation, and suggestions for improving memory usage estimation to better inform downstream applications.
   • Number of comments this week: 5

2.2 Top 5 Stale Issues:

We consider stale issues to be issues that has had no activity within the last 30 days. The team should work together to get these issues resolved and closed as soon as possible.

As of our latest update, there are no stale issues for the project this week.

2.3 Open Issues

This section lists, groups, and then summarizes issues that were created within the last week in the repository.

Issues Opened This Week: 108

Summarized Issues:

• ROCm and HIP Backend Issues: Multiple issues report crashes, build failures, and output corruption related to ROCm and HIP backends on AMD GPUs, including segmentation faults in libamd_comgr, missing typedefs causing build failures, and corrupted inference results on gfx1151 platforms. These problems affect model loading, execution correctness, and performance, with some workarounds involving downgrades or switching to Vulkan backend.
  • issues/27275, issues/27303, issues/27506, issues/27556, issues/27557, issues/27579

• Speculative Decoding and MTP Bugs: Several issues describe bugs and performance degradations related to multi-token prediction (MTP) and draft-mtp speculative decoding, including output corruption due to state leakage, GPU crashes on AMD RADV Vulkan backend, concurrency race conditions causing empty completions, and significant slowdowns when using multi-GPU setups. These affect both correctness and throughput of inference workflows.
  • issues/27296, issues/27306, issues/27373, issues/27428, issues/27572

• Vulkan Backend Stability and Performance: Multiple reports highlight crashes, hangs, and performance regressions on Vulkan backend, especially on AMD hardware, including device loss errors, tensor split mode crashes, deadlocks under load, and severe throughput degradation with large batch sizes or long contexts. Some issues are linked to kernel or driver interactions, while others involve internal resource management bugs.
  • issues/27334, issues/27360, issues/27458, issues/27460, issues/27505, issues/27544, issues/27604, issues/27638

• CUDA Backend and Multi-GPU Split Mode Issues: Several issues describe crashes, hangs, and degraded performance when using CUDA backend with tensor or row split modes on multi-GPU setups, including launch timeouts, partial offloading failures, and unsupported split buffer errors on certain GPUs. These problems prevent stable multi-GPU model loading and inference.
  • issues/27329, issues/27330, issues/27366, issues/27465, issues/27467, issues/27577

• Server Stability and Request Handling Bugs: Multiple issues report server crashes, hangs, or incorrect error reporting during inference or model loading, including failure to abort inference on Stop button, server hangs under load or streaming aborts, incorrect readiness reporting after fatal errors, and crashes on concurrent requests or large input sizes. These affect reliability and user experience.
  • issues/27309, issues/27326, issues/27387, issues/27388, issues/27427, issues/27456, issues/27619, issues/27627, issues/27604

• JSON Schema and Grammar Parsing Errors: Several issues describe bugs in JSON schema to grammar conversion and PEG chat parsing, including premature JSON generation termination, failure to parse regex escape sequences, and crashes due to malformed or unexpected grammar inputs. These cause HTTP 400 or 500 errors and prevent proper tool-calling or chat completions.
  • issues/27279, issues/27397, issues/27580, issues/27597

• Performance Regressions and Optimization Requests: Multiple reports highlight performance degradations in various backends and configurations, including MoE model slowdowns due to disabled optimizations, Vulkan prompt processing collapse at large context sizes, and requests for autotuning tools and improved shader parameters to boost throughput. These issues impact inference speed and resource efficiency.
  • issues/27327, issues/27420, issues/27425, issues/27553, issues/27649

• Model Loading and Memory Management Bugs: Several issues describe crashes or hangs caused by improper memory handling during model loading or inference, including division by zero errors on zero free memory devices, progressive VRAM overflow on SYCL backend, and out-of-memory errors causing silent failures or crashes. These affect stability and usability on constrained hardware.
  • issues/27440, issues/27454, issues/27595, issues/27527

• Concurrency and Deadlock Problems: Multiple issues report deadlocks or stalls in server or backend components under concurrent or sustained workloads, including KV cache deadlocks, slot management stalls, and blocking I/O causing indefinite hangs. These issues degrade throughput and responsiveness in multi-user or long-running scenarios.
  • issues/27361, issues/27505, issues/27587

• Feature Requests for Model and Server Enhancements: Several requests propose new features such as HTTP gzip/brotli compression support, stable public C API for multi-token prediction, just-in-time expert streaming for MoE models, router mode UI improvements, reasoning budget controls, and single-binary ASR CLI integration. These aim to improve usability, performance, and extensibility.
  • issues/27357, issues/27469, issues/27562, issues/27563, issues/27564, issues/27571, issues/27636

• Backend Compilation and Build Failures: Several issues describe compilation errors and warnings across different backends and platforms, including CUDA assembly errors on Nvidia H100, Metal shader undeclared identifiers on Apple Silicon, SYCL backend signature mismatches, and HIP backend loop unroll warnings. These prevent successful builds and deployment on affected systems.
  • issues/27415, issues/27459, issues/27473, issues/27479, issues/27536

• Tool-Calling and Chat Model Bugs: Multiple issues report bugs related to tool-calling in chat completions, including system message placement causing HTTP 500 errors, system prompt leaks between consumers, dropped tool calls during continuation prefill, and premature termination of multi-tool reasoning streams. These affect correctness and reliability of tool integrations.
  • issues/27367, issues/27422, issues/27588, issues/27613

• Metrics and Monitoring Failures: Several issues describe problems with metrics reporting, such as token per second gauges showing zero during active inference, and cached tensors not being utilized or deleted, leading to ineffective monitoring and resource leaks.
  • issues/27364, issues/27436, issues/27585

• Model-Specific Bugs and Limitations: Some issues describe model-specific problems, including empty completions for qwen35moe models beyond certain prompt lengths, embedding function failures for qwen35 embeddings with pooling, and severe decode throughput collapse in Qwen3.8-27B at very long contexts. These limit model usability in certain scenarios.
  • issues/27442, issues/27445, issues/27623

• Server Signal Handling and Process Control Issues: Issues report that the server container ignores SIGINT during model downloads and that DELETE requests to cancel streams do not properly abort child processes, causing resource occupation and unresponsive behavior.
  • issues/27581, issues/27481

• Research and Experimental Features: Some issues document ongoing research or proposals for advanced decoding methods like Parent Conditioned Drafting Trees and full-bandwidth transformers with latent feedback, noting no current implementations but exploring potential improvements in decoding efficiency and accuracy.
  • issues/27448, issues/27499

2.4 Closed Issues

This section lists, groups, and then summarizes issues that were closed within the last week in the repository. This section also links the associated pull requests if applicable.

Issues Closed This Week: 88

Summarized Issues:

• GPU Memory Reporting and Allocation Issues: Several issues report bugs related to incorrect GPU memory reporting or allocation, causing reduced performance, crashes, or out-of-memory errors. These include UMA detection logic limiting GPU memory on AMD APUs, overestimation of memory allocation with --fit-params, memory over-allocation on ROCm devices, and failures to load models due to VRAM or tensor count mismatches, all impacting model loading and runtime stability.
  • issues/18159, issues/25408, issues/25487, issues/27288, issues/27349

• Model Loading and Compatibility Failures: Multiple issues describe failures to load specific models or incompatibilities with certain hardware or quantization methods, resulting in crashes or errors. Problems include failures with MiMo V2.5 in tensor split mode, MobileLLM-R1-140M with q8_0 quantization, Qwen3.8-27B Ridge model crashes, and issues loading models on Windows with SYCL or Vulkan backends.
  • issues/23361, issues/25269, issues/27299, issues/27253

• Backend and Compilation Issues: Several reports cover compilation failures and backend-specific problems, including SYCL backend compilation errors resolved by using Intel's compiler, Vulkan backend crashes on AMD Polaris hardware, and OpenCL backend producing incorrect results on Adreno 740 GPUs. These issues highlight challenges in supporting diverse hardware and compiler environments.
  • issues/20702, issues/27403, issues/25477

• Crash and Runtime Errors in Model Evaluation and Decoding: Various issues describe crashes during model evaluation or token generation caused by assertion failures, illegal instructions, or memory access errors. Examples include crashes with Step 3.7 flash model on Vulkan, segmentation faults with Gemma 4 12B using flash attention, and CUDA illegal memory access errors during decoding or tensor initialization.
  • issues/24257, issues/27314, issues/27429, [issues/27430](https://github.com/issues/27430]

• Speculative Decoding and Parallelism Bugs: Issues report bugs related to speculative decoding and parallel processing, such as failures when using MTP with tensor parallelism, garbled output with deepseek4 model under parallel speculative decoding, and CUDA errors during DSpark speculative decoding on multi-GPU setups. These problems cause crashes or corrupted outputs in concurrent or advanced decoding modes.
  • issues/24366, issues/26741, [issues/26554](https://github.com/issues/26554]

• Security Vulnerabilities in ggml-rpc Server: Critical security issues include remote unauthenticated denial-of-service vulnerabilities caused by unbounded vector resizing and block size alignment violations, as well as a control-flow hijack risk due to deserialization of function pointers without validation. These vulnerabilities can cause server crashes or potential exploitation.
  • issues/25287, issues/25288, [issues/25289](https://github.com/issues/25289]

• Parsing and Grammar Processing Errors: Multiple issues describe parsing failures and crashes in Jinja template parsing, JSON-Schema to grammar conversion, and PEG parser errors due to malformed inputs or missing lenient fallbacks. These cause failures in template rendering, grammar generation, or chat completion responses.
  • issues/20305, issues/25282, issues/25283, issues/25284, [issues/25321](https://github.com/issues/25321]

• Performance Regressions and Optimization Opportunities: Reports include a 7-9% performance regression on CUDA builds, decode throughput degradation due to build environment mismatches, and suggestions for improving Vulkan prefill throughput and CUDA GEMV dispatching. These highlight areas where performance has declined or can be improved.
  • issues/25422, issues/27444, issues/25491, [issues/27464](https://github.com/issues/27464]

• Audio and Media Input Handling Bugs and Feature Requests: Issues cover bugs in audio embedding and encoding causing high word error rates, silent failures of video input due to MP4 MOOV atom placement, and feature requests for microphone button support and media tokenization tools to improve audio and image input handling.
  • issues/25496, issues/24394, issues/24375, [issues/27391](https://github.com/issues/27391]

• Server and Networking Issues: Problems include llama-server crashes when unknown tools are specified, networking request length limits causing 400 errors, proxy failures reading connections, and issues with Docker volume mounts failing due to invalid characters in task IDs. These affect server stability and network communication.
  • issues/27384, issues/25290, issues/25396, [issues/27570](https://github.com/issues/27570]

• Model Output and Template Rendering Bugs: Several issues describe corrupted or gibberish output caused by backend options, template rendering bugs causing degenerate token repetition, and silent dropping of tool calls or images in chat messages, leading to degraded user experience or incorrect results.
  • issues/23321, issues/27606, issues/27363, [issues/27313](https://github.com/issues/27313]

• Build and Environment Configuration Problems: Issues include Makefile sensitivity to GCC versions causing build failures, read-only files causing permission errors during build cleanup, and ignored command-line options for CPU affinity or Docker repository preloading, leading to unexpected behavior or build difficulties.
  • issues/25253, issues/25374, issues/26997, [issues/27159](https://github.com/issues/27159]

• Quantization and Model Compression Feature Requests: Requests for adding support for TurboQuant and GLM 5.2 sparse attention acceleration aim to reduce memory usage and improve performance, enabling larger models to run on smaller hardware and accelerating specific attention layers.
  • issues/20977, [issues/24730](https://github.com/issues/24730]

• Miscellaneous Bugs and Feature Requests: Other issues include bugs with non-ASCII filename encoding on Windows, clipboard image pasting under Wayland, silent failures when adjacent images have identical dimensions, and proposals for native WebP support and diffusion model integration.
  • issues/25200, issues/25381, issues/27313, issues/27443, [issues/25427](https://github.com/issues/25427]

2.5 Issue Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed issues that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed issues from the past week.

III. Pull Requests

3.1 Open Pull Requests

This section provides a summary of pull requests that were opened in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Opened This Week: 108

Key Open Pull Requests

1. ui: Added agent skills support: This pull request implements comprehensive support for agent skills in the UI and server, including skill discovery, catalog management with token budget enforcement, resource handling with secure path verification, skill activation via slash commands and model-driven tools, persistent activation storage, project-level skill trust controls, and a read-only skill reference system, all designed to enable modular, secure, and user-controllable integration of skills following the Agent Skills specification.

• URL: pull/27312

• Associated Commits: 30fc7, 14d7b, b8d8a, 35ee1, 29ada, 40b99, 762e3, 8237a, 4aac7, a79d7, 02e99, 3519d, ada30, 5cfcc, 02984, 8a78e, 38787, b3999, e718d, d7ee5, 7fb86, 1ab31, 77d6b, be7f7, 9e6e7, 3542e, ce249, 010c4, 1ccdf, 9223f, a1412, 90432, 96fa7, 101d0, 2df60, 7dc80, 9b13f, 4112a, 50c81, 7b1c3, 1b892, 6f321, 57530, 8246f, d9b61, bbc96, 139fa, 7c92a, 5ae74, 6b47c, bc58a, defc4, 251e5, 0cc87, 8eb42, 71cf0, 2150c, 3014d, 20c8c

2. quantize: add IQ2_NL and IQ3_NL types (CPU + Metal + CUDA): This pull request introduces two new 32-element block quantization types, IQ2_NL and IQ3_NL, for CPU, Metal, and CUDA backends to improve quantization efficiency for tensors whose row lengths are multiples of 32 but not 256, thereby extending the existing non-linear quantization family downward and addressing performance limitations in handling misaligned tensors.

• URL: pull/27325

• Associated Commits: f6602, a6c99, 427d2, da119, f4afc, c3ad8, 29191, 6ea87, 3d9f4, 92f87, 937be, dceb9, 3362a, 02b8e, ae848, 789e3, 37eea, 596ec, 62f7a, 700df, e746f, 1acbf, 58bcb, 79f93, 7c348, 2168e, d07a9, 18254, 6c8eb, ac9a9, bb8e0, 2be13, d79ad, ff3d9, 30cf4, 67897, 2ccab, 2cd66, d66e2, 37d9c, f7fad, 7d371, c8db7, 83adc, b75b6, b0420, f8f4d, 98558, f4285, e6cf9, d6b47

3. SVE 128 bit Implementation of gemm_q4_k_8x8_q8_k kernel: This pull request implements 128-bit SVE (Scalable Vector Extensions) support for the ggml_gemm_q4_K_8x8_q8_K kernel using i8mm and vector instructions, resulting in a verified accuracy matching the NEON implementation and improving LLM inference prompt evaluation speed by approximately 4 to 5% on Nvidia Grace hardware.

• URL: pull/27491

• Associated Commits: 4efdf, 03ed7, edc8e, 71fed, e8495, c124b, a6119, 88ca4, e0864, 70cfc, 62617, 2a2e1, 618b6, 93951, 60bd0, 9b33d, 0eebf, ce43d, e117e

Other Open Pull Requests

• Scheduler and Backend Stability Improvements: This topic covers enhancements to the scheduler graph input handling using a ring buffer to safely manage asynchronous device reads, improvements to the scheduler sanitizer with non-fatal reporting and direct-write detection, and fixes for memory aliasing and reuse in multi-backend scenarios. It also includes detailed documentation for the backend scheduler and ring buffer, along with various hardening and correctness improvements to prevent race conditions and ensure stable graph input rotation and backend operation placement.
  • [pull/27311]

• Quantized Model Performance Optimizations: Multiple pull requests introduce AVX2-optimized kernels for IQ-quantized models and STQ1_0 quantization formats, as well as SVE 128 and 256 vector extensions for q4_k gemv kernels, significantly improving performance on x86 and ARM architectures. These changes include block decoding of weights for large batch prompt processing and porting ARM NEON implementations to AVX2, resulting in up to 10x speedups and notable gains on dense and MoE models.
  • [pull/27402, pull/27377, pull/27470, pull/27472]

• OpenCL Backend Fixes and Enhancements: This group of pull requests addresses multiple bugs and stability issues in the OpenCL backend, particularly for Adreno GPUs, by improving device version checks, skipping zero global work size kernel dispatches, enforcing operand contiguity, and adding memory allocation retries with fallbacks. Additional enhancements include adding new elementwise unary operations, broadening CONCAT support, and optimizing matrix multiplication kernels for small batch verification to improve performance and correctness.
  • [pull/27630, pull/27632, pull/27633, pull/27635]

• Model and Architecture Support Additions: These pull requests add support for new models and architectures, including draft-mtp for dots3note, comprehensive TeleChat4 model integration, and LIGHTNING_INDEXER operation support on Vulkan backend for DeepSeek V4. They also include refactoring and testing across ROCm and Vulkan platforms, enabling broader model compatibility and backend functionality.
  • [pull/27618, pull/27435, pull/27453]

• Server Features and Observability Enhancements: This topic includes the introduction of a new --sleep-mode rst server option that resets the process to address residual memory issues while preserving cached responses and metrics, implemented via a POSIX-only exec restart mechanism. It also covers the addition of optional OpenTelemetry tracing as a compile-time feature to enhance observability and align with standard tracing SDKs.
  • [pull/27418, pull/27280]

• Bug Fixes in Parsing, Conversion, and Shader Handling: Pull requests in this category fix a grammar parsing error related to escaped hyphens in character classes, resolve tensor naming mismatches in LoRA adapter conversion to GGUF format, and correct handling of infinity values in WebGPU ARGSORT and TOP_K shaders to prevent improper sorting. These fixes improve robustness and correctness in parsing, model export, and shader operations.
  • [pull/27591, pull/27509, pull/27538]

• Memory and State Management Improvements: This set of changes ensures proper failure containment by clearing or discarding key/value cache and recurrent memory tensor data after failed state restores, fixing rollback asymmetries and improving robustness without full transactional rollback. It also includes fixes for out-of-contract memory reads in OpenCL Adreno image kernels by clamping out-of-bounds accesses and enforcing strict tiling and layout requirements.
  • [pull/27530, pull/27632]

• Test Coverage and Code Simplification: This topic covers the addition of comprehensive test coverage for various matrix multiplication and flash-attention shapes emitted by real models, ensuring backend kernels are properly exercised without code changes. It also includes removal of redundant in-kernel dequantization code from Metal backend flash attention kernels, simplifying the codebase and reducing library size.
  • [pull/27641, pull/27398]

• Embedding and Token Processing Optimizations: These pull requests introduce zero-copy embedding optimizations and event-based synchronization for draft models, significantly improving token processing speed on specified hardware. They also add reordered Q2_K MMVQ and ESIMD kernels to improve multi-token call performance in the SYCL backend, achieving up to 1.9x throughput improvements without affecting perplexity.
  • [pull/27396, pull/27509]

• CUDA Backend Stability Fixes: This pull request fixes a critical bug in the CUDA graph cache eviction mechanism that caused crashes under KV-cache saturation by detecting stale graph entries, synchronizing CUDA streams, and safely destroying stale graph objects. This ensures safe CUDA graph lifetime management and prevents hard crashes related to cublasSgemm INVALID_VALUE errors.
  • [pull/27368]

3.2 Closed Pull Requests

This section provides a summary of pull requests that were closed in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.

Pull Requests Closed This Week: 190

Key Closed Pull Requests

1. model: GraniteSWAForCausalLM / GraniteMoeSWAForCausalLM: This pull request adds support for forthcoming Granite models in the llama.cpp project by implementing interleaved Sliding Window Attention with Attention Sinks, including conversion infrastructure, per-layer RoPE/NoPE handling, and integration of GraniteSWAForCausalLM and GraniteMoeSWAForCausalLM architectures, while mirroring related work from the Hugging Face transformers repository and ensuring compatibility with internal checkpoints.

• URL: pull/25505

• Associated Commits: 3a193, 7a799, a7674, 803d5, ece84, 83f84, edd29, 68f93, 7be41, 4bbf9, c7404, 1f374, f7d05, 44215, cf315, f5d4e, 9d75e, ef14e, c32fe, 36aa2, 94d26, dc7f1, 90c3d, 0b1df, 8865f, 8e5bc, 97743, e400d, e7695, 03067, 93f6c, 60061, 94fc6, 283c4, c94d5, b5b7d, f7b13, eeb83, d7a74, de5f6

2. ui: Chat Conversation Tabbed navigation: This pull request introduces a browser-style tabbed navigation system above the chat screen to manage multiple open conversations with features including persistent tabs synced with the URL route, improved horizontal scrollable carousel UI, keyboard shortcuts for cycling and closing tabs, automatic tab closure when conversations are deleted, a user setting to enable or disable tabs, and synchronization of the sidebar state with the tab bar for enhanced user experience.

• URL: pull/27263

• Associated Commits: 19eb0, 4389e, f46de, 36a12, ed33f, e0312, 190f1, c7ef9, 8b9cd, 975c2, 33846, 3dcff, 096db, d900e, 65688, fed47, 495fe, 66225, 26da5, 7b674, 1b183, 0ed2f, b4c67, e5af8, 4a0a2, 707e7, 890b4, 89776, 01d0f, 54b74, e4c77, 128dd

3. Optimise memory usage by evicting weights after processing each layer: This pull request introduces a mechanism to optimize memory usage by evicting tensor weights from memory immediately after processing each layer, thereby releasing memory pages that are no longer needed.

• URL: pull/22877

• Associated Commits: 0a068, 1f1db, 11ad4, f7e9d, d982a, 7251c, 0335a, 7d9c2, 7f355, 6d6a8, 9125f, 85025, 20126, 5ea06, 8221e, 9f3f1, 6af52, e250d, 99c89, ffd38

Other Closed Pull Requests

• JSON abstraction and compile-time optimization: This pull request introduces a lightweight common/json.h abstraction encapsulating nlohmann::json via a pimpl pattern to reduce compile times and binary sizes. It minimally changes downstream code by renaming nlohmann::json to common_json and adjusting JSON usage for clarity and efficiency.
  • pull/27511

• Compiler warnings and build environment fixes: Multiple fixes address compiler warnings in the SYCL backend related to icx/icpx on Linux and Windows-specific warnings, maintaining a clean build environment. Additionally, the Windows CI setup replaces a non-redistributable debug libomp with the official LLVM OpenMP release to improve compatibility and licensing compliance.
  • pull/26713, pull/26678

• Server and service refactoring: Refactors include improving the server's sleep functionality with cached responses and race condition fixes, though the PR was not merged due to branch issues. Other refactors relocate stream fetches, event handling, and conversation import/export into dedicated service modules to improve code organization and API clarity.
  • pull/27417, pull/27239

• Device and backend optimizations: Introduces a --mmproj-device argument for device selection in mmproj processing, enabling iGPU offloading for large models while maintaining backward compatibility. Backend optimizations include improved MXFP4 W4A4 activation quantization on CUDA, Metal backend preprocessing for dequantizing KV cache formats into float16 buffers, and Vulkan backend improvements by reducing redundant dequantization operations.
  • pull/23255, pull/27315, pull/27390, pull/25494

• Continuous integration enhancements: CI improvements include adding and fixing Android Vulkan workflows with a shift to manual dispatch, applying an enhanced ccache-clear action with new options to all ccache-using jobs, and refining cache key management and compatibility across platforms.
  • pull/27518, pull/27602

• Performance optimizations in tensor operations: Optimizes ggml_compute_forward_concat_any by replacing many per-element memcpy calls with fewer row-level calls, significantly improving tensor concatenation performance. Also, introduces the ggml_rope_set_offset function supporting CPU and Metal backends for advanced rotary positional embeddings, coordinating with CUDA support.
  • pull/24575, pull/27120

• Release automation and build system improvements: Extends the make-release workflow to create GitHub pre-releases with changelogs and nightly build links, bumps project version, and improves release description scripts. Fixes build process for xcframework by adding vendored hash library dependencies and introducing vendor:: alias targets for modularity.
  • pull/27302, pull/27304

• OneDNN and GPU feature gating: Updates gating logic for OneDNN features on Alchemist GPUs by restricting SPDA routes to head size 64 shapes, improving prompt processing performance while maintaining correctness and providing validation scripts.
  • pull/26635

• Bug fixes and test enhancements: Fixes a DeepseekV4 rollback bug by replicating s_copy semantics from llama-hybrid memory, adds a new test case, and integrates it into the test suite. Also fixes non-deterministic results in the FLASH_ATTN_EXT pipeline by correcting queue ordering and optimizing diagonal scale matrix handling for performance and reduced memory usage.
  • pull/26756, pull/27042

• Settings and UI improvements: Reorganizes settings navigation by converting the registry into an ordered array for sidebar control, refactors localStorage persistence into a service, adjusts routing, centralizes syncable parameter derivation, and enhances API key masking for better UI behavior.
  • pull/27241

• RVV tensor hashing and ESIMD kernel additions: Adds RVV tensor hashing using xxh3 to the gguf-hash tool with related fixes. Also introduces a new ESIMD kernel for Q5_K data type in DMMV QK computation, completing prior ESIMD work and demonstrating performance improvements without affecting model perplexity.
  • pull/18576, pull/26376

• MoE and GPU kernel optimizations: Optimizes the gpt-oss MoE feed-forward network by fusing bias additions into OpenCL kernels, reducing memory operations and improving performance on Adreno GPUs. Ports the fused per-token SSM_SCAN kernel for Mamba-2 from CPU fallback to GPU using OpenCL, enabling float32 operations on supported hardware with opt-out options.
  • pull/26431, pull/26439

• OpenCL kernel fixes for Adreno GPUs: Fixes multiple code generation defects in the q6_K flat mul_mat OpenCL kernel on older Adreno E031 compilers by implementing compiler-version-gated workarounds. These fixes restore correctness and improve performance by avoiding unnecessary workarounds on unaffected compiler versions.
  • pull/26476

• KV cache data structure improvements: Proposes replacing std::bitset with a uint64_t array and std::map with a custom seq_pos_t structure in llama-kv-cache kv-cells to improve asymptotic lookup and modification performance.
  • pull/27061

3.3 Pull Request Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open and closed pull requests that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open or closed pull requests from the past week.

IV. Contributors

4.1 Contributors

Active Contributors:

We consider an active contributor in this project to be any contributor who has made at least 1 commit, opened at least 1 issue, created at least 1 pull request, or made more than 2 comments in the last month.

If there are more than 10 active contributors, the list is truncated to the top 10 based on contribution metrics for better clarity.