Weekly GitHub Report for Llama.cpp: July 27, 2026 - August 03, 2026 (21:17:54)
Weekly GitHub Report for Llama.cpp
Thank you for subscribing to our weekly newsletter! Each week, we deliver a comprehensive summary of your GitHub project's latest activity right to your inbox, including an overview of your project's issues, pull requests, contributors, and commit activity.
Table of Contents
I. News
1.1 Recent Version Releases:
The current version of this repository is b4991
1.2 Version Information:
The version released on March 29, 2025, introduces key updates that enhance overall performance and stability, reflecting a continued focus on optimizing user experience and system reliability. Notable highlights include improved processing speed and bug fixes addressing previous issues.
II. Issues
2.1 Top 5 Active Issues:
We consider active issues to be issues that that have been commented on most frequently within the last week. Bot comments are omitted.
-
[BUG-UNCONFIRMED] Eval bug: Gemma 4 12B produces garbled output on large prompts, Intel Arc Pro B70 (Xe2): This issue reports that the Gemma 4 12B model running on an Intel Arc Pro B70 GPU produces garbled and incoherent output when given large prompts (~5000+ tokens), with the corruption persisting across subsequent requests on the same model instance until a full reload is performed. The problem appears to be related to persistent state corruption in GPU compute buffers or per-slot KV caches rather than prompt length or cache reuse alone, and it affects dense models like 12B and 31B but not the 26B MoE model, with attempts to disable certain optimizations or downgrade drivers failing to resolve the issue.
- Commenters confirmed the issue occurs on single GPU setups and shared detailed reproduction steps and configurations; attempts to fix by toggling environment variables or driver versions were unsuccessful, while testing showed the 26B MoE model does not exhibit the problem, and one user was unable to reproduce the bug on a similar setup, leaving the issue unconfirmed.
- Number of comments this week: 11
-
[BUG-UNCONFIRMED] Eval bug: GGML_OP_TOP_K falls back to CPU on HIP/ROCm above ~3–4K context — 6.4× token-generation loss on DeepSeek-V4-Flash: This issue describes a performance regression in the HIP backend of GGML where the GGML_OP_TOP_K operation falls back to CPU execution when the context size exceeds approximately 3–4K tokens, causing a severe 6.4× drop in token-generation speed on the DeepSeek-V4-Flash model despite full GPU residency. The root cause is identified as a backend-wide limitation where HIP lacks a CUB-based top-k implementation, triggering a size gate that forces CPU fallback for large inputs, and the suggested fix involves integrating hipCUB/rocPRIM or extending the existing bitonic sort to handle larger sizes on HIP to eliminate the CPU fallback.
- Commenters confirmed the issue across multiple AMD GPUs and models, shared profiling insights, and discussed partial fixes including a referenced PR that mitigated CPU utilization and restored performance; ongoing work aims to implement a proper HIP equivalent of the CUDA CUB top-k to fully resolve the problem without regressions.
- Number of comments this week: 11
-
[BUG-UNCONFIRMED] Eval bug: [SYCL] empty/gibberish output on hybrid models + ggml_sycl_op_mul_mat crash (qwen3next/qwen35 arch) on Intel Arc Pro B60 — regression between pinpointed b9128- b9159 (2026-03-23) and server-intel latest (2026-06-03): This issue reports a regression in the SYCL backend of llama.cpp affecting hybrid Qwen3.5+ models on Intel Arc Pro B60 GPUs, where newer builds between commits b9128 and b9159 cause crashes during matrix multiplication or produce empty/gibberish output, while older builds work correctly. The problem appears linked to recent changes in Level Zero memory allocations and Intel compute-runtime versions, with users narrowing down the regression window and discussing workarounds such as disabling VMM or pinning to a known good build.
- The comments detail extensive user testing across multiple builds confirming the regression onset between b9128 and b9159, suggest potential causes including Level Zero allocation changes, provide debugging logs and reproduction steps, and share partial workarounds; some users report inability to reproduce on single GPU setups or different hardware, while others confirm crashes that can be mitigated by disabling VMM, and a merged PR is proposed as a possible fix.
- Number of comments this week: 8
-
[BUG-UNCONFIRMED] Eval bug: vk::DeviceLostError within a few turns on DeepSeekv4-Flash (RADV_STRIXHALO): This issue reports a crash occurring within a few turns when running the DeepSeekv4-Flash model on a Strix Halo GPU using the RADV Vulkan driver, resulting in a vk::DeviceLostError. The problem appears specific to this model and hardware combination, with logs and kernel messages indicating GPU compute queue timeouts and device resets, and users have discussed debugging steps, driver timeouts, and partial workarounds such as increasing driver timeout settings.
- The comments include requests for exact command lines and debug logs, sharing of kernel and driver error outputs, attempts to reproduce the issue on different hardware, suggestions to enable Vulkan debug options, and observations that the problem may be related to driver timeouts on AMD RADV but not Nvidia, with some users providing detailed logs and noting partial workarounds like increasing timeout values.
- Number of comments this week: 7
-
[BUG-UNCONFIRMED] Eval bug: same K and V cache type enforced for models with no V cache: This issue reports a problem when loading the GLM-5.2 model with a specific cache type setting where the code enforces that the key (K) and value (V) cache types must be the same, even for models that do not support a separate V cache, resulting in an error. The discussion clarifies that this enforcement is intentional to maintain existing logic, but it causes regressions for users who previously used different quantization settings for K and V caches to optimize VRAM usage and model accuracy, and suggests possible workarounds or improvements such as setting the V cache type to match K automatically or providing an override flag.
- The comments explain that the enforcement is by design due to the unified latent representation for K and V caches in certain models, meaning only the K cache type is actually used; users express concerns about VRAM and performance regressions, and developers discuss the technical reasons behind the enforcement, potential fixes, and recent changes that increased memory usage, including a proposed PR to reduce indexer cache size that may alleviate some issues.
- Number of comments this week: 7
2.2 Top 5 Stale Issues:
We consider stale issues to be issues that has had no activity within the last 30 days. The team should work together to get these issues resolved and closed as soon as possible.
As of our latest update, there are no stale issues for the project this week.
2.3 Open Issues
This section lists, groups, and then summarizes issues that were created within the last week in the repository.
Issues Opened This Week: 94
Summarized Issues:
- GPU Vulkan and Multi-GPU Output Corruption: Several issues report corrupted or garbled output when using Vulkan GPU backend or multi-GPU setups, including problems on Adreno X1-85, AMD GPUs, Samsung S26 Ultra, and multi-GPU CUDA configurations. These bugs often relate to shader paths, data passing between GPUs, or backend-specific bugs causing incorrect dequantization, memory access, or synchronization failures.
- CUDA and GPU Crashes or Initialization Failures: Multiple issues describe crashes or illegal memory access errors on NVIDIA GPUs during model loading or inference, including RTX 4070 and RTX 5090, often linked to kernel initialization or shared object errors. These crashes sometimes require downgrading runtime versions or are accompanied by silent aborts without error messages.
- Performance Regressions on Specific Hardware and Backends: Several reports highlight significant performance drops on various GPUs and backends, including Windows mingw-w64 with OpenMP, RDNA4 GPUs after kernel removal, Intel Arc Pro B70 with oneMKL GEMM Flash Attention, HIP/ROCm backends, and Apple Silicon Metal backend. These regressions affect throughput, token generation speed, or prompt processing and sometimes have suggested mitigations or workarounds.
- Model Loading and Quantization Bugs: Issues include failures to load models due to tensor offset mismatches, unsupported quantization formats, or backend-specific errors such as assertion failures on AMD MI50 or errors with Dspark and GLM-5.2 models. Some bugs cause infinite loops or crashes during model initialization or inference.
- KV Cache and Prompt Caching Issues: Problems include silent contamination of outputs due to reuse of cached prompt key-value pairs with different LoRA adapters, corrupted or degraded embedding vectors on repeated batch requests, memory leaks in rerank-only servers due to improper prompt cache usage, and bugs in KV cache compression causing inference on integrated GPUs instead of dedicated GPUs.
- Server and API Stability and Crash Issues: Several issues describe server crashes or aborts due to assertion failures, missing tensors, or invalid input handling, including crashes when enabling speculative MTP decoding with embedding mode, multi-node deployment crashes on empty assistant messages, and llama-server.exe failing to spawn on non-ASCII paths.
- Vulkan Backend Bugs and Compatibility Problems: Bugs include pipeline creation failures on Raspberry Pi 5 due to hardcoded workgroup sizes, Vulkan TOP_K unit test crashes on Intel UHD 770, and Vulkan SDK version being outdated in Linux builds preventing full extension support. These issues cause corrupted output or build failures.
- Memory Allocation and Offloading Problems: Issues report VRAM not being allocated on GPUs with ROCm backend causing CPU fallback, incorrect double addition of memory margin causing overestimation of free memory and offloading layers to CPU, and silent spilling of large buffers into system memory during large context speculative decoding causing throughput collapse.
- Model and Tooling Feature Requests: Requests include adding support for new quantization formats (Escha-W2 2-bit), Apertus family models, OpenEval dataset import/export, MOE optimizations for SSDs and host RAM PCIe DMA, inline image rendering in WebUI, cumulative speculative decoding metrics, and native HarmonyOS NPU support.
- Security Vulnerabilities and Reporting: A security vulnerability is reported where the /slots?action=save API improperly handles symlinks allowing arbitrary file overwrite, and a request is made for a private security reporting channel to coordinate confidential disclosure.
- Command Line and Input Handling Bugs: Bugs include failure to escape special tokens causing generation errors, incorrect tokenization ignoring "parse_special": false parameter, Windows MSVC build returning invalid UTF-8 bytes in JSON for CJK output, and argument parsing issues replacing argv unexpectedly on Windows.
- Model Output and Reasoning Behavior Issues: Reports include models producing infinite or repeated tokens, reasoning flags not triggering expected behavior, disappearance of reasoning UI elements, and progressive degradation of output quality due to inter-request state retention in MTP-enabled models.
- Build and Compilation Failures: Compilation errors occur on Ubuntu Jammy due to type mismatches in JSON vector push_back, RHEL8 build failures due to missing posix_spawn_file_actions_addchdir_np in older glibc, and macOS 15 arm64 binaries crashing due to missing symbols from newer SDK targeting.
- Regression and Bug Tracking for Deepseek and Related Features: Tracking progress on Deepseek4 features including tensor split mode, sparse attention, chat template fixes, backend support, and FP8 base model conversion, alongside bugs causing crashes or validation errors during speculative decoding and graph splitting.
- Memory and Dimension Handling Bugs: Bugs include division-by-zero errors in gguf C++ reader due to zero tensor dimensions, off-by-one errors causing out-of-bounds access in dimension checks, and cache type enforcement errors restricting K and V cache configurations.
- WebUI and Client Display Issues: The WebUI MCP client fails to display tool call structuredContent when content arrays are empty, and MCP servers configured in the UI no longer appear on the configuration page, forcing manual addition or selection from public lists.
- Miscellaneous Bugs and Issues: Includes Windows Defender false positive Trojan detection on extracted binaries, doubled audio encoding latency in Voxtral model, and decode bandwidth limitations on Raspberry Pi 5 ARM CPU backend.
2.4 Closed Issues
This section lists, groups, and then summarizes issues that were closed within the last week in the repository. This section also links the associated pull requests if applicable.
Issues Closed This Week: 106
Summarized Issues:
- Model Loading and Format Compatibility Issues: Several issues report failures or errors during model loading due to unsupported architectures, missing tensors, or format conversion problems. These include errors with Laguna XS.2 architecture, missing tensors in Nex N2 Pro, invalid vector subscripts in Gemma 4 draft models, and GGUF conversion mismatches causing garbled output.
- issues/22337, issues/23249, issues/23758, issues/24211
- Performance Regressions and Slowdowns: Multiple reports describe significant performance drops in token generation, prefill, or prompt processing speeds across various backends and models. Causes include changes in CUDA kernel qualifiers, layer allocation between CPU and GPU, and overhead in SYCL MTP on Intel Arc GPUs.
- issues/24315, issues/24514, issues/23533, issues/24596, issues/25162, issues/26222, issues/24628
- Crashes and Stability Problems: Several issues describe crashes occurring under specific conditions such as voice input processing, CUDA out-of-memory errors, Vulkan device loss, RPC backend failures, and memory leaks. These crashes affect various models including Gemma 4, Qwen3.6, DeepSeek V4, and MiniMax-M3, often linked to backend-specific bugs or resource management problems.
- issues/23075, issues/23154, issues/24432, issues/24537, issues/24824, issues/25902, issues/26151, issues/26155, issues/26263, issues/26260, issues/26424
- Web UI and User Interface Enhancements and Bugs: Requests and bug reports focus on improving usability and accessibility of the WebUI, including adjustable chat width, artifact side panels, audio input support, conversation organization, runtime parameter editing, and mobile device compatibility. Bugs include infinite loading spinners, duplicated reasoning blocks, and inaccessible model management on mobile.
- issues/18067, issues/22868, issues/22982, issues/23609, issues/23622, issues/24090, issues/24464, issues/24465, issues/24619, issues/24164, issues/24221
- Speculative Decoding and Multi-Token Prediction (MTP) Issues: Several issues describe bugs and performance problems related to MTP and speculative decoding, including infinite loops of unused tokens, assertion failures, slowdowns, and crashes during draft model usage or heavy agentic workloads. Proposed improvements include confidence-scheduled verification and semi-autoregressive draft models.
- issues/21516, issues/23803, issues/24576, issues/24596, issues/25096, issues/23533
- Backend and Hardware Support Requests and Bugs: Requests and issues cover adding support for new backends like SYCL for Q1 quantization, systemd socket activation, Vulkan SDK updates, and ARM kernel selection. Bugs include memory leaks on macOS Metal, CUDA build warnings, and backend identity representation problems.
- issues/21641, issues/23213, issues/26299, issues/25937, issues/26272, issues/26334, issues/24815
- Model Cache and Management Features: Feature requests include tools for listing and deleting cached models, adding inference presets, system message injection, and exposing real-time job statistics. Issues also report problems with model management on mobile and router mode ID mismatches.
- issues/16393, issues/22523, issues/24246, issues/25519, issues/24690, issues/24619
- Memory Management and Resource Optimization: Issues describe memory leaks, out-of-memory crashes, and requests for per-device memory budget flags and thread count optimizations. Problems include memory pressure after model wake, CUDA memory leaks, and inefficient embeddings mode execution.
- issues/24350, issues/24372, issues/24626, issues/24713, issues/24424, issues/26290
- Image and Audio Processing Bugs: Bugs include server crashes when processing large images with large models, incorrect image resizing algorithms, and lack of OGG audio file support in the WebUI.
- issues/21750, issues/24629, issues/24164
- Tool and Server Security and Permissions: Requests and discussions focus on adding configurable approval settings for MCP tool calls, refactoring tool registries for unified management, and implementing dedicated permissions for shell command tools to enhance security and control.
- issues/20343, issues/20673, issues/24348
- Model Conversion and Format Creation Requests: Requests include creating GGUF formats for specific models like openai/privacy-filter and adding support for new model families such as Macaron-V1 and Nanbeige.
- issues/23783, issues/26066, issues/26086
- Bug Fixes in Code and Build Processes: Issues report fixes for memory leaks in tensor allocation, compilation errors due to missing headers, and build warnings related to potential buffer overflows.
- issues/24557, issues/26253, issues/26272
- Privacy and Security Concerns: One issue highlights a privacy risk where the tab title leaks the first prompt across browsers without encryption, suggesting truncation to mitigate exposure.
- issues/23854
- Agent and OS-Level Context Management Proposals: Proposals include AgentOS for efficient local agent inference with hierarchical registries and cloud memory, and AI autonomic agent code creation with free functionality.
- issues/24741, issues/26203
- Miscellaneous UI and UX Bugs: Issues include the system prompt fade-out effect being cut off prematurely and the WebUI not respecting plain text copy settings for attachments.
- issues/24453, issues/25785
- Model Download and Network Issues: One issue reports model download hangs due to broken IPv6 connectivity and lack of fallback to IPv4, causing long timeouts.
- issues/24783
2.5 Issue Discussion Insights
This section will analyze the tone and sentiment of discussions within this project's open and closed issues that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.
- Search skills, Patches, Mode, Codespace, App Programs Full Fr for create programs Ai Autonomic Agent Code and Create programs full free autonomic in reference.
- Toxicity Score: 0.75 (Rapid escalation, aggressive language)
- This GitHub conversation involves a single user posting a comment that is identified as spam, indicating a lack of constructive engagement and potentially triggering moderation concerns.
III. Pull Requests
3.1 Open Pull Requests
This section provides a summary of pull requests that were opened in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.
Pull Requests Opened This Week: 116
Key Open Pull Requests
1. hexagon: support for multi-NPU devices (IQ9, IQ10) and fully asynchronous backend: This pull request overhauls the Hexagon backend to add support for multi-NPU devices like the IQ9 and IQ10 series and implements a fully asynchronous backend with async graph-compute, events, tensor-copy, and cross-device synchronization, including updates to device naming, buffer management, and new scripts for building and running on these devices.
- URL: pull/26501
- Associated Commits: 6da87, 114d8, 40cfe, 65946, f181b, c1f66, edf22, 09e8d, 05667, db297, cae1f, 88dc3, 0800f, 3f4ed, f2a84, 9ad28, d5c5b, 70b96, 77035, 90c89, 46c75, 612ae, fe220, 234a9, 1b6e5, 1b70c, 57133, b6636, 1d008, cbd7a, a2a5c, 4a95c, 922ef, 68c43, 3f842, d593a, a7e83, e025e, ca7e0, cda0a, 3cdaa, d57d0, 4bad4, 4b14b, fc212, 265e6, 4af4f, 6976c, 1ab0d, c838c, dacc0, 1132c, 15b7b, 868d1, 4ddd8, f7b49, 05b71
2. mtmd: support Qwen3-TTS (note: breaking change to llama-tts binary): This pull request adds support for the Qwen3-TTS text-to-speech model to the mtmd framework, including integration of a new speaker encoder, codec embeddings, and a code predictor model, while revamping the llama-tts binary with breaking changes to enable multi-language TTS with speaker reference audio and optimized generation of acoustic tokens within the existing llama.cpp infrastructure.
- URL: pull/26254
- Associated Commits: ae766, 8892b, de0ac, 39b9c, 4da8d, b98af, 894ac, ebd17, 85af6, eea7c, ed0b0, bd38c, 4712d, 2e9e8, d9826, e6888, bf0e6, ff6b2, 84559, 79a99, 13bfd, 51027, 2348d, a9df0, 3e9e4, 1f096, ffe53, 914fd, 06aae, eae9c, 86020, 22bc9, 8f94f, bd52f, 82dc8, 511cc, bfc07, 4523d, 8a402, e298a, f695c, 4a9e3, 026d3, 2307b, dc5dd, 6e4da, 0b6e3, 1df59, d827c, 2f240, 127db
3. model: Add support for Motif 3 Beta : This pull request adds comprehensive support for the Motif 3 Beta model, including implementation of GDLA attention with grouped low-rank projections, a new Grouped PolyNorm activation for routed experts, manifold-constrained hyper-connections optimized with fused operations to reduce graph complexity and improve speed, interleaved SWA and Yarn attention layers with precise scaling adjustments, explicit tensor mapping with selective quantization exclusions, and thorough verification against reference implementations, while also addressing tokenizer compatibility and conversion nuances to enable efficient memory usage and coherent generation up to 32k context length.
- URL: pull/26298
- Associated Commits: 51b71, 743da, 42ce6, de8eb, 77605, c0786, 7f015, c7251, f95f9, 25186, 2746f, e1b14, 65658, bc6b0, 4ff3c, 7a56c
Other Open Pull Requests
- ET backend matrix multiplication improvements: Multiple pull requests enhance the ET backend by adding support for K quantizations, rewriting GEMV decode paths, and introducing high-performance double-buffered matrix-engine GEMM kernels for F16 and F32 precisions. These changes optimize throughput, fix dispatch logic, and improve performance on various hardware including ET-SoC-1 and Llama-3.2-1B models.
- OpenCL and GPU backend optimizations: Several pull requests focus on optimizing OpenCL kernels and GPU backend performance, including decode-side optimizations for K-quant dense models on Adreno GPUs, flash-attention kernel improvements, and performance tuning for Intel Xe-LP GPUs. These optimizations reduce register pressure, improve throughput by up to 32%, and increase prefill performance by 11-12% while maintaining correctness.
- Matrix-engine GEMM kernel enhancements: New and rewritten matrix-engine GEMM kernels for Q4_0, Q5_K, and Q8_0 quantization formats introduce static-reuse dequantization, fused bias support, and adaptive dispatch thresholds. These kernels achieve significant speedups ranging from 1.04x to 9.5x on prefill workloads, improving performance on Llama-3.2-1B and other models.
- CUDA and mixed matrix-vector quantization: A CUDA-based implementation of linear weight repacking combined with mixed matrix-vector quantization (MMVQ) for NVFP4 precision is introduced to optimize data reuse and cache efficiency. Initial benchmarks demonstrate improvements including 32-byte loads, row reuse, and FP16 register savings.
- Model and attention mechanism innovations: The Kimi-K3 text model is introduced featuring hybrid KDA and MLA attention mechanisms with novel architectural enhancements such as cross-layer residual attention and latent Mixture of Experts. Additionally, sliding-window attention support is added for Qwen3 models with per-layer configuration and GPU memory optimizations.
- Speculative and suffix decoding improvements: A model-free suffix decoding method is implemented using an online suffix tree to improve speculative decoding, complemented by CLI options and tests. This enhances decoding efficiency and accuracy in token generation workflows.
- Cache and server endpoint enhancements: A new server endpoint
/slotswith aclone_toaction enables efficient cloning of the key-value cache between slots, allowing multiple parallel requests to share a common prefilled cache prefix. Benchmarks show up to 2.12x speedup on Apple Silicon hardware by reducing redundant prefill computations.
- Metal backend fixes and tuning: The Metal backend is improved by fixing a critical bug in the
mul_mm_idkernel that caused NaN outputs due to activation overflow, and by refactoring the FA-vec tuning process into a standalone binary with a cooldown mechanism to mitigate thermal drift. These changes restore correct outputs and improve tuning reliability on Apple silicon.
- AMD and ROCm support and diagnostics: Continuous integration workflows for AMD ROCm are added targeting new AMD boards, while driver stability is improved by fixing submission batching sizes and adding debug tools to reduce DeviceLost errors on AMD Linux drivers. These efforts enhance testing and diagnostics for AMD GPU users.
- Unit testing and code refactoring: Unit tests are added for ngram-map decoding simulations, and model resolution logic is refactored into a standalone unit to improve testability without network access. These changes increase code reliability and maintainability.
- Chat template updates: The DeepSeek V4 chat templates are updated to fix preview template encoding and add a new Flash 0731 template with different reasoning prompts and tests, improving chat format handling and user experience.
- SYCL kernel optimizations: Flat path optimizations for
*GLUSYCL kernels consolidate shared codepaths and add performance benchmarks showing up to 14% throughput improvements on Intel Arc B70 hardware, along with new SWIGLU test cases.
3.2 Closed Pull Requests
This section provides a summary of pull requests that were closed in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.
Pull Requests Closed This Week: 149
Key Closed Pull Requests
1. Docs/i18n readmes cli logo: This pull request proposes adding documentation and internationalized readme files, a command-line interface, and an ASCII logo for the project, enhancing user guidance and branding without merging the changes.
- URL: pull/26503
- Associated Commits: 5e7ce, 8b781, e2b0b, 51c61, c5887, 5b5e6, bef05, e99aa, 6df8f, dd2a9, 53752, a3d18, ae4f9, a6760, 692b9, a6ec0, a8d15, 8a5ec, d4503, ac173, e1f3b, 7bbbb, d129c, 8c51d, 77107, e660e, 51b20, 93cea, 63ba2, a7e8b, e6550, 1ab2a, 786a1, f180e, c29dc, fd275, e1b01, 605f7, 46981, 31b12, d34af, 41d78, 548e7, bc8e1, 073b7, 6d29d, f0745, 9409e, b5e38, 61ec6, a858e, 8c1ae, 82dcb, 16595, 54519, 445df, ccf4b, 13b48, 5bfcc, f0e53, 5c381, 48ddc, bacae, 8c294, c1719, a5435, bc4c4, cd9e4, 04ce0, b28e9, 9cdfa, f5d3b, 17745, 18fa7, ca27f, 5ec69, 0e7a8, a7830, 31387, 30a49, 900c0, 1aa55
2. mtmd : add Nemotron 3 Nano Omni support (parakeet): This pull request adds support for the subsampling and encoder components of the Nemotron Nemo 3 Nano Omni model by integrating the Parakeet subsampling/encoder code from whisper.cpp, aiming to maintain close code compatibility for easier future patching between the two projects.
- URL: pull/22520
- Associated Commits: 0024a, e5574, a9929, 8e279, ffd1b, 8af10, b5a35, 7ed92, 49658, 9a839, 6ba52, 385b2, cef7f, 681a1, 44cb5, 0cd9e, 78e28, 96b13, 65643, 79e1d, 1fcd9, 5b741, 4f888, 01a1f, c1d46, e49c0, 33783, 816d7, 882c9, 79baf, 8835a, 602c2, 2ab3b, 1337d, dbdd9, 81937
3. 24654 encrypt indexeddb data: This pull request introduces encryption for IndexedDB data by adding an encryption service with passphrase-wrapped data encryption keys, encrypting messages and conversation names at rest, implementing bulk encrypt/decrypt operations for existing data, adding an unlock gate for encrypted conversation data, and providing security settings and encrypted export/import options to enhance data security.
- URL: pull/26274
- Associated Commits: 22f12, 2d977, 1b8a1, 81ad4, 14891, 2361a, fa126, ba499, 0e379, 9c3c9, ed0b3, a60f4, d17f1, cc02c, 868f6, 78c9f, 32b5a, 3f328, 5f1a8, a427d, 46be5, 32e44, baef7, ee9ec, dc75f, 4c092
Other Closed Pull Requests
- Performance optimizations in Metal backend and Apple silicon devices: Multiple pull requests introduce significant performance improvements for Apple silicon by adding tuned lookup tables for kernel configurations, implementing a Fast Walsh-Hadamard Transform kernel, and creating a Metal backend for the Lightning Indexer operation. These changes leverage device-specific optimizations and new kernels to boost throughput and efficiency without sacrificing correctness.
- WebGPU and Vulkan backend fixes and enhancements: Pull requests address binding aliasing issues, improve test coverage, fix zero-sized concat input handling, and add full support for the POOL_1D operation in the Vulkan backend. These updates enhance compatibility and correctness across different GPU architectures and improve shader logic and boundary conditions.
- Model format support and conversion improvements: A pull request fixes tokenizer imports, vocabulary handling, tensor remapping, and loader bugs to enable conversion of the Hugging Face Kimi K3 model checkpoint into the GGUF format compatible with llama.cpp. This allows running the model despite some intentional feature drops affecting output quality.
- Documentation and UI improvements: Updates to the main README include centering badges, adding new links, removing outdated sections, and reorganizing contributing guidelines. Additionally, architectural improvements enhance UI rendering performance by refactoring parsing utilities, optimizing message lookup, and fixing cache invalidation.
- Conversation data handling and IndexedDB reliability: Enhancements ensure atomic message-tree writes, prevent orphaned messages, fix conversation recency ordering, optimize bulk deletes, improve system message editing UX, and implement retryable store initialization. These changes improve data integrity and user experience in conversation management.
- Backend sampling and decoding improvements: A pull request adds support for generating multiple outputs per sequence to enable speculative decoding with configurable parameters, while another migrates penalties sampling from CPU to GPU to improve token generation speed and maintain compatibility with reduced vocabulary samplers.
- Mixture of Tensor Products (MTP) and model accuracy enhancements: Support for MTP in the Qwen3-Next model is added, including performance benchmarking and optional scaling tensors for attention and MoE feed-forward matmuls, resulting in improved accuracy and efficiency across prompt types.
- OpenVINO backend phase splitting and shared KV cache: Introduction of environment variables enables running prefill and decode phases on different devices, with shared KV cache allocation in USM host memory to avoid copying on UMA systems. Integration tests and documentation demonstrate significant performance improvements on mixed CPU+iGPU setups.
- Shell command limiting and security: A simple implementation provides whitelisting and blacklisting of shell tools, basic screening for command delimiters, Python tests, and a tool to view the whitelist, noting some limitations and potential future enhancements.
- SYCL-based flash attention for Intel GPUs: A new SYCL-based flash attention implementation leverages oneMKL GEMM to accelerate key matrix multiplications on Intel GPUs with XMX hardware, significantly improving performance for large context lengths and various KV cache types while maintaining compatibility with existing kernels.
- CUDA backend performance and correctness fixes: Pull requests improve CUDA backend performance by extracting Q2_0 quantized elements with
__byte_permintrinsic and fix data races in theblock_reducefunction by implementing double-buffering and synchronization, ensuring correct and reproducible softmax computations.
- Chunked SSD matrix multiplication for CUDA SSM scan: A new chunked SSD matrix multiplication path accelerates Mamba-2 prefill for long sequences by replacing sequential scans with parallel batched GEMMs and fused tensor-core kernels, improving performance across models and GPUs while preserving original kernels for short sequences and non-NVIDIA hardware.
- Intel GPU driver version check on Windows: A driver version check replaces a previous crash guard to ensure the fast Walsh–Hadamard transform is only used on supported drivers, preventing crashes on older systems and improving performance on Xe2/3 architectures.
- Memory management improvements for tensor views: The
ggml_set_outputfunction is updated to mark source tensors of view tensors as outputs, preventing memory allocator reuse and ensuring proper memory management for views.
- Qwen 3 model parser enhancements: A specialized parser recognizes multiple reasoning end sequences, supports older models omitting certain tags, and reintroduces optimized argument permutation handling to accommodate argument reordering without harsh grammar constraints.
- Stable recurrence and CPU threading optimizations: Performance improvements include a stable recurrence mechanism with Euler step scaling, key-value cache decoupling, a new fuse-gate-up option, CPU threading optimizations, benchmark scripts, and expanded multi-platform documentation.
- Grouped 1D convolution operation for future models: The
ggml_conv_1d_groupedoperation is added to support Compressed Convolutional Attention for future Zyphra models, with optimizations to minimize memory copies and comprehensive tests verifying correctness.
- MoE model hot experts pinning feature: The
--pin-hotexpertsfeature dynamically tracks and pins frequently used experts in RAM usingmlock()to reduce latency spikes and improve throughput, with configurable memory budgets, real-time usage tracking, thread-safe global ranking, and periodic statistics reporting.
3.3 Pull Request Discussion Insights
This section will analyze the tone and sentiment of discussions within this project's open and closed pull requests that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.
Based on our analysis, there are no instances of toxic discussions in the project's open or closed pull requests from the past week.
IV. Contributors
4.1 Contributors
Active Contributors:
We consider an active contributor in this project to be any contributor who has made at least 1 commit, opened at least 1 issue, created at least 1 pull request, or made more than 2 comments in the last month.
If there are more than 10 active contributors, the list is truncated to the top 10 based on contribution metrics for better clarity.
| Contributor | Commits | Pull Requests | Issues | Comments |
|---|---|---|---|---|
| Raudbjorn | 249 | 0 | 0 | 0 |
| ngxson | 175 | 14 | 1 | 18 |
| allozaur | 161 | 8 | 0 | 28 |
| max-krasnyansky | 180 | 2 | 0 | 0 |
| marty1885 | 160 | 1 | 0 | 3 |
| 134 | 0 | 0 | 0 | |
| ggerganov | 64 | 9 | 0 | 39 |
| ServeurpersoCom | 97 | 4 | 0 | 3 |
| wanghqc | 79 | 16 | 0 | 0 |
| pwilkin | 76 | 5 | 0 | 4 |