Weekly GitHub Report for Llama.cpp: July 13, 2026 - July 20, 2026 (21:06:39)
Weekly GitHub Report for Llama.cpp
Thank you for subscribing to our weekly newsletter! Each week, we deliver a comprehensive summary of your GitHub project's latest activity right to your inbox, including an overview of your project's issues, pull requests, contributors, and commit activity.
Table of Contents
I. News
1.1 Recent Version Releases:
The current version of this repository is b4991
1.2 Version Information:
The version released on March 29, 2025, introduces key updates that enhance overall performance and user experience, with notable improvements in system stability and feature optimization. This release reflects a continued focus on refining core functionalities and addressing user feedback.
II. Issues
2.1 Top 5 Active Issues:
We consider active issues to be issues that that have been commented on most frequently within the last week. Bot comments are omitted.
-
[BUG-UNCONFIRMED] Compile bug: GGML_SYCL_DEVICE_ARCH=xe2 gives segfault: This issue reports a segmentation fault occurring when compiling with the
-DGGML_SYCL_DEVICE_ARCH=xe2flag enabled for the SYCL backend on Linux, specifically when running the llama-server with certain environment variables and build options. The user experiences crashes despite attempts to disable various runtime flags, and the discussion reveals that the problem is likely related to the use of Ahead-Of-Time (AOT) compilation triggered by this flag, with recommendations to avoid certain cache persistence settings and to follow the documented build procedures for SYCL.- The comments focus on troubleshooting steps including removing problematic environment variables, avoiding AOT unless necessary, clarifying misunderstandings about AOT usage, and suggesting a clean rebuild without cache persistence; the user confirms the issue persists with AOT enabled but not without, and others share related experiences and advice on proper build configurations and flags.
- Number of comments this week: 27
-
[BUG] [PERFORMANCE] [ROCM] Eval bug: Decreased perfomance because of input layers on CPU for AMD Strix Halo: This issue reports a performance degradation when running the Qwen 3.6 35B MoE Q8 model on an AMD Strix Halo system using the ROCm backend, caused by input layers being processed on the CPU rather than the GPU, which leads to high CPU usage and reduced GPU utilization. The discussion explores the root cause related to synchronous memory copies in ROCm, contrasts it with Vulkan's asynchronous behavior, and investigates potential fixes including offloading input layers to the GPU and backend-specific optimizations, with mixed results across different environments.
- The comments analyze why ROCm suffers from high CPU usage due to synchronous input tensor copies, unlike Vulkan's asynchronous approach; they discuss backend implementation details, share benchmark results, and test various patches and configurations, concluding that while some improvements are possible, the issue is complex and environment-dependent, with ongoing investigation and partial upstream integration of fixes.
- Number of comments this week: 26
-
[BUG-UNCONFIRMED] Eval bug: GGML_ASSERT(n_inputs < GGML_SCHED_MAX_SPLIT_INPUTS) failed on DS v4 Flash on single GPU: This issue reports a failure triggered by a GGML assertion when running the DeepSeek-V4-Flash model with a large context size on a single GPU, likely related to how the CUDA backend handles memory and operation offloading. The problem appears to stem from tensor reshaping and view operations in the model's weight handling, which conflicts with the expected tensor info logic, causing the scheduler to exceed input limits and crash.
- The comments include detailed backtrace analysis confirming the regression from a recent commit, reproduction logs with debug info, and discussion about how certain tensor reshaping in the model conversion process leads to improper device offloading and scheduling errors; a proposed fix involves updating the model conversion to avoid reshaping weights and align with tensor info expectations, with consensus that this approach should resolve the issue.
- Number of comments this week: 15
-
[BUG-UNCONFIRMED] Eval bug: Qwen3-VL image embedding doesn't work: This issue reports a bug where the Qwen3-VL image embedding model fails to process image inputs correctly, returning a server error due to a mismatch between the number of media markers in the prompt and the number of images provided. The problem stems from the server expecting a media marker placeholder in the prompt for each image, which is missing in the current implementation, causing tokenization to fail; a proposed patch adds support for a structured "messages" input format that correctly inserts these markers based on the model's chat template.
- The comments confirm the issue and discuss a patch that fixes the problem by parsing a "messages" field to handle multimodal inputs properly, with detailed code examples and a proposed non-breaking change to the server embedding handler; however, there is skepticism about merging the patch due to project backlog, and users debate whether the fix should be handled client-side or require server changes, ultimately concluding that a server-side update is necessary but unlikely to be merged soon.
- Number of comments this week: 13
-
Eval bug: official Gemma 4 QAT GGUF (google/gemma-4-E2B-it-qat-q4_0-gguf) aborts at vocab load: GGML_ASSERT(id_to_token.size() == token_to_id.size()): This issue reports a deterministic abort during the loading of the official Gemma 4 QAT GGUF model due to a failed assertion that the sizes of
id_to_tokenandtoken_to_idmappings are unequal, caused by duplicate token strings in the model's vocabulary. The user requests maintainers to consider implementing a graceful load error instead of an abort and to discuss whether the loader should tolerate duplicate strings since the official Google export contains these duplicates.- Commenters confirmed the issue affects multiple official Gemma 4 QAT GGUF models due to duplicate token strings introduced in a recent metadata update, with detailed analysis showing exact token collisions; Google acknowledged the problem and released corrected model files that load successfully, while the discussion also covered potential improvements in error handling within the loader.
- Number of comments this week: 8
2.2 Top 5 Stale Issues:
We consider stale issues to be issues that has had no activity within the last 30 days. The team should work together to get these issues resolved and closed as soon as possible.
As of our latest update, there are no stale issues for the project this week.
2.3 Open Issues
This section lists, groups, and then summarizes issues that were created within the last week in the repository.
Issues Opened This Week: 80
Summarized Issues:
- Crashes and Assertion Failures in DeepSeekV4 Flash Model: Multiple issues report crashes and assertion failures in the DeepSeekV4 Flash model related to unimplemented memset functions in the RPC backend, tensor stride mismatches during multi-GPU loading, and large context size handling causing CUDA backend assertion errors. These problems lead to crashes during model initialization, decoding, or first decode steps, often requiring manual workarounds or patches.
- RPC Server Memory Corruption and Use-After-Free Bugs: The ggml-rpc-server suffers from multiple memory safety issues including use-after-free errors due to stale graph references, out-of-bounds writes caused by improper parameter handling, and heap-buffer-overflow vulnerabilities triggered by negative stride values or unchecked tensor data. These bugs cause crashes, undefined behavior, and potential security risks during RPC operations.
- Vulkan and SYCL Backend GPU Issues and Performance Problems: Several issues describe GPU-related problems on Vulkan and SYCL backends including performance drops due to weight eviction on Intel discrete GPUs, GPU hangs on Intel Arc Pro B70, crashes on AMD and Adreno GPUs, and failures in split-mode tensor processing. These issues cause slowdowns, crashes, or incorrect outputs on specific hardware configurations.
- CUDA Backend Crashes and Memory Errors: Multiple CUDA backend issues cause crashes including illegal memory access during vision request processing, out-of-bounds reads during evaluation, premature BF16 compute triggering on pre-Ampere GPUs, VRAM memory leaks with CUDA Graphs, and crashes on Windows with specific GPUs. These errors often relate to flash attention, KV cache layouts, or memory management bugs.
- Model Output and Inference Bugs: Several models exhibit output correctness issues such as repeated token sequences due to out-of-bounds reads in Metal kernels, empty or control token outputs on Metal backend with Mistral Small 4 119B, garbled output on ggml-hexagon HMX backend, and forgetting key details during evaluation with SWA on Gemma 4. These bugs affect transcription accuracy and model consistency.
- Web UI and Tool-Calling Bugs: The Web UI and tool-calling features have multiple issues including incorrect MCP toggle states on new conversations, broken image attachment display, failure to copy text attachments as plain text, instruction injection vulnerabilities via query parameters, and grammar generation failures causing HTTP 400 errors. These problems degrade user experience and pose security risks.
- Model Loading and Initialization Failures: Issues include failed model loading due to duplicate token strings causing assertion failures, OpenVINO backend crashes on first load due to RoPE shape mismatches, and Vulkan backend tensor offset errors preventing model runs on certain hardware. These failures block successful model initialization and inference.
- Performance Degradations and Bottlenecks: Performance issues include slow prefill times for DeepSeek-v4 Flash model, CPU-bound input layer processing on AMD ROCm backend, decoding slowdowns due to inactive KV cache data, and serialized large model loading over RPC causing long load times. These degrade throughput and responsiveness.
- Memory and Resource Management Bugs: Problems include VRAM memory leaks with CUDA Graphs, host memory exhaustion errors when offloading MoE experts on Intel Arc GPUs, and buffer overflow in test harnesses due to incorrect buffer sizing assumptions. These issues cause crashes or resource exhaustion.
- Model-Specific Bugs and Feature Requests: Bugs include tensor name suffix mismatches in hy_v3 MoE expert bias causing loading failures, vision embedding errors in LLaVA mtmd path, and missing generation settings in API responses. Feature requests include adding MonkeyOCRv2 support, server-side read_image tool, multimodal document support in rerank endpoint, and Intern-S2-Preview-397B model integration.
- Server and Tool-Call Logic Bugs: Bugs include server errors due to overlapping parameter tokens in DeepSeek v4 tool calls, inability to rename MCP servers or add multiple with the same name, and llama-server tool-call grammar generator failures with nested string properties. These issues cause server errors and limit configuration flexibility.
- Backend Compatibility and Build Issues: Problems include ROCm 7.14 binaries failing due to missing shared libraries, SYCL backend segfaults with specific device architecture flags, and OpenVINO NPU backend crashes due to race conditions or missing attributes. These hinder deployment on certain platforms.
- Security Vulnerabilities: The llama-server Web UI is vulnerable to instruction injection via query parameters, potentially allowing arbitrary shell command execution and data exfiltration if certain tools and permissive settings are enabled.
2.4 Closed Issues
This section lists, groups, and then summarizes issues that were closed within the last week in the repository. This section also links the associated pull requests if applicable.
Issues Closed This Week: 88
Summarized Issues:
- GPU Backend Compatibility and Performance Issues: Multiple issues report problems with GPU backend compatibility, including AMD ROCm GPU detection failures on Windows, GPU synchronization hazards causing crashes on AMD Radeon GPUs, and Vulkan backend performance regressions and slowdowns on various GPUs such as AMD Radeon RX 9060 XT and ARM Mali integrated GPUs. These issues also cover bugs causing gibberish output or empty outputs on Intel Arc GPUs and NVIDIA DGX Spark, as well as pipeline parallelism inefficiencies on ROCm AMD GPUs, indicating widespread challenges in GPU support and optimization across platforms.
- Crash and Assertion Failures in Model Loading and Execution: Several issues describe crashes and assertion failures during model loading or execution phases, including heap corruption on macOS with embedding models, segmentation faults on Intel Arc GPUs with tensor-parallel options, assertion failures with MTP and Vulkan backend, and crashes related to draft-mtp warmup and speculative decoding. These failures often involve out-of-bounds accesses, invalid vector subscripts, or memory corruption, causing instability in server and CLI operations.
- issues/23072, issues/23301, issues/24280, issues/25828, issues/25873, [issues/25779](https://github.com/issues/25779]
- Bugs and Limitations in llama-server Tool and API Behavior: Multiple issues report bugs in llama-server related to tool execution, response formatting, and context size management, including 400 errors due to structured tool call outputs, improper handling of the
-npparameter causing context size errors, missing or incorrect JSON schema enforcement, and tool-call grammar build failures with nested patterns. These problems lead to incompatibilities with downstream AI agents, silent semantic failures, and server crashes or aborts.
- Speculative Decoding and Draft Mode Bugs: Several issues highlight problems with speculative decoding and draft modes, including crashes due to verification batch size mismatches, draft acceptance rate collapses caused by quantized KV cache bugs, and poor draft acceptance rates with official drafters. These bugs affect the reliability and performance of speculative decoding features, limiting their effectiveness in speeding up token generation.
- Model Conversion and Export Script Errors: Issues with the
convert_hf_to_gguf.pyscript include crashes due to missing keys likeimage_meanin multimodal projector exports and broken parsing of metadata fields for MoE models, causing incorrect GGUF file generation and runtime errors. These bugs hinder smooth model conversion workflows from Hugging Face to GGUF format.
- WebUI and User Interface Bugs: Several issues report UI problems such as missing reasoning settings on mobile devices, incorrect textbox heights, disconnected toggles for tool call progress, and bugs in model tag display. These UI regressions degrade user experience and accessibility across devices and browsers.
- Memory Management and Resource Leaks: Issues include key-value cache not being reclaimed after slot release causing context size errors, significant host RAM usage increases due to large context sizes in DeepSeek-V4, and perceived host memory leaks in llama-server that are actually due to prompt cache accumulation. These problems impact server stability and resource efficiency during long or repeated requests.
- Feature Requests for Speech Recognition and Streaming: Requests include designing a generic API for realtime automatic speech recognition (ASR) with phased implementation steps, and implementing OpenAI's realtime streaming speech-to-text protocol via websocket to enable responsive live audio transcription. These features aim to enhance audio processing capabilities in the project.
- Feature Requests for Tooling and Usability Enhancements: Requests include adding an optional external tool executor or sandbox wrapper for secure tool execution, adding FlashAttention support for OpenCL backend, adding OpenCL support for Gemma4 models on Qualcomm Adreno GPUs, adding power throttling options to reduce heat and noise, and adding a feature to define paired sequences to prevent infinite reasoning loops. These aim to improve security, performance, and user control.
- Build and Compilation Issues: Problems include missing header includes causing build failures on macOS Catalina, CUDA backend crashes due to tensor offset mismatches, and the need for compile-time size fallbacks for Vulkan extensions on Apple Silicon. These issues complicate building and running the project on various platforms.
- Pipeline and Parallelism Limitations: Issues report that pipeline parallelism does not improve prompt processing speed on ROCm AMD GPUs due to lack of peer-to-peer transfers, and that offloaded-MoE prefill processes suffer from GPU idle time due to serial host-to-device copies without overlap, suggesting room for optimization in parallel execution and data transfer.
- Model Loading and Runtime Errors: Reports include unknown model architecture errors when loading specific GGUF files, failures to load Ternary 1.58-bit models on Windows CUDA backend, and crashes due to missing or invalid tensor data during model loading or runtime. These errors prevent successful model usage in certain environments.
- issues/24001, [issues/25824](https://github.com/issues/25824]
- WebUI MCP Panel and Server Management Bugs: Issues include removal of the llama-proxy checkbox, lack of timeout causing indefinite lockouts when reaching MCP servers, and toggles that delete MCP servers instead of disabling them, leading to poor server management experience.
- Context and Prompt Handling Bugs: Problems include duplicate model entries caused by model presets, keep_first feature incorrectly retaining only the first image token, and context number in the WebUI not increasing monotonically causing extra recomputation. These bugs affect prompt processing correctness and UI consistency.
2.5 Issue Discussion Insights
This section will analyze the tone and sentiment of discussions within this project's open and closed issues that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.
Based on our analysis, there are no instances of toxic discussions in the project's open or closed issues from the past week.
III. Pull Requests
3.1 Open Pull Requests
This section provides a summary of pull requests that were opened in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.
Pull Requests Opened This Week: 94
Key Open Pull Requests
1. ui: Sidebar Conversations Bulk Action + Improved Settings logic/UI: This pull request introduces bulk action capabilities for sidebar conversations along with enhanced settings logic and user interface improvements, including unified dialog components, optimized bulk deletion handling, and better multi-selection support via keyboard.
- URL: pull/25815
- Associated Commits: 4ca6b, 29820, c4c89, 139a4, c5f88, 33071, fefed, 52d59, fd4ed, bdc8d, e03cf, 887dd, 875aa, 9bcd0, 0deb4, 1b53d, 7ac77, 18802, 45b53
2. Thinking sampling parametrization: This pull request introduces the ability to apply different sampling parameters, such as temperature, separately to the "thinking" and "non-thinking" sections of model output, enabling more nuanced control over token generation by maintaining distinct sampler chains and activating overrides based on reasoning start/end tags, specifically demonstrated for the Qwen 3.6 27B model.
- URL: pull/25709
- Associated Commits: 457b3, e12ba, 43d7d, 16e75, 6876b, ceb3f, 6a26b, e9f22, 30a1c, fa88e, fb247, e01bf
3. vulkan: use spec constant for matrix matrix multiplication A-type: This pull request attempts to optimize Vulkan matrix multiplication shaders by introducing a specialization constant for the type of the A input matrix, replacing static shape definitions with a map for improved tuning and flexibility, while addressing compatibility issues with Nvidia hardware related to shared memory requirements.
- URL: pull/25773
- Associated Commits: cde70, a040f, 30a97, 5954d, 00065, 4c589, 537ce, 0e843, 39897, 496b0, b3edd, bdccb
Other Open Pull Requests
- Metal backend performance and fixes: Multiple pull requests enhance the Metal backend by introducing a compile-time per-device tuning table for optimized kernel configurations, implementing gated_delta_net cache fusion to improve recurrent-state snapshots, and fixing crashes caused by unsupported F16 variants with defensive error handling. These changes collectively improve performance and stability across various models and configurations.
- [pull/25750, pull/25788, pull/25648]
- DeepSeek V4 and related multi-turn reasoning improvements: Pull requests fix and align DeepSeek V4 parsing and encoding to the official implementation, add the DeepseekV4 MTP feature for significant speedups, and improve reasoning behavior in the web UI by adding a "Default" option and removing obsolete code. These updates enhance multi-turn usage, reasoning control, and performance on related benchmarks.
- [pull/25891, pull/25784, pull/25846]
- Speculative decoding and MTP enhancements: Several pull requests introduce an adaptive draft length heuristic for mtp, add support for Gemma4 MTP speculative decoding with AI-assisted code, and improve Vulkan backend performance on AMD hardware by disabling MMVQ during small token speculative-decode steps. These changes optimize draft generation and increase throughput and acceptance rates.
- [pull/25726, pull/25679, pull/25666]
- CUDA backend optimizations and sparse attention support: Pull requests add support for sparse KV indices in the MMA flash attention kernel, introduce XOR swizzling to reduce shared-memory bank conflicts, and improve NVFP4 W4A4 activation quantization with per-channel scaling and fused kernels. These improvements enhance performance and enable efficient sparse attention computations on CUDA devices.
- [pull/25917, pull/25635, pull/25730]
- Prompt prefill and server tooling enhancements: One pull request introduces optional disaggregated prompt prefill workers to prevent blocking decode slots, while another adds experimental support for connecting llama-server to external MCP servers with lazy spawning and cross-platform process handling. These features improve prompt handling and extend tool integration capabilities.
- [pull/25675, pull/25736]
- Backend and build system improvements: Updates include switching Windows build configuration to use LLVM's redistributable OpenMP library for Windows on Arm, adding guidelines for submitting new backend proposals to CONTRIBUTING.md, and adjusting ggml backend logic for correct offloading and device placement at CPU-GPU transitions. These changes streamline development and ensure proper backend execution.
- [pull/25782, pull/25820, pull/25832]
- Bug fixes and UI improvements: Pull requests fix a server-side crash by returning 400 errors on validation failures, correct single-model mode behavior in the web UI to prevent erroneous load prompts, and update the chat differential template analyzer to recognize "thinking" as an alias for "reasoning_content." These fixes improve stability and user experience.
- [pull/25738, pull/25701, pull/25695]
- Graph and attention state management: One pull request refactors non-causal attention handling in the mtmd module using RAII for robust state management, while another modifies the llama-graph component to assert tensor views cannot be outputs, ensuring correct data reuse by the graph allocator. These changes improve internal consistency and resource management.
- [pull/25723, pull/25729]
3.2 Closed Pull Requests
This section provides a summary of pull requests that were closed in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.
Pull Requests Closed This Week: 135
Key Closed Pull Requests
1. hexagon: L2 cache handling rework (dirty bit tracking with lazy flushing) and more MUL_MAT updates: This pull request reworks the Hexagon L2 cache handling by replacing per-operation eager flushes with a dirty-bit tracking and lazy flushing model, introduces tiled activation processing and robustness improvements for MUL_MAT operations, optimizes multi-threaded cache flushes with scalable strategies based on flush size, restructures core internal memory layout for faster cache recovery, and includes a comprehensive API cleanup for work queues, resulting in measurable performance gains across supported Hexagon NPU devices.
- URL: pull/25762
- Associated Commits: 77909, 836eb, 8f3c7, c482f, 89c5e, a6138, 0b35b, dcb68, 38fd0, c6188, a85f7, 54a4b, 37d5d, 398cf, 97c3f, 61f05, ac7d4, 04312, 6dc25, a8626, 97dfa, fa958, fb734, ab5c4, 5eb80, cddf0, 56d78, 4f831, 80ae6, f912d, e5132, 0c123, e9764, 3e29f, c27ad, d9008, 3d47e, 44c8e, 7b0c7, 4d3a7, 34d46, 1a95b, d677d, 0497a, 7d60d, 29b1c, 335d9, 3541c, cacbf
- Associated Commits: 77909, 836eb, 8f3c7, c482f, 89c5e, a6138, 0b35b, dcb68, 38fd0, c6188, a85f7, 54a4b, 37d5d, 398cf, 97c3f, 61f05, ac7d4, 04312, 6dc25, a8626, 97dfa, fa958, fb734, ab5c4, 5eb80, cddf0, 56d78, 4f831, 80ae6, f912d, e5132, 0c123, e9764, 3e29f, c27ad, d9008, 3d47e, 44c8e, 7b0c7, 4d3a7, 34d46, 1a95b, d677d, 0497a, 7d60d, 29b1c, 335d9, 3541c, cacbf
2. ui: Agentic Content UX improvements: This pull request implements a series of user experience improvements to the Agentic Content UI, including enhanced animations, redesigned collapsible content components, conditional setting displays, refined styling, new tool integrations, improved auto-scroll behavior, and various fixes and refactors to streamline interaction and presentation within the chat interface.
- URL: pull/25450
- Associated Commits: f2e07, ea0b9, a7ee0, 0b990, 12f3f, 698c2, 07b3d, bc396, 1f93f, 6a815, 7f138, c5c60, 53991, 4638d, 44cfa, e33e7, 3969a, 2593d, b0451, 18427, bf5d6, dd35b, 7e87b, 4fa0f, dd31e, d46bf, 8e202, 18253, 81d11, 4a158, 7865c, 570bd, ed92d, c2161, e37fa
- Associated Commits: f2e07, ea0b9, a7ee0, 0b990, 12f3f, 698c2, 07b3d, bc396, 1f93f, 6a815, 7f138, c5c60, 53991, 4638d, 44cfa, e33e7, 3969a, 2593d, b0451, 18427, bf5d6, dd35b, 7e87b, 4fa0f, dd31e, d46bf, 8e202, 18253, 81d11, 4a158, 7865c, 570bd, ed92d, c2161, e37fa
3. Infcore: This pull request introduces the "infcore" framework, a separate modular layer built around the llama.cpp engine without modifying its core, providing an OpenAI-compatible gateway with authentication, RBAC, audit logging, model registry, configuration validation, deployment tooling, and extensive operational hardening for secure, offline, and compliant local deployment primarily targeting the Russian Federation environment.
- URL: pull/25662
- Associated Commits: 4f832, a63da, f8996, b1a09, d033c, 70898, 1a397, 8d5fe, c4694, 219e2, 4f903, 2cfaa, 87b9a, cc52a, 6657c, dd9b3, 93a58, e1cd4, a39c0, 78624
- Associated Commits: 4f832, a63da, f8996, b1a09, d033c, 70898, 1a397, 8d5fe, c4694, 219e2, 4f903, 2cfaa, 87b9a, cc52a, 6657c, dd9b3, 93a58, e1cd4, a39c0, 78624
Other Closed Pull Requests
- User Interface Enhancements: This pull request revamps the server's UI by adding a fixed header and footer, a mode changer, chat input, and a styled sidebar for configurations, while improving range input styling and integrating a zero md web component for markdown with syntax highlighting. It also enables loading custom prompts from an external repository without altering the original UI code structure.
pull/4400
- SYCL Flash Attention Improvements: Two pull requests introduce and optimize SYCL-based flash attention mechanisms using oneDNN's fused scaled dot-product attention (SDPA) primitive and the XMX engine on Intel Xe2 hardware. These changes significantly improve performance and prefill speed while maintaining correctness and fallback compatibility.
pull/25312, pull/25222
- DeepSeek and MoE Model Support: Multiple pull requests add comprehensive support for Tencent's Hy3 299B MoE model and DeepSeek-V4 speculator models, including architecture extensions, speculative decoding heads, draft sampling improvements, and integration with existing decoding machinery. These contributions also include validation, performance improvements, and fixes for rotary pairing and draft acceptance statistics.
pull/25395, pull/25683, pull/25623, pull/25585, pull/25823
- MCP Panel UI Fixes and Server Stability: This pull request fixes multiple issues in the MCP panel UI, restoring important toggles, ensuring clean failure on unreachable servers, maintaining server visibility, and applying timeout settings live. It also consolidates settings and adds unit tests to prevent regressions related to server toggling.
pull/25631
- Memory Detection and CUDA Backend Fixes: A pull request addresses incorrect UMA memory detection on AMD APUs by using
hipMemGetInfo()for HIP/ROCm builds and enabling UMA only via an environment variable. Additional fixes include relaxing tensor contiguity requirements for CUDA backend concatenation to support multi-stream quantized KV cache tensors and refactoring CUDA MMQ kernel configuration for flexible tuning and improved compilation.
pull/20472, pull/25649, pull/24127
- Quantization and Kernel Bug Fixes: Several pull requests fix bugs in quantized MUL_MAT implementations for Adreno GPUs, correct SYCL kernel row calculations and guard checks, and implement rotation fixes in the dflash module's key/value quantization. These fixes resolve issues causing incorrect outputs, corrupt results, and positional signal problems.
pull/25910, pull/25690, pull/25678, pull/25823
- Data Type and Backend Support Enhancements: This pull request standardizes the im2col output data type to prevent precision loss and crashes, adds full f16 support for SET_ROWS in Vulkan and CPU backends, and improves backend tests and hardware-specific tuning.
pull/23660, pull/25432
- Server and Prompt Cache Refactoring: A pull request refactors server prompt cache state ownership by clearing checkpoints when prompts are cleared, separating prompt state data from KV cache data, and fixing edge cases in batch processing to improve stability.
pull/25649
- Security and CORS Configuration: This pull request adds four new
--cors-*command-line options to the llama-server to provide configurable CORS settings, addressing security concerns from the new--agentmode by allowing users to restrict origins, methods, headers, and credentials while maintaining backward compatibility.
pull/25655
- Performance and Testing Improvements: Pull requests optimize quantized data loading for q4_K and q5_K operations by changing data types for better GPU compiler performance, enhance test coverage by creating a small qwen35 model for recurrent state rollback tests, and improve continuous integration by adding Ubuntu 20.04 Linux binaries and reorganizing dynamic library placement.
pull/25758, pull/25780, pull/25758
- Server Error Handling Fix: This pull request fixes a server error where invalid POST requests with the X-Conversation-Id header returned a 500 error instead of a 400 validation error by adding a guard to prevent invoking uninitialized functions during validation failure.
pull/25623
3.3 Pull Request Discussion Insights
This section will analyze the tone and sentiment of discussions within this project's open and closed pull requests that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.
Based on our analysis, there are no instances of toxic discussions in the project's open or closed pull requests from the past week.
IV. Contributors
4.1 Contributors
Active Contributors:
We consider an active contributor in this project to be any contributor who has made at least 1 commit, opened at least 1 issue, created at least 1 pull request, or made more than 2 comments in the last month.
If there are more than 10 active contributors, the list is truncated to the top 10 based on contribution metrics for better clarity.
| Contributor | Commits | Pull Requests | Issues | Comments |
|---|---|---|---|---|
| allozaur | 139 | 4 | 0 | 23 |
| marty1885 | 150 | 0 | 0 | 2 |
| am17an | 72 | 11 | 0 | 24 |
| Raudbjorn | 97 | 0 | 0 | 0 |
| max-krasnyansky | 93 | 1 | 0 | 0 |
| ServeurpersoCom | 56 | 10 | 0 | 17 |
| ggerganov | 34 | 9 | 0 | 24 |
| pwilkin | 56 | 2 | 0 | 9 |
| ngxson | 61 | 1 | 0 | 1 |
| timkhronos | 62 | 1 | 0 | 0 |