Weekly GitHub Report for Llama.cpp: May 19, 2025 - May 26, 2025 (12:00:59)
Weekly GitHub Report for Llama.cpp
Thank you for subscribing to our weekly newsletter! Each week, we deliver a comprehensive summary of your GitHub project's latest activity right to your inbox, including an overview of your project's issues, pull requests, contributors, and commit activity.
Table of Contents
I. News
1.1 Recent Version Releases:
The current version of this repository is b4991
1.2 Version Information:
This version, created on March 29, 2025, introduces key updates and changes, though specific details are not provided in the data. Notable highlights or trends cannot be identified without additional information.
II. Issues
2.1 Top 5 Active Issues:
We consider active issues to be issues that that have been commented on most frequently within the last week. Bot comments are omitted.
-
Eval bug: Segmentation fault when loading SmolVLM-500M-Instruct-Q8_0.gguf on Termux / Android ARM64, only in Termux, not in Prooted ones, other gguf work fine: This issue involves a segmentation fault occurring when attempting to load the SmolVLM-500M-Instruct-Q8_0.gguf model on Termux/Android ARM64, which does not occur in Prooted environments, indicating a potential problem with the Termux setup or the specific model file. The problem is suspected to be related to malformed or corrupted strings in the model's metadata, which may cause undefined behavior during the parsing process, particularly when using
std::ostringstream.- The comments discuss various attempts to resolve the issue, including rebuilding with address sanitizer enabled, which led to linking issues with the sanitizer library. Suggestions include obtaining a full call stack using gdb, which reveals a segmentation fault in the minja template rendering process, possibly due to null or invalid data structures. The issue appears to be specific to Termux's environment, as it does not occur in other Linux environments like Ubuntu or Prooted Debian.
- Number of comments this week: 11
-
Compile bug: ‘ggml_gelu_erf’ was not declared in this scope; did you mean ‘ggml_gelu’: This issue is about a compilation error encountered while trying to build the
llama.cppAUR package on a Linux system, where the functionggml_gelu_erfis not recognized, suggesting a possible oversight in the recent commit that introduced this function. The error suggests usingggml_geluinstead, indicating a potential mismatch or missing synchronization between thellama.cppandggmlrepositories.- The comments discuss the need to sync changes between repositories, with a specific commit mentioned as a potential solution. There is a discussion about a broken
DIVoperation that needs fixing before the final sync, with suggestions to either revert changes temporarily or directly fix the issue. The conversation concludes with a plan to merge a revert if no updates occur by the next day. - Number of comments this week: 9
- The comments discuss the need to sync changes between repositories, with a specific commit mentioned as a potential solution. There is a discussion about a broken
-
Misc. bug: Speed degradation in
bin-win-cpu-x64compared tobin-win-avx2-x64on Intel Core i7-12700H: This issue reports a significant speed degradation in evaluation time on Windows 11 using an Intel Core i7-12700H CPU when running thellama-cliwith thebin-win-cpu-x64build compared to thebin-win-avx2-x64build. The problem persists in the latest release, with evaluation times being approximately ten times slower than in previous builds, and the issue is traced back to a specific commit.- The comments discuss testing with fewer threads, which shows no speed degradation with
-t 6, and another user reports a similar issue with a different setup, suggesting it might be related to a recent update. It is clarified that the issues are different, and the user is advised to open a new issue for their problem. The final comment indicates that the latest release should have resolved the original issue. - Number of comments this week: 6
- The comments discuss testing with fewer threads, which shows no speed degradation with
-
webui: First user prompt sometimes disappears after sending: This issue describes a bug in the llama.cpp project where the first user prompt sometimes disappears after being sent, resulting in the assistant generating a response based on the system message instead of the user's input. The problem is intermittent, primarily occurring when starting a new conversation, and is suspected to be related to a constraint error in the transaction due to ID collisions.
- Multiple users have experienced the issue across different operating systems and browsers, with one suggesting that switching from timestamp-based IDs to UUIDs or a static counter could resolve the problem. Another user can consistently reproduce the issue by quickly submitting a prompt, indicating a potential timing or storage-related cause. A call for a pull request to fix the issue has been made.
- Number of comments this week: 4
-
Eval bug: swa_full = true is slower than false: This issue reports a performance discrepancy in a GitHub project where setting
swa_full = trueresults in slower execution times compared to when it is set to false, contrary to the expectation that context caching should improve performance. The user provides detailed test results and observations, noting a significant increase in memory usage and questioning whether this behavior is a bug or expected.- The comments discuss similar observations by another user, an explanation that the behavior is expected due to increased memory usage and slower computation with
--swa-full, and a clarification that--swa-fullis not beneficial forllama-cliorllama-benchunless specific features like context shift are needed. - Number of comments this week: 4
- The comments discuss similar observations by another user, an explanation that the behavior is expected due to increased memory usage and slower computation with
2.2 Top 5 Stale Issues:
We consider stale issues to be issues that has had no activity within the last 30 days. The team should work together to get these issues resolved and closed as soon as possible.
- Kompute-based Vulkan backend shows an GGML_OP_GET_ROWS error: This issue pertains to a problem with the Kompute-based Vulkan backend, which is generating a GGML_OP_GET_ROWS error. It is noted that this error does not occur with the other Vulkan backend, suggesting a specific incompatibility or bug within the Kompute-based implementation.
- Question: How to generate an MPS gputrace: This issue is about a user seeking guidance on how to generate a Metal Performance Shaders (MPS) gputrace for the llama.cpp project during model inference, as part of efforts to enhance the Metal backend for a project at Hugging Face. The user is specifically interested in obtaining a debugger output similar to what is provided by the Metal Debugger in Xcode, and is inquiring if there is a documented or known method to achieve this.
- common: download from URL, improve parallel download progress status: This issue addresses the need to improve the progress status display for parallel downloads when retrieving sharded models, as the current implementation causes conflicts in the progression indicators. The proposed solution involves properly implementing the
CURLOPT_NOPROGRESSoption to ensure accurate and non-conflicting progress reporting during these parallel download operations. - kubernetes example: This issue is about the need for a Helm chart for the
llama.cppserver to facilitate its deployment on Kubernetes, which is a popular platform for managing containerized applications at scale. The author has initiated the development of this chart and is seeking assistance from the community to continue the work. - Misc. bug: Retrieval sample not decoding token successfully: This issue pertains to a bug in the llama.cpp project where the retrieval sample fails to decode tokens successfully due to a problem with the kv-cache logic when pooling is active. The problem arises because a specific conditional check was removed, leading to an unintended search for an unused slot, which results in decoding errors and the inability to retrieve embeddings for tokens.
2.3 Open Issues
This section lists, groups, and then summarizes issues that were created within the last week in the repository.
Issues Opened This Week: 36
Summarized Issues:
- Feature Requests: The llama.cpp project has several feature requests aimed at enhancing its functionality. These include updating the README for optimal tensor placement on GPUs, supporting new architectures like Qwen and Falcon-H1, and adding video file input support. Additionally, there are requests for improving performance metrics in tools and enhancing speculative decoding functions.
- Bugs in Model Execution and Evaluation: Several issues have been reported regarding bugs in model execution and evaluation processes. These include problems with model splitting across GPUs, garbled outputs, and unexpected halts in generation. Additionally, there are issues with specific models not utilizing CUDA properly, leading to performance degradation.
- Compilation and Build Errors: The llama.cpp project faces several compilation and build errors. These include issues with GPU detection during builds, missing directories for Vulkan shaders, and unrecognized functions due to unsynchronized changes. These errors hinder the build process and require manual workarounds or fixes.
- Memory Management and Performance Issues: There are several issues related to memory management and performance in the llama.cpp project. These include memory not being freed correctly, significant speed degradation on specific builds, and performance discrepancies due to context caching settings. These issues affect the efficiency and speed of model execution.
- Bugs in User Interface and API: The llama.cpp project has reported bugs in its user interface and API. These include disappearing prompts in the web UI, incorrect parameter functioning in the API, and server crashes under high load. These issues impact user interaction and server reliability.
- Model Quantization and Tokenization Issues: The project faces issues with model quantization and tokenization. These include failures in quantizing models with specific tokenizers and inaccurate outputs in local model runs. These issues affect the model's ability to process and generate accurate results.
- Runtime and Execution Errors: Several runtime errors have been reported in the llama.cpp project. These include segmentation faults, crashes during model loading, and errors related to environment variables. These errors disrupt the execution of models and require debugging to resolve.
- Miscellaneous Bugs and Requests: The project also includes various other bugs and feature requests. These range from outdated files affecting GPU support to inquiries about open-source datasets for quantization. These issues and requests contribute to the ongoing development and improvement of the project.
2.4 Closed Issues
This section lists, groups, and then summarizes issues that were closed within the last week in the repository. This section also links the associated pull requests if applicable.
Issues Closed This Week: 17
Summarized Issues:
- Server Crashes and Memory Issues: This topic covers server crashes and memory allocation problems in various models and backends. The Phi-4-mini-reasoning-Q8_0.gguf model with Vulkan backend on an AMD RX 7600 GPU experiences crashes potentially due to memory allocation or buffer operation errors. Similarly, a segmentation fault occurs in the
llama-clitool with the phi-4 model on an NVIDIA GeForce RTX 3060 and AMD Ryzen 7 3800X, with a suggested workaround of downgrading to an earlier version.
- Performance Discrepancies and Backend Issues: This topic addresses performance issues and backend inefficiencies. The HIP backend underperforms on an AMD Ryzen AI MAX 395 system compared to the Vulkan backend, possibly due to inefficient rocBLAS kernels. Additionally, a significant performance discrepancy is noted between the Qwen3 32B and Qwen3moe 30B.A3B models, with the latter being faster due to its Mixture-of-Experts architecture.
- CUDA and GPU Allocation Problems: This topic involves issues with CUDA compatibility and GPU allocation. On a Jetson AGX Xavier device, CUDA fails to initialize due to an insufficient driver version after an 'apt update'. Similarly, the HIP/ROCm memory allocation fails to detect the GPU in release b5450, defaulting to CPU allocation, which was resolved in a later release.
- Assertion Failures and Crashes: This topic covers assertion failures leading to crashes in various scenarios. An assertion failure occurs in the llama.cpp project when using embedding models with mean pooling, especially with multiple slots. Another assertion failure in the
llama-ttstool occurs when processing longer lines of text, causing the tool to abort.
- Token Generation and Output Issues: This topic involves issues with token generation and output. The llama.cpp project experiences repetitive outputs when the -np parameter is set greater than 1. Additionally, a significant slowdown in token generation rate is observed in the llama-server software and the Qwen3-30B-A3B model on Windows, affecting performance.
- Script and Configuration Bugs: This topic covers bugs related to scripts and configurations. The
llama-batched-benchscript generates no output due to incorrect configuration of parallel sequences. A bug in the UGM tokenizer of thenomic-embed-text-v2-moemodel results in tokens being output in the wrong order.
- Vulkan and OpenCL Backend Issues: This topic involves issues with the Vulkan and OpenCL backends. A pre-allocated tensor in a Vulkan buffer cannot execute a copy operation, causing crashes during KV cache defragmentation. The OpenCL backend crashes when encountering unsupported operations, with a suggested CPU fallback mechanism.
- Conversion and Overflow Warnings: This topic covers warnings during model conversion. A runtime warning of an overflow is encountered during the conversion of a trained 32B Lora model to the gguf format, initially thought to be a hardware issue but later linked to a bad drive.
2.5 Issue Discussion Insights
This section will analyze the tone and sentiment of discussions within this project's open and closed issues that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.
Based on our analysis, there are no instances of toxic discussions in the project's open or closed issues from the past week.
III. Pull Requests
3.1 Open Pull Requests
This section provides a summary of pull requests that were opened in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.
Pull Requests Opened This Week: 26
Key Open Pull Requests
1. [CANN]: add the basic supports of Flash Attention kernel: This pull request introduces basic support for the Flash Attention (FA) kernel in the CANN backend, specifically for F16 KV tensors without logit softcap, and has been tested on the Ascend 910B platform, as detailed in the commits and documentation updates.
- URL: pull/13627
- Merged: No
- Associated Commits: 72df3, 3a731, 6a39d, 8a902, f5e24, c8c29, 47f2c, fb62f, b266b, 8a112, 1779e, 092cc, c3803, 89f88, 1a3bf, 3b084, d2369, 8a782
2. kv-cache : rework kv_cell: This pull request reworks the key-value cell editing logic in the llama-kv-cache.cpp by introducing a new struct llama_kv_cells_unified in src/llama-kv-cells.h, which simplifies the implementation by automatically tracking used cells and the has_shift flag, improves cache locality with a structure of arrays, and replaces the sequence tracking mechanism with a std::bitset for better efficiency.
- URL: pull/13706
- Merged: No
3. add GGML_USE_NUMA_MIGRATE feature to optimize cross NUMA op computation: This pull request introduces the GGML_USE_NUMA_MIGRATE feature to optimize cross-NUMA operation computation by addressing cross-NUMA memory access bottlenecks, enhancing the ggml_barrier() for cross-NUMA scenarios, adding a build option to enable this feature, and providing a command-line option to migrate pages across NUMA nodes, resulting in significant performance improvements when running the llama3 model on systems with multiple NUMA nodes.
- URL: pull/13649
- Merged: No
Other Open Pull Requests
- Jina Embeddings V3 Model Support: This topic involves the introduction of work-in-progress support for the Jina Embeddings V3 model, focusing on tasks such as model conversion, inference, and vocabulary implementation. The pull request also addresses specific issues but requires further work on LoRA task conversion and prompt prefix selection.
- Audio Model Support: This topic covers the addition of support for the Qwen2-Audio and SeaLLM-Audio models, despite challenges with the Qwen2-Audio model's performance and hallucination issues. The pull request includes updates to documentation and discussions to address these challenges.
- SYCL Debugging Enhancements: This topic involves the introduction of additional debugging prints to the SYCL operations, enhancing the logging of operation calls and completions. The pull request provides detailed information about operations' destination and input tensors, including specific debug prints for matrix multiplication and conversion kernels.
- OpenCL and oneDNN Integration: This topic addresses the issue of the nightly dpcpp compiler no longer shipping with libOpenCL.so by adding a
find_package(OpenCL)call. The pull request ensures that the targetOpenCL::OpenCLis available, resolving linking issues with the llama project.
- Web UI File Attachment Editing: This topic introduces a feature allowing users to edit file attachments when editing messages in the web UI. The pull request includes enabling this functionality, refactoring the ChatInput component, updating the build, and reverting a style change.
- Custom Modals Implementation: This topic involves replacing default alert and confirm dialogs with custom modals due to security restrictions in VS Code's Webview. The pull request introduces a Modal Provider, increases the z-index of modal dialogs, and updates the index.html.gz file.
- llama_kv_cache Interface Simplification: This topic aims to simplify the abstract interface of the
struct llama_kv_cacheby adapting the recurrent cache to the new interface. The pull request tests the optimization workflow and plans to handle compute errors fromllama_decode.
- nomic-bert-moe Mask Token Fix: This topic addresses an issue with the
nomic-bert-moemask token by fixing its conversion and setting thelstriptoken attribute at runtime. The pull request also corrects a vocab padding error to ensure logical consistency.
- Memory Hierarchy Unit Tests: This topic introduces a scaffold for unit tests targeting the memory hierarchy, ported from a previous pull request. The pull request facilitates the development of a hybrid cache implementation, noting that tests for the new iSWA memory are not yet included.
- pyproject.toml Update: This topic updates the
pyproject.tomlfile to align with the latest standard format by addressing deprecated fields in Poetry's configuration. The pull request adds support foruv, ensuring backward compatibility while preparing for future changes in dependency management.
- CMAKE_CUDA_COMPILER Fix: This topic addresses an issue where the CMAKE_CUDA_COMPILER was not being found. The pull request includes a fix to resolve this error in the project.
- MLA kv Cache System Fix: This topic addresses an issue in the MLA kv cache system where computation was unexpectedly assigned to the GPU backend. The pull request ensures that the output node for MLA is forced to the CPU backend, aligning computation with the user's configuration.
- 64-bit Platform Optimization: This topic aims to optimize data structures for 64-bit platforms by aligning them to reduce CPU cacheline size. The pull request improves performance through decreased costs in copying, moving, and creating object structures.
- xtheadvector Extension Support: This topic introduces support for the xtheadvector extension in the GGML project, enhancing k-quant support for the older RVV v0.7.1 implementation. The pull request updates zfh extension detection and provides performance evaluations and build instructions.
- SYCL Graph Update Simplification: This topic aims to remove templates from the
soft_max_f32_submitterfunction to facilitate updates to the SYCL graph. The pull request ensures that only kernel parameters are used, preventing failures due to differing node types.
- NUMA Memory Access Optimization: This topic addresses the issue of cross-NUMA memory access penalties in multi-node systems by introducing an
mbindcall. The pull request ensures optimal NUMA locality and includes necessary NUMA headers and updates the build to link against the NUMA library.
- SYCL Copy Kernels for kv Cache: This topic involves implementing several copy kernels for the same quantized type in SYCL to support kv cache defragmentation. The pull request includes initial tests passing but requires further testing before merging.
- CMake RPATH Configuration: This topic modifies the CMake configuration to set the
RPATHto$ORIGINon Linux. The pull request allows the commands to be executed from any working directory.
- Moondream2 Model Introduction: This topic introduces the Moondream2 model to the project and updates the GGUF model to the latest version compatible with llama.cpp. The pull request fixes a link to ggml.org and addresses issues related to the default chat template and model optimization.
- SYCL GELU Kernel Introduction: This topic introduces a new SYCL kernel for the Gaussian Error Linear Unit (GELU) function. The pull request specifically implements the error function (erf) variant as part of the ggml-org/llama.cpp project.
- SYCL Code Reversion: This topic temporarily reverts a previous change to the SYCL code due to issues with the fp16 DIV operation. The pull request details the reversion in the commit with SHA 4470bcd5e6736954b32c811a912d1ede912a39e2.
- SYCL mrope Kernel Introduction: This topic introduces a new SYCL-based mrope kernel to the llama.cpp project. The pull request is indicated by the commit message and lacks additional descriptive information in the body.
- GGML_VULKAN_PERF Feature Reintroduction: This topic reintroduces the GGML_VULKAN_PERF feature, previously removed in pull request #9118. The pull request sets it up to submit operations individually, similar to GGML_VULKAN_CHECK_RESULTS, with successful test results indicating the feature's functionality.
3.2 Closed Pull Requests
This section provides a summary of pull requests that were closed in the repository over the past week. The top three pull requests with the highest number of commits are highlighted as 'key' pull requests. Other pull requests are grouped based on similar characteristics for easier analysis. Up to 25 pull requests are displayed in this section, while any remaining pull requests beyond this limit are omitted for brevity.
Pull Requests Closed This Week: 65
Key Closed Pull Requests
1. mtmd : add ultravox audio input: This pull request introduces an audio input feature specifically for the Ultravox model, which is a fine-tuned Whisper encoder with a custom projector, allowing the conversion of PCM audio to mel spectrograms for transcription, and includes updates to the API to support audio input, while also deprecating certain image-related APIs in favor of a more unified media approach.
- URL: pull/13623
- Merged: 2025-05-22T18:42:48Z
- Associated Commits: 4fa0c, 8b731, 4ac79, 42824, 45cdb, f3605, 1804f, bc708, de20a, bbe49, 4d444, 8d7d7, dce79, f1518, 9a0dc, 1a903, 4a8c0, 6f23a, cf38b, cf4f5, 3bbb2, 3ce96, cf961, 23d0d, 7033a, e7c8a, 111c8, e6416, 36a1a, 544f4, 7602e, 9afb3, 10779
2. mtmd : (WIP) add vision support for llama 4: This pull request aims to add vision support for Llama 4 by enabling image inference with a resolution limit of 336x336, although the perceived image is currently incorrect, and it critiques the official implementation for its complexity and redundancy in handling vision support features.
- URL: pull/13282
- Merged: 2025-05-19T11:04:15Z
- Associated Commits: c912c, a67a1, 10db5, 55ad3, c50e6, 8775b, 7341e, 893ad, 15605, 32a62, 97a5c, c6c2d, 224cb, 9d1a4, 532c3
3. sycl : reviewing the backend documentation: This pull request involves reviewing and updating the documentation and examples for the SYCL backend, including the removal of an unclear seed from examples, the addition of information about SYCL Docker images in the CI, and various improvements and fixes such as correcting a wrong UR code, enhancing out-of-memory troubleshooting guidance, and addressing formatting and feedback comments.
- URL: pull/13544
- Merged: 2025-05-19T13:38:20Z
Other Closed Pull Requests
- Memory and Performance Optimization: Several pull requests focus on optimizing memory usage and performance across different components. These include addressing a memory leak in the tensor override parser, optimizing alignment and buffer management, and enhancing inference speed. Additionally, improvements in device-to-device memory copying and performance optimizations in the SYCL backend are highlighted.
- Functionality Enhancements: Enhancements to the project's functionality are addressed in multiple pull requests. These include the introduction of the
ggml_gelu_erf()function for better approximation of the GELU activation function and the addition of aload_progress_callbackto facilitate load cancellation during model loading.
- Synchronization and Bug Fixes: Pull requests have been made to address synchronization issues and bug fixes. These include fixing a missing backtrace on Linux, resolving a segmentation fault in the mnist module, and correcting the method of setting fields in the token batch to prevent segmentation faults.
- Configuration and Compatibility Updates: Updates to configurations and compatibility are covered in several pull requests. These include disabling SWA for Phi models due to configuration inconsistencies and synchronizing the project with the latest open-source updates.
- Server and API Improvements: Improvements to the server and API functionalities are addressed in various pull requests. These include adding new endpoints to the llama-server, improving model metadata response for compatibility, and introducing support for audio input in the server.
- CUDA and GPU Enhancements: Enhancements related to CUDA and GPU performance are highlighted in multiple pull requests. These include improvements to the CUDA FlashAttention vector kernels and the addition of a CUDA kernel for the
ggml_gelu_erf()function.
- Build and Compilation Improvements: Several pull requests focus on improving build and compilation processes. These include building the CPU backend separately on Windows for better performance and optimizing the continuous integration processes.
- Interface and API Simplification: Simplification of interfaces and APIs is addressed in pull requests. These include simplifying the KV cache interface and switching the retrieval process to use
llama_encode.
- Platform and Architecture Support: Support for various platforms and architectures is enhanced in several pull requests. These include adding the
torchpackage for the s390x architecture and enabling support formuBLASandMMAon theMUSA (QY2)platform.
- Miscellaneous Improvements: Various other improvements are made across different components. These include addressing a bug in the OpenAI SDK and setting the OpenMP thread block time to enhance performance.
3.3 Pull Request Discussion Insights
This section will analyze the tone and sentiment of discussions within this project's open and closed pull requests that occurred within the past week. It aims to identify potentially heated exchanges and to maintain a constructive project environment.
Based on our analysis, there are no instances of toxic discussions in the project's open or closed pull requests from the past week.
IV. Contributors
4.1 Contributors
Active Contributors:
We consider an active contributor in this project to be any contributor who has made at least 1 commit, opened at least 1 issue, created at least 1 pull request, or made more than 2 comments in the last month.
If there are more than 10 active contributors, the list is truncated to the top 10 based on contribution metrics for better clarity.
| Contributor | Commits | Pull Requests | Issues | Comments |
|---|---|---|---|---|
| ngxson | 311 | 13 | 4 | 71 |
| ggerganov | 82 | 14 | 2 | 52 |
| slaren | 68 | 6 | 1 | 50 |
| CISC | 63 | 7 | 2 | 43 |
| JohannesGaessler | 48 | 2 | 3 | 42 |
| gabe-l-hart | 48 | 3 | 1 | 1 |
| matteoserva | 33 | 2 | 4 | 5 |
| jeffbolznv | 30 | 2 | 0 | 10 |
| qnixsynapse | 24 | 5 | 0 | 12 |
| No author found | 31 | 0 | 0 | 0 |