llama.cpp
LLM inference in C/C++
Description
The main goal of llama.cpp is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud. * Plain C/C++ implementation without any dependencies
* Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate
and Metal frameworks
* AVX, AVX2, AVX512 and AMX support for x86 architectures
* 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization
for faster inference and reduced memory use
* Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs
via HIP and Moore Threads MTT GPUs via MUSA)
* Vulkan and SYCL backend support
* CPU+GPU hybrid inference to partially accelerate models larger than the
total VRAM capacity
The compute functionality is provided by ggml. By default, ggml's CPU backend
is installed, but there are many other backends for CPUs and GPUs.Upload more screenshots
Please help extend the collection of screenshots. Just make a screenshot and upload it here. You don't need to register or anything.
Upload a screenshotHint: upload an image here from your clipboard with Ctrl-V
Homepage
https://github.com/ggml-org/llama.cpp/
Install this software package
If the package is available for the distribution you are currently using on your computer then install the software by clicking on…
Install llama.cpp