quantization

llama.cpp

llama.cppAIOpen

Inference engine that runs LLMs on almost any hardware

llama.cpp is the layer most local model tooling is built on. It is a C and C++ inference engine with no Python runtime, aggressive quantisation support, and hand-tuned kernels for a remarkably wide range of hardware.