inference

Ollama

OllamaAIOpen

Run open language models locally with one command

Ollama reduced local model hosting from an afternoon of CUDA troubleshooting to a single pull command. It handles model download, quantisation selection, GPU offloading, and memory management, and exposes an OpenAI-compatible endpoint on localhost.

llama.cpp

llama.cppAIOpen

Inference engine that runs LLMs on almost any hardware

llama.cpp is the layer most local model tooling is built on. It is a C and C++ inference engine with no Python runtime, aggressive quantisation support, and hand-tuned kernels for a remarkably wide range of hardware.

whisper.cpp

whisper.cppAIOpen

Speech recognition that runs anywhere, with no runtime

whisper.cpp is a C and C++ port of the Whisper speech recognition model with no Python dependency, quantised weights, and hardware acceleration across Apple Silicon, CUDA, and plain CPU.

LM Studio

LM StudioAI

Desktop application for running language models locally

LM Studio made local models approachable for people who do not want a terminal. Browse a catalogue, download a quantisation that fits the machine, and chat, with hardware requirements explained in plain language.