Open-source local LLM runners

vLLM

vllm-project/vllm

High-throughput LLM inference and serving engine with an OpenAI-compatible server; the default choice for GPU serving.

uv pip install vllm

LocalAI

mudler/LocalAI

Self-hosted, OpenAI-compatible API server for LLMs, vision, voice and image models; runs on CPU, no GPU required.

Says it is an open-source alternative to OpenAI API (from its README)

docker run -ti --name local-ai -p 8080:8080 localai/localai:latest

exo

exo-explore/exo

Run large models across a cluster of your own devices (Macs, PCs) by pooling their memory and compute.

brew install --cask exo

textgen (text-generation-webui)

oobabooga/textgen

Popular desktop/web app for running local LLMs with many backends, tool calling and an OpenAI/Anthropic-compatible API.

git clone https://github.com/oobabooga/textgen

SGLang

sgl-project/sglang

Fast serving framework for LLMs and multimodal models, used in production for high-throughput GPU inference.

uv pip install --prerelease=allow sglang

llamafile

mozilla-ai/llamafile

Package an LLM and its runtime into one executable file that runs on most OSes and CPUs with no install.

Install: Download a .llamafile from releases/Hugging Face, chmod +x and run.

KoboldCpp

LostRuins/koboldcpp

Single-file, zero-install app to run GGUF models locally with a web UI and OpenAI/Ollama-compatible APIs; also image gen and TTS.

Install: Download the single executable from GitHub Releases.

LMDeploy

InternLM/lmdeploy

Toolkit for compressing (quantizing), deploying and serving LLMs and VLMs efficiently.

pip install lmdeploy

MLX LM

ml-explore/mlx-lm

Apple’s MLX-based package to run, quantize and fine-tune LLMs on Apple silicon Macs.

pip install mlx-lm

RamaLama

containers/ramalama

Run and serve AI models from any registry (Hugging Face, Ollama, OCI) inside containers with one CLI.

curl -fsSL https://ramalama.ai/install.sh | bash