Make AI run onevery machine.

Open source · MIT · v4.8.0

Make AI run on every machine.

Text, voice, vision, images, video, 3D and agents, from one open runtime. It works on the laptop you already own, and scales to a room full of GPUs when you have one. When the engine we need is too heavy, too closed, or does not exist, we write it.

Install LocalAI → Read the docs ★ Star on GitHub

48,067GitHub stars

73Backends

19Engines we wrote

1,585Models, one click

localai · chat CPU only, no GPU

The runtime

LocalAI is the engine everything else plugs into.

One binary with an OpenAI-compatible API in front of it. Point an existing client at it and the calls keep working, except now the model is on your machine. It also speaks the Anthropic, Ollama and ElevenLabs APIs, so most tools need a URL change and nothing else.

Underneath, a small core pulls each engine in as a separate backend, only when a model asks for it. That is why one install covers this much ground without becoming a 9 GB download.

OpenAI APIAnthropic APIOllama APIElevenLabs APIRealtime over WebRTC

ReasonLanguage models, tool calling, structured outputllama.cpp · vLLM · MLX ListenRealtime voice, transcription, diarizationparakeet · whisper SpeakSpeech synthesis and voice cloningmoss-tts · piper SeeVision, detection, recognition, depth, 3Drf-detr · depth-anything CreateImages, video, music and sounddiffusers · ace-step ActAgents, MCP, skills, RAG, interactive toolsagents · MCP apps

Read the documentation →

localai · model gallery 1,585 models

Why the project exists

Local AI should not need a datacenter.

01 / HARDWARE

Every feature ships a CPU path first.

Not a degraded mode that technically runs. The real one, tested in CI, on the hardware most people already have. GPUs make it faster, they are not the price of entry.

x86\_64 · ARM64 · CUDA · ROCm · SYCL · Metal · Vulkan

02 / REALTIME

You can talk to it, and it answers.

Speech in, tool calls in the middle, speech out over WebRTC, fast enough to feel like a conversation. Transcription, diarization and speech synthesis all run without a GPU.

Realtime API · WebRTC · streaming ASR · TTS · VAD

03 / DISTRIBUTED

Plug in a second machine and stop there.

Routing, VRAM-aware placement, prefix-cache affinity and failover are the runtime's problem. You add hardware, the cluster works out what to do with it.

Smart routing · autoscaling · P2P · NATS · federation

Senses

Give the model eyes and ears.

A chat model can only work with what somebody types at it. Our engines change what it has access to: what is happening in the room, who walked into it, where things are in space, and how to answer out loud. All of it runs locally, most of it on a CPU.

Hear words

Live transcription with speaker labels and timestamps, fast enough to keep up with a meeting while it is still happening.

parakeet.cppmoss-transcribe.cpp

Hear the room

527 kinds of sound event: a door, a dog, breaking glass, a smoke alarm. The model notices things nobody thought to type.

ced.cpp

Know who

Recognise a voice, recognise a face, and tell a live person from a photo held up to the camera.

voice-detect.cppface-detect.cpp

See things

Ask for "the red mug on the left" in plain language and get back coordinates, not a caption.

locate-anything.cpprf-detr.cpp

See space

Distance in metres from one ordinary photo, and a full 3D reconstruction from a handful of them. No rig, no camera poses, no GPU.

depth-anything.cppfree-splatter.cpptrellis2.cpp

Speak

Long-form speech in a cloned voice, across dozens of languages, up to 48 kHz.

moss-tts.cppmagpie-tts.cppvibevoice.cppvoxtral-tts.c

Hear clearly

Echo cancellation, noise suppression and dereverberation, so a voice loop survives a real room with a real speaker in it.

LocalVQE

Forget on purpose

Names, addresses and card numbers get caught and redacted on the machine, before anything is sent anywhere.

privacy-filter.cpp

One session can do all of it at once: hear the room, work out who is talking, read what is on the desk, call a tool, and answer out loud. One API, one machine, nothing leaves the building.

Made with LocalAI

Nobody was at the keyboard.

Sound on

Play with sound

Every part of this clip came out of LocalAI, and the agent in the binary drove the machine that made it. It opened the app, ran the demo and captured the screen, while a local model wrote the script, a cloned voice read it, and the video endpoint generated and lip-synced the presenter. No human touched the keyboard, and nothing left the building.

DirectionThe agent in LocalAI drove the machine end to end

ScriptWritten by a local language model

VoiceCloned from a few seconds of reference audio

PresenterGenerated and lip-synced through the video endpoint

How the voice pipeline works →

Engines we build

19 engines, written from scratch.

Those backends the core pulls in on demand are mostly ours. Most projects wrap somebody else's engine. We wrote ours, because the thing we needed came as a 9 GB Python install, or was closed, or did not exist. Each one is a binary and a GGUF file, checked against the reference implementation in CI.

parakeet.cpp

Twenty-seven times faster than whisper.cpp, on a CPU.

NVIDIA NeMo Parakeet, ported to C++ and ggml. Ten checkpoints, all of them verified at WER 0 against NeMo, which means the transcript comes out byte for byte identical while finishing first. Cache-aware streaming with end-of-utterance detection handles live audio, and the multilingual streaming model covers 40 or more locales.

27xvs whisper.cpp, CPU

1.40xvs NeMo, CPU median

WER 0Parity with NeMo

37%Size at q8\_0

parakeet.cpp on GitHub ↗

Same audio, same words, ours finishes first

CPU speedup over NeMo, by dtype

depth-anything.cpp

Beats PyTorch on CPU, in half the memory.

Depth Anything 3 as a 99 MB file. It gives you metric depth, per-pixel confidence, camera intrinsics and extrinsics, and a back-projected point cloud you can export to glb or COLMAP. Output correlates 1.0 with the reference implementation, component by component, and there is no Python or CUDA toolkit anywhere at inference.

1.31xvs PyTorch, CPU

363 MBPeak RAM, q8\_0

6.7xFaster to load

99 MBSmallest build

depth-anything.cpp on GitHub ↗

One photo in, distance in metres out, ahead of PyTorch on the same CPU

And the rest of them, running.

Every clip comes out of the benchmark suite that runs in CI on that engine. Where you see two panes, ours is racing the reference implementation on the same machine, on the same input.

parakeet.cpp long-form transcription vs NeMoced.cpp sound events, live 527 classesface-detect.cpp detect, landmark, recognise no Pythonface-detect.cpp same person, different photo 1 to Nvoice-detect.cpp who is speaking vs referencedepth-anything.cpp metric depth vs PyTorch, CPUlocate-anything.cpp say it, find it open vocabularymoss-tts.cpp 48 kHz voice cloning vs referencemagpie-tts.cpp 9 languages, 5 voices vs reference

vllm.cppparakeet.cppmoss-transcribe.cppmoss-tts.cppmagpie-tts.cppced.cppvoice-detect.cppvoxtral-tts.cvibevoice.cpprf-detr.cpplocate-anything.cppdepth-anything.cppface-detect.cppfree-splatter.cpptrellis2.cppprivacy-filter.cppLocalVQElocal-storeapex-quant vllm.cppparakeet.cppmoss-transcribe.cppmoss-tts.cppmagpie-tts.cppced.cppvoice-detect.cppvoxtral-tts.cvibevoice.cpprf-detr.cpplocate-anything.cppdepth-anything.cppface-detect.cppfree-splatter.cpptrellis2.cppprivacy-filter.cppLocalVQElocal-storeapex-quant

In development, not announced yet

#### vllm.cpp

vLLM ported to C++20, with paged attention, continuous batching and prefix caching, on CPU, CUDA, Metal and Vulkan. It installs as 66 MB instead of a 9.1 GB virtualenv, and it stays ahead of vLLM at every concurrency level we have measured so far. Still being finished, so treat the numbers as provisional.

All 19 engines →

APEX quantization

The model you could not fit, on the card you already own.

The engine decides how fast a model runs. The weights decide whether it runs at all, so we build those too. APEX assigns a different precision to every tensor and every layer: a 35B mixture-of-experts model goes from 64.6 GB, out of reach of any consumer GPU, to 12.2 GB at 74 tokens a second. That is more than twice the speed of the original, and quality barely moves. The file is an ordinary GGUF, so stock llama.cpp opens it with no patches, and 201 of them are already sitting in the LocalAI gallery.

F16 · 64.6 GB30.4 t/s

Q8\_0 · 34.4 GB52.5 t/s

APEX Quality · 21.3 GB62.3 t/s

APEX Mini · 12.2 GB74.4 t/s

| Build | Size | Perplexity | HellaSwag | MMLU | tg128 t/s | | -------------------- | ------- | ---------- | --------- | ----- | --------- | | F16 | 64.6 GB | 6.537 | 82.5% | 41.5% | 30.4 | | Q8\_0 | 34.4 GB | 6.533 | 83.0% | 41.2% | 52.5 | | Unsloth UD-Q8\_K\_XL | 45.3 GB | 6.536 | 82.5% | 41.3% | 36.4 | | APEX Quality | 21.3 GB | 6.527 | 83.0% | 41.2% | 62.3 | | APEX I-Quality | 21.3 GB | 6.552 | 83.5% | 41.4% | 63.1 | | APEX Compact | 16.1 GB | 6.783 | 82.5% | 40.9% | 69.8 | | APEX Mini | 12.2 GB | 7.088 | 81.0% | 41.3% | 74.4 |

Qwen3.5-35B-A3B on an NVIDIA DGX Spark (GB10). Perplexity on wikitext-2-raw at context 2048. Full methodology in the technical report.

APEX models on Hugging Face ↗ Technical report ↗

Distributed

Two machines behave like one.

01

Start a worker

One command on any box. It reports what hardware it has and which backends it can run, then joins the pool.

02

Work goes where it is cheapest

Requests land on the replica that already holds the model and the matching prefix cache, sized against real free VRAM rather than a guess.

03

Losing a node is boring

In-flight work reschedules, the model loads somewhere else, and the client never finds out.

node-0124 GB · CUDA

node-0216 GB · ROCm

node-03CPU · 64 GB

routerprefix affinity · VRAM aware · autoscaling

The gallery

1,585 models. No notebook, no conversion script.

Quantizations201 APEX buildsEvery tier of every model we quantize, ranked against the hardware you actually have and installed with one click.Browse the gallery → Speechitalian-asrItalian speech recognition trained and published by the team, streaming on a CPU through parakeet.cpp.Model card → Voices60 Piper voicesForty-two languages of text to speech, small enough to run on a Raspberry Pi, installed from the web interface.Voice catalogue →

the agent

The agent is already in the binary.

Run local-ai chat and you are talking to an agent that already knows where your models are. It runs shell commands behind an approval gate you control, delegates to sub-agents, and loads MCP servers, plugins and skills. Claude Code plugins load as they are.

It is also nib, a single \~20 MB Go binary with no runtime and no daemon, so you can drop the same agent on any box you SSH into and press Ctrl+Space.

local-ai chatMCPpluginsskillssub-agentszero dependencies

Read the docs → nib on GitHub ↗

local-ai chat the agent, in the binary

Since March 2023

Forty-eight thousand stars, and still shipping every week.

LocalAI has been trending on GitHub repeatedly since it launched, and it is one of the most starred self-hosted AI projects there is. 225 people have contributed code, 3,187 are in the Discord, and the README is kept translated into eight languages because the users are everywhere.

0Stars

0Forks

0Contributors

0Releases

0In Discord

0Shipping since

MAR 2023

#### First commit

An OpenAI-compatible API in front of llama.cpp, so a laptop could answer the same calls as the cloud.

JUL 2025

#### The core gets small

Every backend moves out of the binary. You install only the engines your models need.

MAR 2026

#### Agents and a new interface

Native agentic orchestration, a full React rewrite with canvas mode, WebRTC realtime audio.

APR 2026

#### It becomes a cluster

Distributed mode with VRAM-aware routing, autoscaling, multi-user auth and per-user quotas.

MAY 2026

#### It sees and hears

Voice recognition, face recognition with liveness, diarization, video generation, drop-in Ollama API.

JUL 2026

#### Nineteen engines of our own

The native C and C++ ports take over the heavy Python backends, one modality at a time.

Where it runs

NVIDIA CUDAAMD ROCmIntel SYCLApple MetalVulkan JetsonRaspberry Pix86\_64ARM64KubernetesDocker

DeutschEspañolfrançais日本語 한국어PortuguêsРусский中文

What other people say

“Some cool ggml-based work by @mudler\_it recently, make sure to check it out.”@ggerganovGeorgi Gerganov, author of llama.cpp and ggmlOn X, 2026 ↗ “What a wonderful project: parakeet.cpp. A ggml based parakeet inference pipeline that is 2x faster than my ONNX parakeet pipeline on Apple Silicon.”@badlogicgamesMario Zechner, author of pi.agentOn X, 2026 ↗ “Build something good enough that the community chooses to carry it beyond your reach. This week, Ettore Di Giacinto brought NVIDIA Parakeet to the CPU.”Adi MargolinOn LinkedInRead the post ↗

Who shows up

225 people have put code in this repository.

Some of them do it from a desk at Microsoft, Spectro Cloud, VictoriaMetrics, Acronis, BearingPoint, Zendesk, Nextcloud, Scaleway, Kubermatic, Avanade, IBM, Ableton, Atlassian, TikTok, CockroachDB, MediaTek, Samsung SDS, Ant Group, Deloitte, solo.io and Qonto. Others from labs at MIT, University of Edinburgh, RMIT, Rochester Institute of Technology, University of Miami and USC.

That is where they work, not who sent them. Check out the full contributor list, 200+ and growing ↗

And many projects integrate it

Click any name to see where they say so.

AnythingLLM LangChain LlamaIndex Open WebUI Dify LibreChat RAGFlow Flowise Continue big-AGI K8sGPT k8sgpt-operator Nextcloud Frigate Kairos AIKit langchain4j promptfoo Mods TypingMind baibot LLM Vision VoxInput Obsidian BMO Chatbot ShellOracle QA-Pilot AnythingLLM LangChain LlamaIndex Open WebUI Dify LibreChat RAGFlow Flowise Continue big-AGI K8sGPT k8sgpt-operator Nextcloud Frigate Kairos AIKit langchain4j promptfoo Mods TypingMind baibot LLM Vision VoxInput Obsidian BMO Chatbot ShellOracle QA-Pilot

Using LocalAI at work? Add your organisation to ADOPTERS.md.

Written about elsewhere

PulumiDeploy low-code LLM apps on AWS with Flowise and LocalAI_A full EKS deployment, with LocalAI doing inference and no GPU in the cluster. SemaphoreLocalAI: replacing OpenAI API with open-source_Swapping an existing OpenAI integration over to LocalAI, endpoint by endpoint. Spectro CloudK8sGPT + LocalAI: unlock Kubernetes superpowers for free_Cluster diagnosis through K8sGPT, answered by a model running on CPU. SUSEA four-part series on running AI locally_Install on openSUSE and drive it from VS Code, add your own documents with RAG, put a Confluence space behind a local model, then train a LoRA and serve it.

Built on, integrated with, written about

@sozercan _builds on_AIKit now offers an extensible solution for finetuning LLMs! Thanks to @UnslothAI, you can finetune fast and efficiently. Then, deploy seamlessly with AIKit using @LocalAI\_API for an end-to-end solution!On X, 2024 ↗ @ivanfioravanti _benchmarks_An M5 Max with 40 GPU cores just beat an M3 Ultra with 80 on parakeet.cpp. Every model. \~1.7x faster on average, up to \~2x. Half the cores.On X, 2026 ↗ @PulumiCorp _integration_Explore how to build and deploy a LLM app using @FlowiseAI and @LocalAI\_API with AWS EKS, Pulumi, and TypeScript! Run your models locally or on-prem.On X, 2024 ↗ @enricoros _ecosystem_Congrats to @LocalAI\_API for launching LocalAGI (Agents), and LocalRecall (Memory). The Local stack is well designed and expanding.On X, 2025 ↗ @UniverseAdam _in print_My hardcopies of the official @Raspberry\_Pi magazine @TheMagPi have arrived! And they have my Automatic Speech Recognition project based around @NordVPN's Meshnet and a self-hosted @LocalAI\_API language model inside.On X, 2024 ↗ @ivanfioravanti _community_LocalAI is becoming stronger and better release, after release! Keep pushing @mudler\_it and @LocalAI\_APIOn X, 2026 ↗ @alepiad _early days_I'm really excited about the prospect of open-source LLMs. In that respect, take a look at @LocalAI\_API, a drop-in replacement for OpenAI's API but serving GGML models right on your own infrastructure.On X, 2023 ↗ @mattapperson _builds on_Oh, high there @LocalAI\_API, nice to see a terminal based UI for ya! (it's a WIP, but just wanted something cleaner then CURL calls)On X, 2023 ↗

From the team

We publish the numbers, including the ones that cost us.

APEX takes a 35B model from 64.6 GB to 12.2 GB, and from 30.4 to 74.4 tokens a second. Perplexity goes from 6.537 to 7.088. That trade is in the post too, with the command that produced it.

1 August 2026What landed in LocalAI 4.8A new inference engine, 3D generation, one backend that serves six audio endpoints, and a web interface 3.48x lighter. 321 pull requests in eighteen days.Read the post → 29 July 2026LocalAI, from March 2023 to nowThree years, 133 releases and 224 contributors later. The four changes that mattered most were making the core small, adding agents, making it a cluster, …Read the post → 24 July 2026Why we write our own C and C++ enginesA 66 MiB binary instead of a 9.1 GiB virtualenv, depth estimation that beats PyTorch on CPU in half the memory, and biometrics that match insightface bit …Read the post →

Get started

Running in about a minute.

A container on any platform, a DMG on macOS, a binary on Linux, or a chart on Kubernetes. Backends download themselves the first time a model asks for one, so the base install stays small.

Installation guide → Join the Discord

Script Docker Kubernetes

bash copy

$ curl -sSL https://localai.io/install.sh | sh detecting platform linux/amd64 fetching latest release binary installed to /usr/local/bin/local-ai

$ local-ai run qwen3.5-35b-a3b-apex pulling backend llama-cpp API ready on http://localhost:8080 model ready · 74.4 tok/s

$ docker run -p 8080:8080 --name local-ai -ti localai/localai:latest starting local-ai detected: CPU (AVX-512), 32 GB RAM API ready on http://localhost:8080

Podman works the same way

$ kubectl apply -f https://localai.io/install/kubernetes.yaml deployment.apps/local-ai created service/local-ai created

or with Helm

macOS: DMGLinux: binaryPodmanHelm chartBuild from source