LocalAI setup guide for a VPS

LocalAI is a drop-in OpenAI API replacement that runs local models on CPU or GPU. It speaks /v1/chat/completions, /v1/embeddings, /v1/audio/transcriptions, and /v1/images/generations — so any tool built for OpenAI works against your server without code changes.

Disclosure: contains an affiliate link to Inferno VPS.

When to pick LocalAI over Ollama

Both are great. The honest comparison:

Pick LocalAI if you need audio transcription (Whisper), image generation, or you want a single API surface across multiple model types. Pick Ollama if you just need chat / completion / embeddings and want the simpler tool.

Prerequisites

Tested on Ubuntu 22.04, Docker CE 27.3, LocalAI 2.21, on a 16 GB Inferno VPS CPU-only (AMD EPYC).

Step 1 — Compose file

mkdir -p ~/localai && cd ~/localai
nano docker-compose.yml
name: localai

services:
  localai:
    image: localai/localai:v2.21-aio-cpu
    restart: unless-stopped
    ports:
      - "127.0.0.1:8080:8080"
    volumes:
      - localai_models:/build/models
    environment:
      DEBUG: "false"
      THREADS: "4"
      CONTEXT_SIZE: "4096"
      MODELS_PATH: "/build/models"
    deploy:
      resources:
        limits:
          memory: 14G
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

volumes:
  localai_models:

The aio-cpu tag bundles a curated set of models that work well on CPU — chat, embeddings, Whisper, a small image generator. For GPU swap to aio-gpu-nvidia-cuda-12 and add runtime: nvidia + deploy.resources.reservations.devices.

Step 2 — Bring it up

docker compose up -d
docker compose logs -f localai

The first start downloads several gigabytes of models. Patience — this is a one-time cost. Once you see "Starting LocalAI using 4 threads, with models path: /build/models," it's ready.

Step 3 — Test the OpenAI-compatible API

# List models
curl http://127.0.0.1:8080/v1/models

# Chat completion
curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4",
    "messages": [{"role": "user", "content": "What is the OSI model?"}],
    "stream": false
  }'

Yes — you can pass "model": "gpt-4" and LocalAI maps it to its configured backend. That's the whole point: tools sending gpt-4 in the model field "just work."

Step 4 — Embeddings and audio

# Embeddings
curl http://127.0.0.1:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model": "text-embedding-ada-002", "input": "Hello world"}'

# Whisper transcription
curl http://127.0.0.1:8080/v1/audio/transcriptions \
  -H "Content-Type: multipart/form-data" \
  -F file="@/path/to/audio.mp3" \
  -F model="whisper-1"

Step 5 — Nginx + HTTPS + token auth

Same pattern as our Ollama HTTPS setup:

sudo nano /etc/nginx/sites-available/localai
map $http_authorization $auth_ok {
    default 0;
    "Bearer YOUR_LONG_RANDOM_TOKEN" 1;
}

server {
    listen 80;
    server_name localai.example.com;

    location /.well-known/acme-challenge/ { root /var/www/html; }

    location / {
        if ($auth_ok = 0) { return 401 "unauthorized\n"; }

        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off;
        proxy_read_timeout 600s;
        client_max_body_size 100M;
    }
}
sudo ln -s /etc/nginx/sites-available/localai /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx
sudo certbot --nginx -d localai.example.com --redirect

Step 6 — Use it from any OpenAI SDK

# Python
from openai import OpenAI
client = OpenAI(
    base_url="https://localai.example.com/v1",
    api_key="YOUR_TOKEN",
)
resp = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)
# JS
import OpenAI from "openai";
const openai = new OpenAI({
    baseURL: "https://localai.example.com/v1",
    apiKey: "YOUR_TOKEN",
});
const r = await openai.chat.completions.create({
    model: "gpt-4",
    messages: [{ role: "user", content: "Hello" }],
});
console.log(r.choices[0].message.content);

Works with LangChain, LlamaIndex, Continue.dev, Aider, and most other LLM tooling.

Loading custom models

Drop GGUF files into the models volume:

docker compose cp ./my-model.gguf localai:/build/models/

Then create a YAML config under /build/models/my-model.yaml describing the model. LocalAI's docs cover the YAML schema in detail. For most users the AIO bundle is enough.

Resource and concurrency tuning

Common LocalAI mistakes

"OOM on first prompt"

The configured model + context size doesn't fit in RAM. Drop to a smaller model (or smaller context), or add swap.

"Model not found"

You requested a model name LocalAI doesn't have configured. Check /v1/models for the actual list. The AIO bundle aliases common OpenAI names; non-AIO setups need explicit aliases.

"Slow but everything else looks healthy"

Wrong THREADS value (often left at 1 by default). Set explicitly.

"Image generation hangs"

The CPU image-gen backend is genuinely slow — minutes per image. Use a GPU image for production image generation, or ComfyUI directly.

VPS for a LocalAI stack?

The aio-cpu image needs 16 GB to be comfortable; 8 GB works for chat-only with smaller models. Inferno's 16 GB and 32 GB tiers fit well.

View Inferno plans →

What to do next


Author: Riku Lehtinen — Tampere. Last tested: 2026-05-22 on LocalAI v2.21 aio-cpu, Docker CE 27.3, Inferno 16 GB AMD EPYC plan.