Running AI Models Locally

Why run AI locally?

Cloud AI services are the easiest way to use the most capable models. But open-weight models — models whose weights are published, such as Meta's Llama, Mistral, Google's Gemma, Qwen and DeepSeek — can run entirely on your own hardware. Reasons to do it:

• Privacy — your data never leaves your machine. Ideal for confidential documents.
• Cost — no per-token fees; once you have the hardware, experimentation is free.
• Offline use — works on a plane, in the field, or in air-gapped environments.
• Control — the model never changes unless you change it; you can fine-tune and customize freely.
• Learning — nothing teaches you how LLMs behave like running and tweaking one yourself.

The trade-off: the models you can run on a laptop are smaller and generally less capable than the largest cloud models, especially for complex reasoning. For many everyday tasks — summarizing, drafting, extraction, classification, coding help — they're very useful.

How much hardware do you need?

The main constraint is memory: the whole model must fit in RAM (or GPU memory) to run at a usable speed. A rough rule:

memory needed ≈ number of parameters × bytes per parameter (+ some overhead)

8 billion parameters × 2 bytes (16-bit)  ≈ 16 GB
8 billion parameters × 0.5 bytes (4-bit) ≈  4–5 GB

Longer conversations need extra memory on top (for the model's working "KV cache"), so leave headroom.

What makes it fast: a dedicated NVIDIA GPU is the fastest common option. Apple Silicon Macs (M-series) work very well because their unified memory is shared between CPU and GPU, so a 32 GB Mac can hold models a typical 8 GB graphics card can't. CPU-only machines work too, just more slowly.

Quantization: the trick that makes it possible

Models are trained with 16- or 32-bit numbers for each parameter. Quantization stores them with fewer bits — 8, 5, 4 or even fewer — shrinking the model dramatically with surprisingly little quality loss. A 4-bit model is about a quarter the size of the 16-bit original.

You'll see quantized files labelled with names like Q4_K_M, Q5_K_M or Q8_0 (from the GGUF format used by llama.cpp). Guidelines:

• 4-bit (e.g. Q4_K_M) — the popular sweet spot of size, speed and quality.
• 5–6 bit — slightly better quality if you have the memory.
• 8-bit — very close to the original; twice the size of 4-bit.
• Below 4-bit — quality drops more noticeably; useful only to squeeze in a bigger model.

Option 1: Ollama (easiest from the command line)

Ollama is a free tool for Mac, Windows and Linux that downloads, manages and runs models with one command. Install it from ollama.com, then:

# download a model (first time only) and start chatting
ollama run llama3.2

# other useful commands
ollama list            # models you've downloaded
ollama pull qwen2.5    # download another model
ollama rm llama3.2     # delete a model to free disk space

Browse the model library on the Ollama website to find models by size and purpose (general chat, coding, vision, embeddings). New models are added frequently.

Using it from your own code

Ollama runs a local server on port 11434 with a simple HTTP API:

import requests

response = requests.post("http://localhost:11434/api/chat", json={
    "model": "llama3.2",
    "messages": [{"role": "user", "content": "Explain embeddings in one sentence."}],
    "stream": False,
})
print(response.json()["message"]["content"])

That's all you need to fill in the call_llm placeholder from Build a RAG App From Scratch — giving you a completely private, offline document assistant:

def call_llm(prompt):
    response = requests.post("http://localhost:11434/api/chat", json={
        "model": "llama3.2",
        "messages": [{"role": "user", "content": prompt}],
        "stream": False,
    })
    response.raise_for_status()
    return response.json()["message"]["content"]

Ollama also offers an OpenAI-compatible endpoint at http://localhost:11434/v1, so many existing tools and libraries can point at it with a one-line configuration change, and it can serve local embedding models too.

Option 2: LM Studio (easiest with a graphical app)

LM Studio is a free desktop app with a chat interface. You search for models, it shows which ones fit your hardware, and you download and chat with a click — no terminal needed. It can also run a local server compatible with the OpenAI API for use from code. Other desktop apps, such as Jan, offer a similar experience.

Option 3: llama.cpp (maximum control)

llama.cpp is the open-source engine that powers much of the local-AI ecosystem. Using it directly gives you fine control over performance settings. Models come as single .gguf files, widely available on the Hugging Face Hub.

# chat with a model in the terminal
llama-cli -m ./models/my-model-Q4_K_M.gguf

# or run an OpenAI-compatible server on port 8080
llama-server -m ./models/my-model-Q4_K_M.gguf --port 8080

For serving models to many users on GPU servers, production-oriented engines like vLLM are a better fit (see the serving layer).

Choosing a model

• Start small: try a 3–8B model first to see what your machine handles comfortably, then size up.
• Use the instruction-tuned version (often labelled instruct or chat) for conversation — base models just continue text.
• Pick specialists where they exist: coding models for code, vision models for images, embedding models for search.
• Check the license. "Open-weight" doesn't always mean unrestricted; some licenses limit commercial use or require attribution.
• Test on your own tasks. Leaderboards are a rough guide; your use case is what counts.

Performance tips

• Fit the whole model in GPU/unified memory — once layers spill over to the CPU, speed drops sharply.
• Keep context lengths reasonable — very long contexts need more memory and run slower.
• Close memory-hungry apps when running larger models.
• Measure speed in tokens per second — roughly 10+ tokens/second feels comfortable for chat.

Key takeaways

• Open-weight models can run privately, offline and free on your own hardware.
• Memory is the main constraint: parameters × bytes per parameter, plus headroom.
• Quantization (often 4-bit) shrinks models about 4× with modest quality loss.
• Ollama and LM Studio get you started in minutes; llama.cpp gives full control.
• Local models are smaller than the best cloud models — choose tasks accordingly and test.