The Modern AI Tech Stack, Layer by Layer

The stack at a glance

When people say "AI tech stack", they mean the full set of technologies needed to take a raw model and turn it into a reliable product. It helps to think of it as layers, where each layer depends on the ones below it:

Layer 1: Compute

AI models are, at heart, enormous amounts of matrix multiplication. GPUs (graphics processing units) do thousands of these multiplications in parallel, which is why they power almost all AI. NVIDIA dominates this market, largely thanks to its CUDA software ecosystem; alternatives include AMD GPUs, Google's TPUs and custom chips from cloud providers.

Two very different workloads use compute:

• Training — creating or fine-tuning a model. Training a frontier model takes thousands of GPUs for weeks or months; fine-tuning a small model can take a single GPU for hours.
• Inference — running a trained model to answer requests. This is the ongoing cost of any AI product, and it's driven by model size and the number of tokens processed.

When you care: only if you self-host models or train/fine-tune them. If you use a hosted API, the provider handles this layer and you pay per token.

Layer 2: Models

The model is the reasoning engine. (New to how they work? Read How Large Language Models Actually Work.) The first big decision is between two families:

Proprietary models via API

Models like Anthropic's Claude, OpenAI's GPT and Google's Gemini are accessed over the internet. You send text, you get text back, you pay per token.

• Pros: typically the most capable, zero infrastructure, instant start, continuously improved.
• Cons: data leaves your servers (check each provider's data policies), per-token costs at scale, less control over model changes.

Open-weight models

Models like Meta's Llama, Mistral, Qwen and DeepSeek publish their weights, so you can download and run them yourself. The Hugging Face Hub is the main place to find them.

• Pros: full control and privacy, can be fine-tuned deeply, no per-token fees (you pay for compute instead), can run offline.
• Cons: you operate the infrastructure; the largest open models need serious hardware.

Beyond text

The model layer also includes specialized models: embedding models (turn text into vectors for search — see Layer 3), speech-to-text and text-to-speech, image generation, and multimodal models that understand images, documents and audio alongside text.

Customizing models

There are three ways to make a model fit your use case, in order of effort:

1. Prompting — instructions and examples in the input. Fastest, cheapest; always start here.
2. Retrieval (RAG) — inject relevant knowledge at query time. Best for facts that change or are private.
3. Fine-tuning — further train the model on your examples. Best for consistent style, format or specialized behavior, not for teaching facts.

Layer 3: Data & retrieval

A model only knows what was in its training data — which is frozen at a cutoff date and doesn't include your company's documents. The data layer fixes that by finding the right information and placing it into the model's context window. This pattern is called retrieval-augmented generation (RAG).

Core components:

• Ingestion & parsing — extracting clean text from PDFs, web pages, docs and databases (tools like Unstructured, Docling, or simple custom loaders).
• Chunking — splitting documents into passages small enough to retrieve precisely.
• Embedding models — converting each chunk into a vector that captures its meaning (see Embeddings Explained).
• Vector databases — storing those vectors and finding the closest matches to a question quickly. Options range from pgvector (an extension for PostgreSQL — great if you already use Postgres) to dedicated databases like Pinecone, Qdrant, Weaviate, Milvus and Chroma (see Choosing a Vector Database).
• Search enhancements — hybrid search (combining vector and keyword search) and rerankers that re-score the top results for higher precision.

We build a complete RAG pipeline in code in Build a RAG App From Scratch.

Layer 4: Orchestration

Orchestration is the "glue code" that turns a single model call into a working system. It covers:

• Prompt management — templates, system instructions and versioning.
• Chaining & workflows — multi-step pipelines such as retrieve → draft → check → format.
• Tool use (function calling) — letting the model request actions such as searching the web, querying a database or calling an API, with your code executing them.
• Agents — loops where the model decides which tools to call, observes the results, and continues until the task is done.
• Memory — storing and recalling information across conversations.

Tools in this layer include the official SDKs from model providers (often all you need), frameworks such as LangChain, LangGraph and LlamaIndex, and agent frameworks from the major providers. The Model Context Protocol (MCP) is an open standard for connecting AI applications to external tools and data sources, so one integration can work across many AI apps.

Layer 5: Serving & inference

If you use a hosted API, serving is handled for you. If you run open-weight models, you need an inference server that loads the model onto GPUs and handles many requests efficiently:

• vLLM and Hugging Face TGI — high-throughput production servers.
• Ollama and llama.cpp — simple ways to run models locally on a laptop or desktop.
• Managed platforms — cloud services (AWS Bedrock, Azure AI, Google Vertex AI) and specialized hosts that serve open models for you.

Key techniques here include quantization (storing weights at lower precision so models fit on smaller hardware), batching (processing many requests together), streaming responses token-by-token, and caching repeated prompt prefixes to cut cost and latency.

Layer 6: Evaluation, observability & safety

This is the layer beginners skip and experts obsess over. AI output is probabilistic: the same input can produce different outputs, and a prompt tweak that fixes one case can break five others. You need:

• Evaluations ("evals") — a test set of realistic inputs with criteria for good answers, scored automatically (by code or by another model acting as a judge) every time you change something. Tools: promptfoo, Ragas, DeepEval, or a simple script.
• Observability — logging every prompt, response, tool call, latency and cost so you can debug real failures. Tools: Langfuse, LangSmith, Arize Phoenix, Helicone, or OpenTelemetry-based tracing.
• Guardrails & security — input/output filtering, protection against prompt injection (malicious instructions hidden in content the model reads), PII handling, and limiting what tools an agent can use.

Layer 7: Applications & interfaces

Finally, the part users see. Common patterns:

• Chat interfaces — conversational assistants and support bots.
• Copilots — AI embedded inside an existing tool (an editor, CRM or spreadsheet).
• Background automation — AI processing documents, emails or tickets with no chat at all.
• Agents — systems that carry out multi-step tasks with some autonomy.

Popular building blocks: Streamlit and Gradio for quick Python demos; Next.js/React with streaming UI libraries for production web apps; and FastAPI for Python backends.

Three example stacks

How to choose

1. Start with the problem, not the tools. Write down the task, who uses it, and what a "good" answer looks like.
2. Prototype with the best hosted model. Prove the task is possible before optimizing cost.
3. Build an eval set early. Even 20–50 real examples will let you compare models and prompts objectively.
4. Add retrieval only when the model needs knowledge it doesn't have.
5. Add complexity only when measurements justify it — agents, fine-tuning and self-hosting each bring real costs.
6. Optimize last: smaller models, caching and self-hosting once quality is locked in.

Key takeaways

• The AI stack has seven layers: compute, models, data, orchestration, serving, eval/ops and applications.
• Most teams rent compute and models via APIs and differentiate on data, orchestration, evals and UX.
• Prompting → RAG → fine-tuning is the usual order of customization.
• Evaluation and observability are what make AI products reliable.
• Start simple, measure, and add layers only when they earn their place.