Multimodal AI: Images, Audio and Beyond

What "multimodal" means

A modality is a type of data: text, images, audio, video. A multimodal model can work with more than one. Today's leading AI assistants can take a photo, a screenshot, a PDF or a voice recording as input — and many can produce speech or images as output.

This matters because most of the world's information isn't plain text. It's in diagrams, scanned forms, charts, product photos, meetings and videos. Multimodal AI lets models work with information the way people actually encounter it.

How AI "sees" an image

Recall from How Large Language Models Actually Work that a language model processes a sequence of tokens, each represented as a vector. The key idea behind vision-language models is surprisingly simple: turn the image into tokens too.

Cut the image into patches. The image is divided into a grid of small squares — for example 14×14 or 16×16 pixels each.
Encode the patches. A vision encoder — usually a Vision Transformer (ViT) — converts each patch into a vector, using attention so each patch "knows" about the rest of the image.
Project into the language model's space. A small adapter maps these image vectors into the same embedding space the language model uses for words.
Process everything together. The language model now sees a sequence like [image tokens…] "What's wrong with this circuit?" and attends across both, generating a text answer as usual.

Key idea: once images become vectors in the same space as words, the same transformer machinery can reason about both together.

A practical consequence: images consume context and cost tokens, often hundreds or more per image depending on resolution. Very large images are typically resized, which is why tiny text in a huge screenshot can be missed.

Connecting pictures and words: CLIP

A landmark step was CLIP (OpenAI, 2021), trained on hundreds of millions of image–caption pairs from the web. It learns two encoders — one for images, one for text — so that an image and a matching description land close together in a shared embedding space.

That shared space enables:

Searching images with words — "red bicycle leaning on a wall" finds matching photos without any manual tags.
Zero-shot classification — compare an image with the captions "a photo of a cat" and "a photo of a dog" and pick the closer one, without training a classifier.
Guiding image generators — text–image alignment like this underpins many text-to-image systems.

How AI creates images: diffusion

Most modern image generators are diffusion models. The intuition:

Training: take real images and gradually add random noise until nothing but static remains. Train a neural network to predict and remove that noise one step at a time.
Generating: start from pure random noise and repeatedly apply the denoiser. Step by step, structure emerges — guided by your text prompt, which is encoded and fed into the network at every step.

Many systems run this process in a compressed "latent" space rather than on raw pixels, which makes it much faster (this is the idea behind latent diffusion, used by Stable Diffusion). The same family of techniques extends to video, with additional machinery for keeping frames consistent over time. Other approaches, including generating images as sequences of tokens, are also used.

Audio: listening and speaking

Speech recognition

Automatic speech recognition (ASR) turns audio into text. Audio is usually converted into a spectrogram — an image-like picture of which frequencies are present over time — which a transformer encodes. OpenAI's open-source Whisper model, trained on a very large amount of multilingual audio, made accurate transcription widely accessible.

Speech synthesis

Text-to-speech (TTS) models now produce natural, expressive voices, and some can imitate a specific voice from a short sample — powerful for accessibility and dubbing, and a serious risk for scams (see Responsible AI).

Voice assistants

Voice AI used to be a pipeline: speech-to-text → language model → text-to-speech. Newer speech-native models process audio directly, which lowers latency and lets them pick up on tone and respond more naturally, including handling interruptions.

What it's useful for

Document understanding — extracting data from invoices, receipts, forms and scanned PDFs, including tables and handwriting.
Charts and screenshots — explaining a dashboard, debugging from a screenshot of an error, turning a whiteboard photo into notes.
Accessibility — describing images for blind and low-vision users; live captions for deaf and hard-of-hearing users.
Meetings and media — transcription, summaries, searchable video archives.
Visual inspection — spotting defects, reading meters, checking shelves (with careful validation).
Computer use — agents that look at a screen and operate software by clicking and typing.
Creative work — concept art, mock-ups, illustrations, video and audio production.

Current limitations

Multimodal models are impressive but less reliable than they appear. Common weaknesses:

Counting and precise spatial reasoning — "how many people are in this photo?" or "is the cup left of the plate?" can go wrong.
Small or dense details — fine print, small labels and crowded diagrams may be misread, especially after resizing.
Visual hallucinations — describing objects or text that aren't actually there.
Exact measurements — reading precise values off charts or estimating sizes and distances.
Image generation quirks — difficulty with exact text, specific counts, or consistent characters across images (improving quickly).

Tips for better results

Use clear, high-resolution images, cropped to the relevant area.
Ask specific questions — "list every line item and its amount" beats "what's in this?".
Ask it to transcribe first, then reason: "first write out all the text in the image, then answer".
Split big documents into pages or sections.
Verify extracted numbers — and for production data extraction, validate with code (totals add up, dates are valid) and measure accuracy on real samples.

Key takeaways

Multimodal models handle more than one type of data: text, images, audio, video.
Images are cut into patches, encoded into vectors and processed alongside word tokens.
CLIP-style models put images and text in a shared embedding space.
Diffusion models create images by progressively removing noise, guided by text.
Speech models transcribe, speak and increasingly converse directly in audio.
Watch for weak spots: counting, small details, visual hallucinations — and verify.