Multimodal AI
Multimodal AI is AI that can process and reason across more than one kind of data — text, images, audio, and video together — uncovering patterns between modalities that single-modality models cannot. The term "modality" refers to a single type of input or output: a computer vision model handles the image modality, a natural language processing model handles text, and a speech model handles audio.
A multimodal model combines these single-modality skills into a single system that relates them. A model that can see an image and describe it in text, or hear spoken audio and answer with a generated picture, is doing something a text-only model fundamentally cannot — it is mapping between representations that were learned independently.
The leap from GPT-3.5 to GPT-4 is the canonical example. GPT-3.5 was trained only on text. GPT-4 was trained with images as well, producing a marked jump in ability that came not just from more parameters but from the cross-modal grounding that images provided. Seeing the visual world alongside language gave the model a richer, more constrained understanding of what words refer to.
Multimodality is widely seen as a step toward artificial general intelligence — a single model able to process any kind of data and perform any task, from translating languages to designing drugs. A system limited to one modality can only ever approach a slice of human-like intelligence, since humans seamlessly combine sight, hearing, language, and touch.
The risks scale with the capability. Multimodal models inherit biases from each modality’s training data — visual stereotypes, textual assumptions, and audio quirks all compound — broadening the surface for bias. The same generative power that lets a multimodal model produce a caption for a photo can produce a convincing fake video or a cloned voice, making misinformation harder to detect.