Multimodal model

For a long time, most well-known AI programs only understood one kind of input: text. You type a question, and a written answer comes back. A multimodal model can handle more than one kind. Experts call each such kind a modality – that is, text, image, sound, video, or even sensor data. A multimodal model accepts several of these kinds and links them together. For example, you can send it a photo of your math homework and ask where the mistake is.

Why a model needs more than text

Most of the world doesn’t exist as text. An X-ray image, a traffic sign, a tone of voice – all of this is hard to put into words without losing information. A model that only reads text is cut off from these things. Multimodality therefore opens up not just a new way of interacting, but a much broader range of tasks.

There’s also a practical point: humans communicate in a mixed way anyway. You point at something while talking about it at the same time. If you want to ask an AI why the washing machine is blinking, you simply photograph the display. This way of interacting, without the detour of a precise description, is one of the main reasons for the success of such systems.

One side effect concerns learning itself. Pure text data from the internet has become scarce, whereas images and videos exist in abundance. They are an additional source from which a model can learn something about the world. Whether this knowledge is really deeper is still being debated in research.

How image and text end up in the same space

A language model doesn’t internally compute with letters, but with long lists of numbers. Each word piece is translated into such a list of numbers, which roughly captures its meaning. For images, essentially the same thing is done. A separate component, the image encoder, breaks the image down into small patches and converts each one into a list of numbers.

The crucial trick is the shared representation. During training, the system learns to place the list of numbers for the word “cat” and the one for a photo of a cat close together. You can imagine this like a huge map of meanings. Text and image are entered on the same map, not on two separate ones.

This is trained using millions of pairs: image plus corresponding description, video plus subtitles, audio recording plus transcript. For sound and video, the process works analogously, just with different encoders. It’s important to draw a distinction: a system that first runs a separate speech recognition program and then passes on the text is not truly multimodal. It’s a chain of separate tools. A genuine multimodal model also hears that someone sounds ironic.

From phone photos to robotic arms

In everyday life, people usually encounter such models through the camera. Current assistants on smartphones explain what’s visible in an image, translate a menu, or read a handwritten note. The voice modes in which you can talk to an AI and it answers with a voice also belong here. For people with visual impairments, they are now partly replacing the help of others.

In business, the use is often less spectacular but worthwhile. Insurance companies have photos of car damage automatically assessed. In factories, cameras inspect components and output their assessment as text. Clinics are testing systems that read scans together with the doctor’s report.

Multimodal models often come up in news reports whenever a provider announces a new version. A typical misconception here is to consider any image output as multimodality. A program that only generates images doesn’t understand modalities – it produces one. It becomes interesting when a system can understand inputs of different kinds at the same time and act on them, for example with robots that combine a camera view and a spoken command.