ExecuTorch

Most AI programs used today don’t actually compute on the user’s own device. The input is sent over the internet to a provider’s large computers, processed there, and the answer is sent back. ExecuTorch takes the opposite approach: the computation is meant to happen on the device itself, i.e. on the phone, the smartwatch, or the smart glasses. To do this, ExecuTorch takes a fully trained AI program and repackages it into a particularly lean form. The tool was developed by Meta, the company behind Facebook and Instagram, and it is freely available. It belongs to the family of tools built around PyTorch, a widely used piece of software with which researchers build and train AI programs.

Why AI is moving onto the phone

Operating a data center costs money, and it does so for every single request. If the user’s phone computes instead, the provider incurs no cost for this. With millions of users, this difference quickly adds up to substantial amounts. For companies, this is a strong economic argument.

Then there’s data privacy. Whatever the device computes itself, it doesn’t need to send anywhere. A voice message that is converted to text on the phone never leaves the phone. This is especially relevant for health data, photos, or private messages. Strict European data protection rules can also be more easily complied with this way.

The third reason is speed. A round trip to the server and back easily takes half a second, depending on the connection. For a pair of translation glasses or a camera that recognizes objects in real time, that’s too long. And without a network, a cloud application doesn’t work at all, whereas a local one still does.

From research model to phone file

At its core, an AI model consists of a very large number of numbers and a set of computational instructions for what to do with them. During development, both exist in a flexible form so that researchers can make changes at any time. However, this flexibility costs memory and computing time. That’s why ExecuTorch first freezes the model: it records which computational steps are needed and in what order. The result is a rigid execution plan, called a graph.

This plan is then slimmed down. Unnecessary steps are removed, and neighboring steps are merged. The numbers in the model are also often stored more coarsely, for example using eight instead of thirty-two bits in memory. This process is called quantization. It often makes the model four times smaller, at the cost of only slightly worse results.

The end result is a single file with the extension .pte. On the device, a tiny playback program reads this file and executes the plan. This playback program is deliberately only a few hundred kilobytes in size, so that it also fits on weaker chips. Through so-called backends, it can address special computing units, such as the graphics chip or the AI unit of a smartphone. A comparison with music helps here: PyTorch is the recording studio, the .pte file is the finished recording, and the playback program is the simple player.

Smart glasses, apps, and competing products

Meta uses ExecuTorch in its own products, including the Ray-Ban smart glasses and applications on Instagram and WhatsApp. Other companies use it too, for instance to run small language models directly within an app. Chip manufacturers such as Qualcomm, Arm, and Apple provide matching integrations so that their processors are well supported.

In the news, the name usually comes up in connection with the buzzword on-device AI, meaning AI running directly on the device. Competing products include Google's TensorFlow Lite, now called LiteRT, as well as Apple’s Core ML. A common misconception is that ExecuTorch can also be used for training. The tool is almost exclusively intended for using finished models, not for their learning phase. And a phone remains limited: the largest language models with hundreds of billions of stored values simply don’t fit on it.