It’s been over a year since I last looked at the state of humanoid robots, and enthusiasm for the technology shows no sign of waning. Startups NEURA Robotics and Figure AI raised $1.4 billion and $1 billion, respectively, and Apptronik raised $520 million. China’s Unitree went public, raising roughly $900 million, and Agility Robotics plans to go public via SPAC later this year. A few weeks ago China held the second annual World Humanoid Robot Games, showing humanoid robots performing all manner of impressive physical feats.

We’ve also seen some impressive-looking robotics demos over the last year, though these must be evaluated with a large grain of salt. Figure showed off its robots sorting packages for hours at a time and autonomously unloading a dishwasher. Physical Intelligence showed off its robot model making coffee and folding boxes in a chocolate factory. Generalist AI showed off its GEN-1.5 model learning new tasks from just a few examples.

Some progress in robots is a function of hardware advances: better actuators, and so on. But the lion’s share of it is due to advances in robot AI, using specially developed AI models to control a robot. As more general AI continues to rapidly advance in capabilities, one obvious question is whether we’re going to see the same sort of acceleration in robotic AI. Right now robotic AI capabilities still seem very limited, but given how fast more general AI improved, I can imagine this changing very quickly. Because of this, it’s worth understanding how the AI models being developed to control robots work.

There are a few different sorts of robotic AIs, which are sometimes referred to as “policies” (where a “policy” is something that maps a particular set of inputs — robot state, sensor data, instructions it’s been given — to a set of robot actions). For this essay, we’ll look at one commonly used architecture, employed by companies such as Figure, Unitree, Physical Intelligence, and Nvidia: the vision-language-action model (VLA). More specifically, we’ll look at one popular VLA, Physical Intelligence’s open-weight π0.5 VLA, which was released in 2025 and has become widely used (though it’s not Physical Intelligence’s most advanced model).

While a large language model takes text as an input and gives text as an output, a VLA takes text, images, and robot information as an input and gives a series of robot actions as an output. It does this using many of the same components that made LLMs so successful, specifically attention and the transformer architecture. It’s not amazingly obvious if VLA’s will remain the primary robotic AI paradigm, but as of now they’re a very common method for controlling robots.

Linear algebra basics

To learn how AI works (be it a VLA or anything else), it’s useful to know a very small amount of linear algebra. Specifically, we want to understand a few different operations for manipulating arrays of numbers, since this is most of what AI models do.

For instance, say we have the following two-dimensional arrays (or matrices) of numbers:

One thing we can do to these arrays is add them together. This works exactly like how you’d expect: each number in one array is added to the corresponding number in the other array, giving you a new array with all the resulting additions.

Another thing we can do is multiply or divide an array by a single value. This also works more or less like you’d expect: each value in the array gets multiplied or divided by the respective value.

But what if we want to multiply two matrices together? To do this, we need an operation called the dot product. The dot product takes two one-dimensional arrays of numbers (also known as vectors), multiplies each value in one vector by the corresponding value in the other vector, then sums the result.

You can think of the dot product as measuring two things: how large two vectors are, and how similar they are to each other. (If you normalize the vectors, scaling the values so everything is between 0 and 1, then the dot product is entirely a measure of how similar two vectors are to each other.)

When we multiply two matrices together, we’re simply doing a bunch of dot products: each value in the resulting matrix is the dot product of the row of one array and the column of another. Because the dot product requires two lists of equal length, multiplication of matrices must be done in a certain way: the number of columns of one matrix must be equal to the number of rows of the other matrix. And the size of the output matrix will be a function of the sizes of the input matrices.

I find it easiest to understand matrix multiplication by putting one matrix on the left and the other matrix on the upper right. The output matrix fits in the space between them, each value the dot product of the first matrix’s rows and the second’s columns:

Neural network basics

Most modern AI models, as we know, are built using neural networks. The classic image of a neural network is something like this:

You have some series of input neurons that represent your input data, which are connected to various intermediate layers of neurons, which then get connected to output neurons. Depending on what data is fed into them and how the connections between different neurons have been set, neurons will “activate” to various degrees, taking some input from the neurons in the previous layer and sending some output along to the neurons in the next layer. This structure is sometimes called a “multilayer perceptron,” or MLP.

However, I think it’s much easier to understand neural networks by looking at how they’re actually implemented, which is done by multiplying arrays of numbers (in machine learning, these arrays are sometimes called “tensors,” where a tensor is just an array of numbers that can be any dimension. Google, for instance, has a machine-learning library called “TensorFlow”). The neural network above, for instance, would actually be implemented via something like this:

The three input neurons become a one-dimensional array, or vector, with three numbers in it, which is our input data. This vector then gets passed to the hidden layer, where it goes through several steps. First, it gets multiplied by a two-dimensional matrix (W1 in the figure). This matrix has four rows (corresponding to the four neurons in the original diagram) and three columns (one for each value in the input vector): it takes a vector three numbers long (or 3-vector) as an input and spits out one four numbers long as an output. The resulting vector of four numbers is then added to another 4-vector (b1 in the figure), the values of which are known as “biases.” The weights and biases of the various layers are known as the “parameters” of the network and are what get modified when the neural network is being trained.

Once the biases have been added, the resulting vector (known as a “pre-activation,” z in the figure) then goes through what’s called a nonlinearity, which is just a function that does different transformations to an input depending on what that input is. A common nonlinearity, which the example above uses, is “ReLU,” for rectified linear unit. All ReLU does is replace any negative numbers in the vector it’s given with zeroes. Other nonlinearities are the sigmoid (an S-shaped function that squeezes values to be between 0 and 1) and GELU (a somewhat ReLU-like function that’s smooth instead of sharply kinked).

In a basic MLP like this one, each hidden layer in a neural network will consist of this matrix multiplication, then bias, then nonlinearity. The network above has just one hidden layer, so after the nonlinearity the resulting vector is then passed to the output neurons. These transform the data again using another matrix multiplication (W2) and bias addition (b2), yielding a vector of two numbers as the output (y).

……（原文過長，此處截斷）