Vision Language Action Models in Robotics
Classical robotic systems are built as pipelines. A perception module estimates the state of the environment. A planner reasons over that state and produces a sequence of subgoals. A motion planner computes trajectories that satisfy those subgoals subject to kinematic constraints. A controller tracks those trajectories at the joint level. Each stage in the pipeline is engineered separately, with its own representation, its own failure modes, and its own set of assumptions about what the upstream stage will provide.
This structure has real advantages. Each component can be tested in isolation. When something fails, the failure is usually localized to one stage. The behavior is interpretable because the plan is explicit. But the engineering cost compounds: every new task requires tuning the perception pipeline to recognize new objects, updating the planner’s world model to include new task structure, and verifying that the controller can execute the new motion class. For a robot that needs to handle dozens of tasks across varied environments, that per-task engineering overhead becomes the primary bottleneck.
Vision Language Action models take a different approach. A single learned model takes the current sensor observations and a natural language instruction, and directly produces an action. The perceptual representation, the task-conditional reasoning, and the action generation are all part of the same computation, trained end to end. The pipeline is not removed; it is internalized into the model weights.
What the Model Learns
The core abstraction in a VLA model is a policy that maps observations and a goal into actions at each timestep:
$$ a_t = \pi(o_t, g) $$where $o_t$ is the observation at time $t$, $g$ is a language instruction or goal description, and $a_t$ is the action to execute. In practice, observations typically include one or more RGB images, often supplemented with the robot’s proprioceptive state (joint positions, end effector pose, gripper state). The action $a_t$ is usually a delta in end effector pose plus a gripper command, though some systems predict joint velocities or discrete action tokens depending on the control interface.
The language instruction $g$ provides task conditioning. The same visual scene produces different actions depending on what $g$ says. “Pick up the red block” and “Pick up the blue block” should produce different grasp targets even if both objects are visible. This is what separates a VLA model from a pure visuomotor policy trained on a single task: the instruction generalizes the policy across tasks without requiring a separate model per task.
Training this mapping is done primarily through behavior cloning on demonstration data. Given a dataset of (observation, instruction, action) triples collected from human demonstrations, the model is trained to minimize the prediction error on held-out demonstrations:
$$ \mathcal{L} = \mathbb{E}_{(o_t, g, a_t) \sim \mathcal{D}} \left[ \| \pi(o_t, g) - a_t \|^2 \right] $$For continuous actions, this is typically a mean squared error loss. For systems that predict action tokens through an autoregressive decoder, it is a cross-entropy loss over the token vocabulary. The distinction matters more than it first appears. Regression-based action heads can predict smooth continuous outputs but struggle with multimodal distributions; autoregressive token prediction handles multimodality naturally but introduces quantization error and decoding latency.
Architecture
Most VLA architectures share a common structure. A vision encoder processes the image observation into a sequence of patch embeddings. A language encoder processes the instruction into a sequence of token embeddings. A fusion module combines the two, and a policy head maps the fused representation to an action prediction.
The vision encoder is usually a Vision Transformer (ViT) or a convolutional backbone pretrained on large image datasets. The language encoder is typically a transformer pretrained on text corpora. What varies significantly across systems is the fusion mechanism and how much of the backbone is shared versus task-specific.
The simplest fusion strategy concatenates the vision and language embeddings and passes them through a cross-attention mechanism or a shared transformer decoder. This works when the vision and language representations live in compatible embedding spaces, which is more likely if both encoders were trained jointly on paired image-text data. Systems that build on pretrained multimodal models like CLIP or PaLI start from encoders that have already learned to align visual and linguistic representations, which simplifies the fusion problem.
The policy head can be structured as a simple MLP on top of the fused representation, predicting a fixed-dimensional action vector. Some systems instead treat action prediction as sequence generation: actions are discretized into tokens and predicted autoregressively, the same way a language model generates text. The RT-2 family from Google DeepMind takes this approach, co-training a visual language model on robotics data alongside internet-scale vision-language data and expressing robot actions as sequences of tokens appended to the language output. This co-training on heterogeneous data lets the model leverage the grounding and reasoning capabilities acquired from web-scale training, not just from robot demonstrations.
OpenVLA follows a similar philosophy but focuses on open-weight access. The base model is a 7B parameter visual language model fine-tuned on the Open X-Embodiment dataset, which aggregates demonstrations from multiple robot platforms and tasks. Action tokens are appended to the standard language vocabulary, and the model is fine-tuned to predict them conditional on the image and instruction. The result is a model that can be further fine-tuned on a specific robot configuration with relatively few demonstrations, inheriting the generalization of the pretrained backbone.
The Data Problem
The limiting factor in training capable VLA models is not architecture. It is data. A single-task visuomotor policy trained on a few hundred demonstrations often achieves high success rates on that task. Generalizing across tens or hundreds of tasks requires orders of magnitude more data, and robotic demonstration data is expensive to collect.
The Open X-Embodiment dataset, which aggregates demonstrations from multiple research labs across different robot morphologies and tasks, contains roughly 22 million robot steps. That sounds large, but in the context of pretraining data for large language models, which routinely use trillions of tokens, it is not. The distribution of tasks and environments in existing robot datasets is also heavily skewed toward table-top manipulation in laboratory settings. Generalization to novel environments, novel object geometries, or novel task structures that differ meaningfully from the training distribution is unreliable.
One response to data scarcity is to use simulation. Physics simulators can generate large volumes of demonstration data without human supervision if a task reward can be defined and an automated policy can be used to collect demonstrations. The problem is the sim-to-real gap. Visual appearance differences between rendered and real images cause the vision encoder to produce different embeddings for scenes that are semantically identical. Contact dynamics in simulation are approximate, and behaviors that work in simulation often fail on real hardware because the controller learned to exploit simulation artifacts that do not exist in the physical world. Domain randomization during simulation training, varying textures, lighting, and contact parameters, reduces but does not eliminate the gap.
Another approach is to use foundation models pretrained on internet-scale data to reduce the number of robot demonstrations required. The argument is that a model that has been trained on billions of images and text documents has already learned representations of object categories, spatial relationships, and common physical interactions. Fine-tuning such a model on robot demonstrations adds only the action prediction head and the specific interaction patterns of the deployment task. OpenVLA and RT-2 both follow this approach and demonstrate that fine-tuning on hundreds rather than thousands of demonstrations can achieve reasonable task performance when the base model is sufficiently strong.
Long-Horizon Tasks and Memory
Single-step tasks like grasping a visible object are tractable for current VLA models. Long-horizon tasks that require executing a sequence of subtasks over an extended period are substantially harder.
The difficulty is that a model predicting $a_t$ from $o_t$ and $g$ alone has no memory of what happened before time $t$. If the task is “open the drawer, take out the red block, and place it on the table,” the model at step $t$ needs to know whether the drawer is already open and whether the block has been extracted. Without a history of observations, that information must be inferred from the current image alone, which is possible if the visual state is fully observable but fails when it is not.
Systems designed for long-horizon tasks typically condition the policy on a history of observations:
$$ a_t = \pi(o_{1:t}, g) $$A transformer policy can process variable-length observation histories by treating each observation as a token in a sequence. The attention mechanism allows the model to reference relevant past observations when computing the current action. This is the approach taken by models in the Gato and Octo families, which train a single transformer on large collections of tasks and use the full episode history as context.
The practical constraint is context length and inference latency. Running a large transformer at every control step, where control frequencies may be 10 to 30 Hz, requires either a very efficient model or a hierarchical architecture where the high-level language-conditioned policy runs at lower frequency and a lower-level controller handles high-frequency tracking. Most deployed systems use some form of this hierarchy: the VLA model predicts a desired end-effector pose or a short action chunk (a sequence of future actions), and a classical impedance or velocity controller executes that target at the joint level. This decoupling also provides a safety layer: the classical controller can be designed to respect joint limits and velocity bounds regardless of what the high-level policy outputs.
Spatial Grounding and Attention
Language instructions frequently involve spatial relationships: “to the left of the cup,” “behind the box,” “on top of the stack.” For the model to act correctly on such instructions, it needs to align the words in the instruction with regions in the image.
In practice, this alignment emerges from cross-attention between the language token embeddings and the image patch embeddings. When the instruction contains the word “red,” the attention weights in the vision-language fusion layer become concentrated on image patches that contain red objects. This is not explicitly trained as an alignment objective; it emerges because correct action prediction requires it. The instruction “pick up the red block” produces wrong actions if the model attends to the blue block, so gradient descent pushes the attention toward the correct spatial region.
This emergent spatial grounding is more robust when the base model has been trained on paired image-text data with dense spatial annotations or referring expression comprehension tasks. Models that have been trained to identify specific regions of an image based on textual descriptions, as in visual question answering or image captioning with grounding, transfer that capability to action-conditioned tasks more readily than models trained only on categorical image-text pairs.
The limitation is that current spatial grounding is brittle outside the training distribution. Unusual viewpoints, occlusions, or spatial descriptions that combine multiple relations (“the object between the cup and the edge of the table”) can produce incorrect attention and therefore incorrect actions. This is an active area of improvement in both the architecture and the data composition used for training.
Interaction with Classical Control
Even the most end-to-end VLA systems do not eliminate classical control entirely. There are practical reasons for this boundary.
High-frequency joint control runs at 500 Hz to 1 kHz on many industrial arms. A neural network inference at that frequency, for a model with hundreds of millions of parameters, is not feasible on the compute available at the edge. The practical operating frequency for VLA inference is 1 to 30 Hz depending on model size and hardware. The gap between the VLA’s output frequency and the joint controller’s required frequency is bridged by a classical controller that tracks the VLA’s high-level targets.
Additionally, stability guarantees for classical controllers are well understood. Lyapunov stability analysis applies to impedance and PD controllers in ways that it does not apply to neural network policies. In manipulation tasks where the robot contacts the environment, unexpected contact forces can cause instability if the controller does not respond appropriately. A classical controller wrapping the neural policy provides a safety layer that limits end effector forces and velocities to safe ranges regardless of the neural policy’s output.
The result is a hierarchy where the VLA model handles semantic reasoning and task-level decision making, and classical control handles the physics of joint actuation. This is not an architectural compromise; it is a practical division of responsibility that maps each component to the problem class it handles well.
Where This Stands
VLA models have moved from research systems demonstrated in single-lab environments to pre-trained open-weight models that can be fine-tuned on a specific robot setup with a modest number of demonstrations. The trajectory from the original RT-1 paper through RT-2, OpenVLA, and the Octo family represents real progress in generalization across tasks and environments.
The unsolved problems are also real. Generalization to significantly out-of-distribution environments remains unreliable. Long-horizon tasks that require persistent reasoning over extended episodes are handled poorly without explicit memory structures. Safety properties of neural policies in contact-rich manipulation are not well characterized, and the failure modes are not always predictable from the training distribution. Data collection at the scale needed to substantially improve these properties is still expensive.
The core abstraction that observation plus instruction should map to action is sound, and the architectural machinery to implement it is now available. What determines how far VLA models generalize is the breadth and quality of the data they are trained on and how well the training distribution covers the deployment conditions.