Reinforcement Learning That Gives VLAs "Reflexes": Real-Time EXPO-FT and Its Application to Vision-Tactile World Models

This is an English translation of the original Japanese article on the author’s Hatena blog: VLAに「反射神経」を持たせる強化学習：Real-Time EXPO-FTと視覚・触覚World Modelへの応用.

I read the paper “Reinforcement Learning for Real-Time Vision-Language-Action Policies,” published on September 16, 2026.

What makes this work interesting is that, instead of leaving every decision to a large VLA, it combines it with a small, fast-reacting policy and uses reinforcement learning to improve the robot’s behavior.

It is closely related to my own work, “Vision-Tactile JEPA World Models for Failure-Aware Long-Horizon Robotic Manipulation,” and is particularly useful when thinking about how to connect failure prediction to action correction.

In this post, I summarize the paper’s technical points and consider how I can apply them to my master’s research.

Paper: https://arxiv.org/pdf/2609.18207
Project: https://pd-perry.github.io/real-time-expo-ft/

1. The problem with VLAs: the world changes while they think

A VLA generates robot actions from camera images and language instructions. However, the larger the model, the longer inference takes.

For example, when a robot tries to grasp a moving object, the object keeps moving while the VLA is recognizing the image and generating an action.

In other words, by the time the action is executed, the observation it was based on is already stale.

Conventional Real-Time Chunking (RTC) has reduced interruptions in motion by asynchronously generating the next action sequence while the current one is being executed.

However, it cannot always adequately respond to environmental changes that occur before the next action sequence is generated.

This work therefore takes the approach of separating action generation by the VLA from action correction based on the latest observations.

2. How Real-Time EXPO-FT works

The proposed method, Real-Time EXPO-FT, combines a pretrained VLA with a small edit policy and a value function (Q-function).

Processing is divided into three main steps.

STEP 1: Action generation by the large VLA

Multiple action chunks (action sequences) are generated asynchronously from slightly older observations.

STEP 2: Action correction by the edit policy

Just before execution, the latest observation is checked and small corrections are applied to the action candidates generated by the VLA.

STEP 3: Action selection by the Q-function

The original and corrected candidates are evaluated, and the action sequence with the highest Q-value is executed.

What is important is that the edit policy is not a simple correction rule but a policy improved through reinforcement learning.

The edit policy learns to modify actions toward higher Q-values, and the Q-function is updated with temporal-difference (TD) learning using the outcomes of actual trials.

Meanwhile, the large VLA is fine-tuned by imitation learning on action data. In the real-robot experiments, π0.5 is used as the base VLA.

To me, this design felt like separating, in human terms, the role of making plans from the role of reacting on the spot.

Rather than speeding up VLA inference itself, it assumes that latency exists and builds a mechanism for correcting decisions right before execution.

3. Results: average real-robot success rate from 42% to 97%

To assess the ability to handle dynamic environments, the paper evaluates the method on four real-robot tasks.

Task	Description
Dynamic Picking	Grasp an object moving on a turntable
Ball Balancing	Stabilize a ball on a plate
Object Passing	Receive a moving object from another robot
Soccer Kicking	Kick a ball in time with a moving target

The robot operation data used for online learning on each task is at most 10 minutes. This does not mean 10 minutes including training compute time or prior demonstration collection.

The real-robot evaluation results are as follows:

Task	Before (SFT)	Real-Time EXPO-FT
Dynamic Picking	19/30	30/30
Ball Balancing	8/30	28/30
Object Passing	10/30	30/30
Soccer Kicking	13/30	28/30
Average success rate	~42%	~97%

From Table I of the paper. Each task was evaluated over 30 trials.

What I especially want to highlight is that, while using a pretrained VLA, the method improves performance on dynamic tasks with only a small amount of real-robot experience.

Note that the paper also includes simulation evaluations on 10 Kinetix tasks, but those use a flow-matching policy that takes state vectors as input, not a VLA.

These need to be distinguished from the real-robot VLA evaluation.

4. Relation to my research: toward VLAs that predict and avoid failure

From here, I consider the relationship to my master’s research.

My research theme is:

Vision-Tactile JEPA World Models for Failure-Aware Long-Horizon Robotic Manipulation

My research aims to build a world model that integrates RGB images, depth, touch, and robot state to predict future states.

A particularly important aspect is predicting grasp failures and object slip in advance, and connecting those predictions to action correction and replanning.

Compared with Real-Time EXPO-FT, the two share some goals but take different approaches.

Aspect	Real-Time EXPO-FT	My research
Main goal	Real-time control under inference latency	Failure prediction and improving long-horizon tasks
Observations	RGB, robot state	RGB, depth, touch, robot state
Core techniques	VLA, edit policy, Q-function	VLA, JEPA world model
Future prediction	Estimating action values via Q-values	Predicting future latent states
Action improvement	Action correction via reinforcement learning	Action correction and replanning based on failure prediction
Time scale	Real-time control of short action sequences	Long-horizon tasks spanning multiple manipulation stages

Real-Time EXPO-FT corrects actions using the latest observations. My research, in contrast, aims to predict future states and avoid failure.

If this difference can be leveraged well, I think it could lead not just to real-time control, but to robots that detect signs of failure and correct their actions before failing.

That said, it is not the case that Real-Time EXPO-FT ignores the future entirely.

The Q-function estimates future cumulative reward. The difference from JEPA lies in whether one estimates the value of an action or predicts the future latent state itself.

5. Combining a JEPA world model with reinforcement learning

The research idea I took from this paper is to use a JEPA world model for action correction.

For example, consider a scene in which a robot lifts an object.

Suppose the VLA generates the action “grasp the object and lift it.”

However, if the fingertip tactile sensors indicate that the object is beginning to slip, lifting it as is may cause it to fall.

Could we then use tactile information and world-model predictions of the future to correct the action to be executed?

Proposed research architecture

RGB / Depth / Tactile / Robot State / Language
                     |
          +----------+----------+
          |                     |
          v                     v
         VLA          Vision-Tactile JEPA
          |                     |
  Generate action        Predict future
     candidates           latent states
          |                     |
          |           Predict contact state
          |                and slip
          |                     |
          +----------+----------+
                     |
                     v
          Failure-Aware Edit Policy
                     |
         Correct action candidates
                     |
                     v
         Action Selection / Replanning
                     |
                     v
               Robot Execution
                     |
                     v
       New observations / learning

This is my own research concept, inspired by the paper’s architecture. It is not an architecture that was validated in the paper.

Simply using JEPA as a feature extractor here would not make the research particularly interesting.

I think the key is to use the future contact states and slip signs predicted by JEPA as input to the edit policy, or in evaluating action candidates.

Failure-risk-aware action selection with a world model

For example, if the world model predicts future latent states and a failure probability can be estimated from those predictions, the following action selection could be considered:

a* = argmax_a [ Q(s, a) - λ × P_fail(s, a) ]

This expression means selecting an action that has high value and a low probability of failure.

The symbols are defined as follows:

Symbol	Meaning
a*	The selected action
argmax_a	Find the action a that maximizes the score
Q(s, a)	Expected cumulative reward of executing action a in state s
P_fail(s, a)	Failure probability estimated from the world model’s predictions
λ	Weight on failure risk

For example, suppose there are two action candidates:

Action A: lift the object quickly
Action B: adjust the grip force, then lift

Let us assign hypothetical values to each:

λ = 2.0

Action A:
Q(s, A)      = 0.9
P_fail(s, A) = 0.4

Score = 0.9 - 2.0 × 0.4
      = 0.1

Action B:
Q(s, B)      = 0.7
P_fail(s, B) = 0.1

Score = 0.7 - 2.0 × 0.1
      = 0.5

In this case, although Action A has a higher Q-value, Action B is selected once failure risk is taken into account.

In other words, if the world model can predict future slip, it may be possible to correct the VLA’s generated action to a safer one before executing it.

However, this is only a simplified calculation example to illustrate the research idea.

If the Q-function has already learned the loss from failure, separately subtracting failure risk may double-count it.

In actual research, how to integrate the Q-function and failure prediction would also need to be verified.

6. What I want to verify in my master’s research

If I build on this paper, rather than applying reinforcement learning to an entire long-horizon task, I first want to check whether local action correction using failure prediction is actually effective.

For example, on a grasp–lift–transport–place task using an SO-101, I would compare vision-only against vision plus touch.

Then, by comparing an edit policy that does not use world-model predictions against a Failure-Aware Edit Policy that does, it should be possible to isolate the value of future prediction by JEPA.

As evaluation metrics, besides task success rate, I could consider failure prediction accuracy, recovery rate from failures, number of replans, and latency of action correction.

Since reinforcement learning on real robots involves challenges in safety and data collection cost, I think a realistic plan is to first verify in simulation or with imitation learning, and then extend to limited real-robot tasks.

Also, Real-Time EXPO-FT itself does not directly examine hierarchical planning for long-horizon tasks or contact prediction using touch.

How to handle these in my own research will be an important consideration.

7. Summary: VLA reflexes and world-model prediction

Reading about Real-Time EXPO-FT, I felt that VLA action generation, world-model future prediction, and action improvement via reinforcement learning do not necessarily have to be combined into a single giant model.

A large VLA generates actions, and a small policy reacts quickly to the latest state. If world-model failure prediction can be incorporated in between, it may enable action correction that looks further ahead.

In my research, while keeping JEPA-based integrated vision-tactile representations and future prediction at the center, I want to draw on the edit-policy idea presented in this paper.

Ultimately, I aim for robots that do not merely retry after failing, but predict signs of failure, correct their actions, and continue long-horizon tasks.

References

Perry Dong, Kuo-Han Hung, Dorsa Sadigh, Chelsea Finn, “Reinforcement Learning for Real-Time Vision-Language-Action Policies,” arXiv:2609.18207, 2026.

Paper: https://arxiv.org/abs/2609.18207
Project: https://pd-perry.github.io/real-time-expo-ft/

Original Japanese article: VLAに「反射神経」を持たせる強化学習：Real-Time EXPO-FTと視覚・触覚World Modelへの応用