A technician kneels beside a commercial water heater, phone propped against a pipe, an AI assistant watching through the camera. She reaches for the wrong valve and the assistant says, “Stop, that’s the supply line.” Whether that counts as useful guidance depends entirely on what happens next. Did the warning arrive in time to prevent an incorrect action? Did she trust it? Did it confuse her about the next step?
Those questions are about to matter much more. AI assistants are leaving the chat window and becoming real-time guides for physical work, watching live video and instructing people through multistep tasks.
The frontier moved quickly this year. In May, Thinking Machines previewed interaction models that continuously take in audio, video, and text and respond in real time, rather than waiting for a user to finish a turn. In June, Meta researchers released a wearable video dataset for proactive procedural assistance, annotated for moments when users go off plan and how to recover. In September, Google introduced Gemini 3.8 Live, combining real-time visual understanding, spoken conversation, and the ability to work on tasks in the background.
But there’s a catch: Most evaluations of visual assistants replay prerecorded video and grade responses against an answer key. That tells you whether a warning was correct. It can’t tell you whether the warning worked, because a recording can’t react.
So Salesforce AI Research built Interactive Visual Gym, a simulated proving ground where AI assistants coach virtual workers that can react to instructions. That way, researchers can see whether the advice actually changes the outcome, develop benchmarks, and ultimately improve these visual assistants.
The feedback loop of a live trial, but at scale
A leader deciding whether to give a visual AI assistant to frontline teams wants to know if those teams are more likely to finish the job, and finish it correctly, with the assistant than without it.
The obvious approach is to test with real people, but that doesn’t scale. Knowing whether an assistant reliably helps technicians would take thousands of training runs with people of different skill levels, making different mistakes at different moments. Recruiting that many participants for every device and procedure would be slow and prohibitively expensive. What’s needed is the feedback loop of a live trial at the scale of a dataset.
That’s what Interactive Visual Gym provides. IVG recreates everyday physical tasks like cooking or managing smartphone settings. It pairs each task with a simulated user performing the task while receiving guidance from the visual AI assistant. The simulated user’s actions and the simulated environment change in response to this guidance.
In a kitchen environment, for example, a simulated cook works through a recipe like assembling a burger: one bun, then the patty, then the cheese. If it grabs the wrong ingredient, the AI assistant watching has to catch the mistake in time so the cook can react to whatever it says.
Three design choices make IVG practical. First, the AI assistants speak in natural language (“go to the pan with the meat”), not in the joint velocities and degree coordinates of classical robotics simulators. Second, we can schedule out-of-plan moments at precise points in a task while keeping the user’s dialogue consistent with its behavior. These include mistakes, mid-task plan changes, questions, and reminder requests. Third, simulated users vary along four axes: skill, trust in AI, preferred execution speed, and tolerance for interruption. A seasoned user who wants to be left alone and a novice who wants every step spelled out need very different help.
What today’s assistants get wrong, and how IVG can help
Using IVG, we built IVG-Bench: 315 scenarios across three environments, used to evaluate 15 assistants each running on a different model. The strongest, running on Gemini 4 Argon, completed only 53.4% of tasks within the time limit, averaged across environments, and open-weight models generally trailed the leading frontier models. This is still a challenging benchmark to achieve.
Our dynamic evaluation exposes several problems with AI visual assistants. The first is that a model can help users finish more often but might actually give clumsier advice along the way, so we need two metrics to measure capability: task success and interaction quality. The second is that dynamic testing exposes failures that static tests miss. Latency turns correct advice into stale advice. Telling the user to approach the table after the user has already arrived there is pointless. Some assistants repeat instructions the user has already acknowledged. For some open-weight models, guidance degrades as the task goes on.
To check that our simulation-based scores reflect real experience, we had people control the cook in the simulator, work with different assistants, and rank them. When people preferred one assistant over another, our automated interaction-quality score from the simulation corresponded to their preference 78.5% of the time. Real people and simulated users also behaved similarly, with one difference: Simulated users speak and reply far more often.
Our simulated environments can also actively improve the performance of visual assistants. In an early proof of concept, we used the simulator to train an open-source model, incentivizing it to prioritize task completion and error prevention. The model’s success rate improved and the assistant became noticeably better at delivering the right advice at the right time, stepping in proactively before mistakes happened.
What this means for leaders
Ultimately, real-time visual assistants will be judged by outcomes. Before deploying one, ask three things: Does it measurably help people finish the task? Does its guidance arrive while it can still change the outcome? Does it hold up deep into a long job, or when the user goes off script? Answering those questions means testing the interaction itself, in environments where advice has consequences. That’s what Interactive Visual Gym is built to do.
- Explore the project and interactive demonstrations
