Concretely, Mk1.5 ingests text, images, video, and audio, and emits text, points, boxes, polygons, clips, and object tracks. We’ve already deployed it on drones, robotic dogs, smart glasses, and smart phones. Our partners are doing the same.
What’s New
Tracking an object through a video
In order for an embodied agent to complete real-world tasks (e.g., inventory tracking, sports analytics), it is often necessary to track unique entities across time. To enable this, we trained Mk1.5 to emit object tracks as timestamped geometries, rather than one-off, per-frame detections.
The result: Mk1.5 leads three of the four video object segmentation benchmarks we measured.
With native object tracking support, Mk1.5 reduces the need for complex re-identification pipelines that sit downstream of most detection models. Whether you’re tracking inventory in a retail environment or players on a sports feed, you can deploy this capability today.
Understanding egocentric video
First-person video understanding enables applications ranging from annotating robotic training data to helping smart glasses interpret and respond to what a wearer sees.
By training specifically on egocentric video we are able to show significant gains on tasks like hand tracking and subtask segmentation: Mk1.5 localizes hands in a first-person frame 50 percent better than the strongest Gemini model we measured.
BENCHMARKMk1.5Gemini3.1 ProGemini3.7 FlashGemini3.8 FlashΔ VS 3.1 PROΔ VS BEST GEMINIPERCEPTRON_EGO · HAND-LEVEL PERCEPTIONscore, 0 to 1hand_box0.94330.44670.59110.6179▲ +0.4966 (+111%)▲ +0.3254 (+53%)subtask0.63310.60560.60150.6253▲ +0.0275 (+5%)▲ +0.0078 (+1%)hand_action_verb0.27520.31370.34260.3026▼ -0.0385 (-12%)▼ -0.0673 (-20%)hand_caption0.54350.45020.52110.4944▲ +0.0933 (+21%)▲ +0.0224 (+4%)EGOSCHEMA · LONG-FORM VIDEO QAaccuracy, %egoschema80.4078.0080.0081.20▲ +2.40 (+3%)▼ -0.80 (-1%)egoschema_hard63.7555.6256.8856.25▲ +8.13 (+15%)▲ +6.87 (+12%)bars start at 0 · bold = best in row▲ Mk1.5 leads▼ Mk1.5 trails
Adding audio as a new modality
Mk1.5 is natively built to understand audio. We do not claim to be at the frontier of audio-visual understanding in this first release, but our current public capabilities are already strong enough to be useful:
MODELDAILYOMNI ↑WORLDSENSE ↑OMNIBENCH ↑AVHBENCH ↑Mk1.574.6750.3251.0580.56AV-Flamingo Instruct72.40[1]50.30[1]48.50[1]≈79.74c[1]AV-Flamingo Think73.90[1]51.60[1]50.60[1]≈83.61c[1]Qwen2.5-Omni-7B62.07[12]45.40[13]56.13[4]–Qwen3-Omni-30B-A3B Instruct71.85[12]54.00[13]––Qwen3.5-Omni-Plus84.68a[12]62.80[13]––Qwen3.5-Omni-Flash81.80[3]57.90[3]––Gemini 3.1 Pro Preview82.79[12]65.50[13]––Gemini 2.5 Pro–65.10b[13]––Mk1.5reportedderived estimatebars start at 0 · bold = highest in column · – = not reported
Audio-visual capabilities enable everything from time-aligned transcription to sound-based event detection.
Learning to use tools
Mk1.5 supports arbitrary tools: any function declared in the OpenAI format with a JSON Schema. Once defined, Mk1.5 decides what to call and when.
Mk1.5 remains competitive with other frontier models on multimodal search evaluated across LiveVQA-W, Browsecomp-VL, MMSearch.
LiveVQA-W
Visual questions on recent news, videos and papers. Overall accuracy, %, higher is better.
Mk1.5 shows a large improvement with tool calls enabled (+36.1 point over no tools on MMSearch). This performance gain is significant because we do not train our models to memorize world knowledge. Instead we train them to learn efficiently and adapt. By demonstrating effective tool use, we show that Mk1.5 is equipped to learn new control surfaces.
In our demo environment, we support web search, page reads, zoom, and reverse image search, enabling Mk1.5 to leverage knowledge that it did not aquire in training:
Tool use can be the difference between a static control model and a physical agent that generalizes to new tasks and functions across many embodiments.
Parallelizing with sub-agents
Beyond making tool calls, Mk1.5 can deploy sub-agents to parallelize and accelerate task completion:
Faster inference while pushing the frontier
Mk1.5 delivers ~2-5x faster end-to-end request completion, while continuing to push the frontier on performance.
MK1.5 VS MK1 · END-TO-END LATENCY
Up to 4.7× faster, end to end
Median time from request to complete answer, lower is better.
Mk1Mk1.5bars scaled within each workload
And the improvements in inference speed are tangible. This example is shown at 1x speed:
Reasoning in visual space
Mk1.5 maintains frontier capability on video and embodied reasoning.
Mk1.5 on video understanding
Accuracy, %, higher is better. A dash means no reported result.
Coupling advanced visual reasoning with 5x inference speed up enables Mk1.5 to produce more accurate answers (e.g., on segmentation tasks) much faster:
Embodied applications
Given its increased capability set, Mk1.5 can be reliably deployed across several new embodiments.
In the below examples, leverage Mk1.5 as part of agentic control loops, exposing each platform's control surface as tools. Mk1.5 then decides the sequence of actions, or tool calls, required to complete the task at hand:
Drones
Quadrupeds
Smart glasses
Personal computing devices
Building with Mk1.5
Mk1.5 is available today. You can send it audio, ask it to track an object through a video, hand it your own tools, or point it at a robot and let it drive.
Mk1.5 is available via the Perceptron Platform and our updated SDK.
If you'd like to learn more about deploying Mk1.5 on your infrastructure, please reach out to sales@perceptron.inc.
