Dev48
Language
  • About
  • Services
  • Industries
  • Technologies
  • Articles
  • Contacts
Book a call
    Home/Articles/Research preview pika enact
Dev48
info@dev48.org+375 29 320-40-96WhatsApp@isky_by

© 2026 · All rights reserved.

Research Preview: Pika Enact

Source: Pika•November 18, 2026 at 07:06 AM

Research Preview: Pika Enact

Pika Enact is a research preview of live staging, actor direction, and camera control, built on Pika H3 Flex to generate scenes that respond as they play.

Creating Scenes that Respond to You

November 19, 2026. By the Pika Labs team

At Pika, we believe the next generation of AI creation should go beyond the prompt box, and more closely resemble how we already make films. Pika Enact is an exciting step in that direction.

Pika Enact is designed to mimic the experience of directing on set. It’s an experimental live-streaming environment built on Pika H3 Flex that generates video and sound in sub-second chunks, carrying scene context forward while incorporating new direction into the ongoing generation.

This allows the creator to interact with “actors,” manually re-stage a scene, and manipulate the camera—all in real time.

[insert launch video here]

The technology enables three new modes of interaction:

  • Live Staging. Place and reposition actors or objects within a running video with your mouse. The scene continues while you move them.
  • Live Actor Direction: Speak directly to actors on set, and they’ll respond conversationally. Give them notes about dialogue, emotion, or movement, and they’ll enact them. Call “Action” to turn the rehearsal into a fresh, continuous take, and the actors will remember everything you’ve discussed.
  • Live Camera Control. Direct camera movement with your mouse around a live, playing scene.

Below, we walk through each one: the data they learn from, their technical architecture, and some demo videos.

Live Staging

Scene design and actor blocking are mainstays of a live directing experience. But AI video creation makes that process inexact and time-consuming. With Live Staging, rather than waiting for a generation to load in order to see the outcome of your direction, you get instantaneous feedback; you can manually shape the scene as it plays.

The live staging within Pika Enact lets the creator drag an object or actor into place with their mouse, which creates several key benefits:

  • Precise placement: Where a prompt can only suggest “a bit to the left,” a cursor drags subjects to the exact intended spot.
  • Consistency with environment: The model generates whatever you move—a person or a chair—together with the rest of the scene. As you drag, the subject responds naturally to the movement, while its pose, lighting, and shadows adapt to its surroundings. The result has no cut-out edges, no hole where it stood, and no compositing pass to clean up afterwards.
  • Real-time responsiveness: The video keeps playing while you drag, and on a single server the model generates it 1.8× faster than it plays. Additionally, the subject actively reacts to being moved, shifting weight and adjusting posture mid-air before settling naturally where you let go.

To keep up with the cursor movement, we distill the model to a few steps and stream the mouse input and decoding in small pieces. In our low-latency setting, the picture responds in under 0.4 seconds on H100, decoding included.

The Data Engine

Training interactive video dragging requires strictly consistent before-and-after video pairs synchronized with frame-level cursor trajectories, which real-world footage cannot provide.

We built a physics-driven synthetic data engine to solve this. Starting from an untouched video, we animate a 3D stand-in along a drag path to simulate natural weight and momentum. Our video models then render photorealistic target clips while automatically logging mouse positions and button states.

This produces scalable, zero-labeling training pairs where only the dragged subject changes, covering lifting, multi-step carries, and inserting new subjects from photos into live scenes. The dataset drives our entire pipeline, from full-clip training and real-time distillation to evaluation.

For data access or more detail on how the data was made, contact our team at enterprise@pika.art.

Technical Architecture

To make Pika H3 Flex draggable in real time, it has to learn two things: follow a cursor, and generate a stream instead of a clip.

Dragging trajectory: following your cursor. To follow a drag, the model has to know three things at every moment: what your mouse is doing, where it is, and what the scene would look like if you left it alone. Each comes in as its own signal. The mouse is sampled 24 times per second, and its button state and motion guide how the grabbed person or object moves at that exact moment. The cursor's position is marked on a heatmap over the frame, so the model knows precisely which spot you grabbed. The untouched video is fed in frame by frame alongside, so everything you don't touch stays as it was. All three enter H3 Flex through small new modules that start from zero, so the model learns to follow your hand while keeping H3 Flex's look.

From a clip to a stream. A model that generates the whole clip at once can't react to you: you would have to finish the drag before it starts, then wait seconds for the result. And when you're holding someone with your mouse, even a 0.7 s delay already feels sluggish. So we cut the video into short chunks and let each chunk look only at the past; the shorter the chunk, the sooner the picture can answer (0.7 s in the main model, 0.17 s in our low-latency model). This is how a language model writes one token at a time, and it brings the same tool: a key-value cache. Each new chunk reads the prompt and source video, the first chunk of the session, and the four most recent chunks from the cache; older chunks drop out. Positions are re-based every chunk so they never grow, which lets a session run indefinitely at constant cost. We first train this causal model on real history (teacher forcing).

But streaming alone isn’t enough. Each chunk still takes around twenty steps, and once the model reads its own output instead of real footage, small errors pile up from chunk to chunk. We fix both with self-forcing distillation. During training, a fast few-step student generates whole videos chunk by chunk from its own output, exactly as it will in use. The original full-clip model and a second, trainable critic both score that video, and the gap between their scores tells the student how to move closer to what the full-clip model would produce (distribution matching distillation, DMD). The full-clip model scores with the mouse signal emphasized, so the student learns to follow the cursor firmly without needing guidance itself. The result: each chunk takes just a few steps instead of twenty, and the model holds up on its own output.

2. Live Actor Direction

Once everything has been staged, the next step is typically directing your cast. With Enact, the people in a video become actors you can talk to. Type or say a note, such as “move closer to her”, “tell him why you’re upset” or “make it a kitchen at night”, and they play it out while the video keeps running.

This mimics the experience of directing on set in several important ways:

  • Communicate conversationally: Give a note and the actors act on it; talk to them and they answer you in character, in their own voices. The video generates picture and sound 1.75× faster than they play, so the interaction happens at a real-time pace.
  • Rehearse first, then shoot. Work through a performance iteratively. Give notes, have the actors try gain, and when it feels right, call Action. Everything you rehearsed becomes one clean take, shot fresh from the first frame.
  • The actors have minds and memories of their own. Each is an AI agent with a distinct personality, its own logic, and the ability to think and respond in character. They remember what you liked, what you didn't, and everything that's happened in the scene. You never have to repeat a note or start from scratch. Every interaction builds on the last, making the characters feel more natural, consistent, and alive.

Data

During distillation, the real-time model learns from Pika H3 Flex itself: it generates audio-video clips chunk by chunk from its own output, and the full Pika H3 Flex model’s judgment of those clips is what it learns toward. For more on the training data, data access or custom training, contact our team at enterprise@pika.art.

Technical Architecture

Live Direction is an agentic model within Pika Enact that thinks and performs at the same time. It understands what you mean, plans what should happen next, remembers your notes, and acts the scene out live, picture and voices together. A script model does the thinking and a real-time video model does the acting. And while you talk to the agent, the performance never stops.

From your notes to beats. The script model, a language model, reads your note together with the opening frame and the agent’s memory of the session, plans what should happen next, and hands it to the video model as one or more beats through tool calls. A beat is a short instruction for the video model: who moves, who says which line and with what feeling, where each person looks, and what the light and setting are. When nothing is queued, the scene holds, with people breathing and blinking where you left them.

A causal model with self-forcing. We distill H3 into a causal model with self-forcing, the same recipe as dragging: it generates short chunks that look only at the past, reuses a key-value cache, and is trained on its own rollouts with distribution matching distillation (DMD), down to a few denoising steps per chunk. Each chunk sees the first frame as an anchor plus the last few seconds, which keeps the cast and the set consistent through long takes. The actors’ voices come from the same model as the picture, with no separate voice generator.

Keeping the scene where you left it. The anchor has a side effect: when the instructions stop changing, the model drifts back toward the opening. Heads turn back to the camera within about a second, and dimmed lights come back up after ten seconds or so. So every beat names where each person is looking, and once the light or setting has changed, Real-Time Director adds short hold beats every few seconds that restate it. They never appear in your script.

Rehearsal and Action. In rehearsal, the actors know the camera is there and talk to you, and every line, yours and theirs, is kept word for word. When you call Action, the script model rewrites that record into one continuous performance without the back-and-forth, and the take starts again from the first frame.

3. Live Camera Control

Framing and camera movement are what turn a scene into something cinematic. Pika Enact lets you steer a virtual camera with your mouse while a scene is continuously generating—panning, dollying in for a close-up, or orbiting around the subjects.

  • You direct the lens in real time. Move and drag your mouse while the stream is playing to adjust camera angles, perspective, and movement on the fly—shaping upcoming frames as the scene unfolds instead of waiting for offline renders.
  • No existing video needed: Unlike post-hoc video editing that modifies an existing clip, the system generates audio and video from scratch. An appearance image anchors the cast and set, while a lightweight 3D motion proxy (such as an untextured Blender viewport) guides the camera path.
  • It runs faster than real time. On an 8×NVIDIA B300 node (640×480 @ 24 fps), the model renders initial frames in 0.51 s and generates subsequent chunks at 1.50× real-time speed, leaving headroom for proxy capture and streaming.

Data

To teach Pika H3 Flex how free-roaming 3D camera moves affect both visual perspective and spatial acoustics, training data must be tightly synchronized down to the frame level.

  • Virtual Soundstage Capture: We built an automated Blender-to-Video synthesis pipeline that simultaneously records multi-angle camera trajectories, untextured 3D motion reference streams, target RGB video frames, and spatial audio tracks in controllable 3D environments.
  • Rigorous Quality Filtering: Using automated optical flow checks and cross-frame identity verification, we filter out clipping artifacts and subject inconsistencies, retaining only high-signal-to-noise pairs for training.

Technical Architecture

Two stages turn offline 3D supervision into a live, responsive camera generator: a full-reference teacher that sees the entire sequence, and a causal student that generates chunk by chunk as you move the mouse.

From motion proxies to live chunks. During streaming, the causal student reads the current window of 3D camera motion alongside the appearance reference and its own generated history, advancing ~0.71 seconds of synchronized picture and sound per chunk. As you move the virtual camera, new trajectory windows immediately steer upcoming frames without stopping the stream.

A causal model with self-forcing. We distill the full-reference H3 teacher into a causal student trained on its own rollouts with distribution matching distillation (DMD), down to 3 denoising steps per chunk. Each chunk attends to the opening frame as an anchor plus a sliding window of recent history with synchronized relative positions—keeping character likeness and room geometry consistent across long orbits while keeping memory bounded.

Breaking the real-time barrier. To push generation below real-time playback, we cache static appearance prefixes across chunks, reuse fixed memory layouts and routing metadata with zero numerical loss, and decode latent frames through a streaming video decoder. Together, these cut steady-state chunk compute to 0.473 s per 0.708 s of audio-video (1.50× real time).

What’s Next

Enact is an early look at what happens when video listens to you—and responds—while it plays. In our own tests, we were startled by how much more real and visceral it feels to generate video this way. So, while it’s only in research preview right now, we think this is the beginning of a new paradigm for creation. It’s the immediacy of directing in real life paired with the expansive possibilities of generative media.

← All articles

What this article says

More in AI & Machine Learning

All →
H2O.ai Appoints Steven Vattuone as Chief Financial Officer
H2O.ai

H2O.ai Appoints Steven Vattuone as Chief Financial Officer

Does a Code-Tuned Embedding Model Pay Off? papr-embed-code-1 in Qdrant
Qdrant

Does a Code-Tuned Embedding Model Pay Off? papr-embed-code-1 in Qdrant

Ai2 and Taylor Geospatial partner to make geospatial AI easier to compare, run, and build on
AI2 (Allen Institute for AI)

Ai2 and Taylor Geospatial partner to make geospatial AI easier to compare, run, and build on

LangSmith for Coding Agents: Now track spend by outcome
LangChain

LangSmith for Coding Agents: Now track spend by outcome

Time-Series Models in the Streamhouse: A New Frontier and Use Cases
Hugging Face

Time-Series Models in the Streamhouse: A New Frontier and Use Cases

Rumble-2-Flash: an open 0.5B speech model for 40+ languages
Hugging Face

Rumble-2-Flash: an open 0.5B speech model for 40+ languages

More from Pika

Pika vs Runway: pricing compared
Pika

Pika vs Runway: pricing compared

Pika H3 Flex Technical Deep Dive
Pika

Pika H3 Flex Technical Deep Dive

Introducing Pika H3 Flex
Pika

Introducing Pika H3 Flex

Pika Is Now Inside ChatGPT
Pika

Pika Is Now Inside ChatGPT

Now Hear This: Pika SFX
Pika

Now Hear This: Pika SFX

Introducing Pika Music
Pika

Introducing Pika Music