Introducing Qwen3.5 4B on LLM Gateway—optimized by AssemblyAI for voice rewrite

A user dictates a message and the raw transcript arrives exactly as spoken: filler words, spoken punctuation, a correction halfway through—"6 of the 8— oh no, 5 of the 8." Before the words can appear on screen, something has to turn them into clean, formatted text. That rewrite runs on every utterance, and plays a large part in the latency perceived by users.

Today, many teams run this straightforward task on models built for much harder problems. Large general-purpose models handle it well but add latency and cost the task doesn't need—and the small open models suited to it are hard to find.

That's why today, we're bringing the Qwen3.5 4B model to the LLM Gateway: hosted by AssemblyAI on our own GPUs and served in a latency-optimized, 32k-context configuration.

It's built for exactly this work—dictation cleanup, transcript rewrite, and other quick, interactive voice features—and on those tasks it averaged 612ms, 1.9× faster than GPT-4.1 at 94% lower cost per hour of audio. The model is available in the AssemblyAI API now—test it on your own voice tasks, no credit card required.

1.9× faster

Average response vs GPT-4.1 on voice rewrite tasks

94% cheaper

Cost per hour of audio vs GPT-4.1

$0.10 / $0.50

Per million tokens (prompt / completion), optimized and hosted by AssemblyAI

The right model for voice rewrite

A small model's value doesn't show up on a general leaderboard, which averages across tasks it was never designed for. It shows up on the specific task it was built to do. The qwen3.5-4b-32k-fast model hosted by AssemblyAI is purpose-built for one class of work: fast rewrites over voice data.

• Dictation cleanup—filler words out, punctuation in, self-corrections resolved

• Transcript rewrite—raw speech-to-text output reshaped into emails, notes, or messages

• Live formatting—applying formatting rules to streaming output while the user is still talking

• Turn summarization—compressing a conversational turn into a line the next system can act on

Frontier models keep getting better, but not every task needs one. For voice tasks, customers are increasingly choosing small, fast, low-cost models instead—and choosing a small model means being clear about the trade: qwen3.5-4b-32k-fast supports max_tokens, temperature, and stream only.

What a rewrite looks like

Dictated recap, one rewrite pass

Raw speech-to-text output

After Qwen3.5 4B

Hi Marissa, quick recap on where we landed with the Q3 vendor review. We got through five of the eight assessments. The last three are waiting on security sign-off. The big thing is that pricing came back higher than we modeled, something like 15% over. I think we need to revisit the budget line before we commit to anything. Can you pull the original forecast for me? I’ll put 30 minutes on the calendar.

622 ms · $0.00011

Notice what the rewrite handled: filler words dropped, both mid-sentence corrections resolved to the speaker's final intent—"6 of the 8— oh no, 5 of the 8" became five of the eight, and "between 12— no, 15%" became 15%—and the recap reformatted into clean paragraphs. The numbers in the corner are real: this exact audio ran through the exact pipeline below, and the rewrite came back in 622 ms at a cost of about a hundredth of a cent.

How Qwen3.5 4B benchmarks on voice rewrite tasks

We benchmark every model on our LLM Gateway on the task they're built for, and for qwen3.5-4b-32k-fast, that task is voice rewrite.1 Dictation cleanup, transcript rewrite, formatting, and turn summarization, measured on representative voice data rather than general-purpose benchmark prompts.

For building dictation features

If you're building dictation—voice typing, dictated notes and messages, spoken commands that become clean text—the pattern we recommend is a two-step loop: our Sync API for turning audio into raw words fast at flagship accuracy—1.59% word error rate on short-form audio in ~134 ms at the median—then a rewrite pass to turn raw words into clean, formatted text with qwen3.5-4b-32k-fast.

Here's the whole pipeline—the same two-step loop we run ourselves—in about 30 lines. One API key covers both calls. Speech in, clean formatted text out.

Request

For live formatting, stream the rewrite and render tokens as they arrive:

Streaming request

For building voice agents

Voice agents make a different demand of their model: tool calling, multi-step reasoning, and accuracy under a live conversation's latency budget. Because Qwen3.5 4B doesn't support tool calling, it isn't the right choice here. For agent workloads, we recommend qwen3-next-80b-a3b, which is strong in a different dimension: accuracy on agent tasks relative to the mid-tier proprietary models, at a fraction of their cost.

Using it is the same integration as Qwen3.5 4B: point your chat-completions client at llm-gateway.assemblyai.com and set the model to qwen3-next-80b-a3b. Tool calling and the full agent loop work through the standard OpenAI-compatible interface, so existing agent frameworks connect without special handling.

Pricing and free credits

LLM Gateway usage draws down your credits at each model's list price, so those credits go a long way on a 4B model: at $0.10/$0.50 per M tokens, a dictation cleanup costs a fraction of a cent.

LLM Gateway is the inference layer for Voice AI, and the Qwen family sits alongside the frontier models your product already depends on—Claude, GPT, and Gemini are all available through the same endpoint and the same API key. Fast open models for the high-volume rewrite path, frontier models for the reasoning path, one integration for both. The full model list and pricing is public, no auth required.

Get started

Create a free account, grab your API key, and the LLM Gateway quickstart defaults to qwen3.5-4b-32k-fast—your first rewrite is one curl command away. If you'd rather start without writing code, you can test speech-to-text and rewrite together on your own audio in the Playground.

Frequently asked questions

It’s an open-source Qwen3.5 4B model running as an AssemblyAI-hosted deployment on our own GPUs, served in a latency-optimized configuration with a 32k-token context window. We built the deployment to run the rewrite step in our own dictation pipeline, and it’s available to all developers through LLM Gateway.

New accounts get free credits with no credit card required, and usage draws down those credits at the model’s list price of $0.10 per million prompt tokens and $0.50 per million completion tokens. It’s free credits rather than a free model—but at 4B pricing, those credits cover a serious volume of testing.

It’s best for rewrite-style work over voice data: dictation cleanup, transcript rewrite, formatting, and turn summarization. It isn’t suited for agents or structured extraction—the model supports only max_tokens, temperature, and stream, without tool calling or structured output. For agent workloads, we recommend qwen3-next-80b-a3b or a frontier model on LLM Gateway.

No. "Optimized and hosted by AssemblyAI" refers to the deployment: we self-host the open-source model on our own GPU infrastructure in a serving configuration tuned for fast Voice AI workloads. The model weights are the open-source Qwen3.5 4B release.

They spike on different tasks. Qwen3.5 4B (qwen3.5-4b-32k-fast) is the speed-and-cost pick for rewrite tasks and doesn’t support tool calling. Qwen 80B (qwen3-next-80b-a3b) is the agent pick: it scored 0.800 on the tau2 retail agent benchmark—above Claude Haiku 4.5 (0.733) and GPT-4.1 (0.700)—at $0.15/$1.20 per million tokens.

Yes. LLM Gateway is one OpenAI-compatible endpoint over frontier and open models alike—Claude, GPT, and Gemini remain fully available with the same API key you use for the Qwen models and for speech-to-text. See the available models reference in our docs for the complete list.

Methodology

1 We transcribed dictated audio clips (35–66 seconds) with AssemblyAI’s universal-3-5-pro async speech-to-text, then ran each transcript through every candidate model on the AssemblyAI LLM Gateway using an identical cleanup prompt, issuing non-streaming sequential calls and recording wall-clock latency and the token counts returned in each response. Back to the benchmarks