Dev48
Language
  • About
  • Services
  • Industries
  • Technologies
  • Articles
  • Contacts
Book a call
    Home/Articles/We made vllm tokenization 120x faster with elixir
Dev48
info@dev48.org+375 29 320-40-96WhatsApp@isky_by

© 2026 · All rights reserved.

We made vLLM tokenization 120x faster with Elixir

Source: Inference.net•November 21, 2026 at 12:36 AM

We made vLLM tokenization 120x faster with Elixir

How we built ChatTokenizer, an Elixir and Rust tokenizer that matches vLLM and SGLang token for token and is up to 120x faster than their own tokenization

We’ve spent months investing in our inference routing stack, doubling down on a custom request router built in Elixir. This is the first in a series of blog posts discussing our progress and learnings in scaling to trillions of tokens a day. This post focuses on tokenizers and how we’ve managed to make SGLang 33x faster and vLLM 12x faster with tokenization alone.

Why does tokenization even matter?

When processing an inference request there are 4 main stages before you get the first output token: Tokenization, Routing, Prefill & Decode.

Our routing layer makes decisions based on input context length. Knowing the length of the input requires you to tokenize it, or use a rough estimate (like one token per four characters of input). Once we’ve routed the request, we prefill and decode using one of the popular open source inference engines. All of those engines have the option to either: accept a chat completion or accept raw token IDs. Historically, we would depend on the open-source inference engines to handle tokenization, prefill and then finally start decoding.

Our routing layer tokenizes the request to get an estimate of how many tokens this request will take. Inference engines can process requests faster when you limit the input context length, and so knowing the length of a request allows you to route short requests to “short pools” that will provide performance gains, as compared to pools that are configured for a models full context length. The tokenizer we used for our production hot path is Toks (from Actual, Inc.) with the caveat that it doesn’t render chat templates and is written in C which isn’t natively in our routing stack and required work to integrate.

As we have scaled demand over the past few months we’ve pushed the open-source inference engines to their limits and now have internal forks of both vLLM and SGLang patching them for efficiency. We still run up against fundamental limits because their frontends are currently written in Python, a single-threaded interpreted language, meaning slow downs, GIL (Global Interpreter Lock) contention and more which all affects your TTFT (time to first token). Instead of sending raw JSON requests, you can cut down on the overhead incurred by tokenization being executed in a Python process by sending the engines raw token IDs!

Tokenization Library Benchmarking

With that all in-mind we built ChatTokenizer to see how much better we could do!

The graphs here come from our own internal benchmark of open-source tokenization libraries and seeing how they perform across different payloads and compute resources. Our methodology is in the appendix at the bottom of this post.

Our goal with the benchmarks was to see:

  • How does tokenization change with payload size? Does it back up the hypothesis we had that large context conversations cause significant slowdowns?
  • What is the best off-the-shelf option?

From the graphs you can see that vLLM and SGLang themselves are by no means the most efficient tokenizers. SGLang ships their own gateway, written in Rust, which is one of the best options (caveat: that it doesn’t tokenize images). Once we realized the gaps and poor performance of the open source options, it was clear that it was time to raise the bar

What makes ours faster?

We’ve been advocates of Elixir and BEAM and have been using it in production since last December for our Router. Ericsson’s Open Telecom Platform (OTP) has been a fundamental reason of how we’ve managed to scale up to 100s of millions of daily requests. However, it is nowhere near fast enough to beat the other tokenizers here, so we decided to lean into Native Implemented Functions (NIFs) allowing us to use Rust, C & Assembly as part of the BEAM supervision tree.

The core of our tokenizer is still Actual’s toks which uses optimized assembly to produce token IDs that are 1-1 with Hugging Face’s tokenizers but at 13-151x the speed! Toks however doesn’t yet support rendering chat templates and images, among other features we needed internally. This led us to build a supporting Rust and Elixir library around it to handle those features and more.

elixir

Headline Improvements

Each of the improvements below is explained in more detail later in the post but to prevent this from being super-lengthy here are the tl;dr’s for some of the big improvements we have made.

  • Deepseek v4.1 Flash and Kimi K3.1 Pro chat templates were ported to native Rust with the Jinja/Python kept as references and validated with unit tests
  • A quadratic loop in the Deepseek encoders was removed and Kimi's repeated template pieces were memoized within a request
  • Token IDs are passed from C/Rust to Elixir as a binary of little-endian u32s rather than a list to avoid the overhead of passing around lists
  • Inline base64 images are found with a SIMD byte scan and handed to Rust as sub-binaries pointing into the original request rather than copying them
  • We avoid parsing the whole JSON body multiple times and instead extract only the fields that the engine needs

Reading image metadata instead of the full image

Agent traffic is full of images. A single 1M token request with 8 inline images is ~30MB of JSON and almost all of that is base64. The naive path (which we originally took) is to parse the whole body, copy every image string out, base64 decode it and open the image just to find out how big it is, because that's all the engine actually needs to know to work out how many image placeholder tokens to insert.

So we stopped doing that:

  • Before we parse anything we do a SIMD byte scan (memchr) over the raw request looking for "data: strings over 4KiB.
  • Each image crosses into Rust as a sub-binary, which is the BEAM's way of pointing into the original request binary rather than copying it. In the JSON we swap the image out for a tiny marker (data:x-chat-tokenizer-ref,0) so the 30MB request parses as the few MB of text it actually is.
  • We read the width and height straight out of the PNG/GIF/WebP/JPEG header while it's still base64, never decoding a single pixel (and apply EXIF orientation where the engine does).
  • The model's own resize rules (Deepseek v4.1, Kimi K3, Qwen-VL, etc.) turn that size into the exact number of image tokens the engine would have produced.

That took us from 22.5 to 167 req/s on million token requests, and a single request on one core dropped from 240ms to 28ms. For comparison, vLLM and SGLang’s own preprocessing does 1.3 and 1.6 req/s on the same requests, roughly 2-3 seconds each.

Only parse the JSON you need

The other thing that surprised us was how much time went into just reading a request to then throw most of it away;

  • serde_json checks UTF-8 on every string with a scalar loop, which was about a tenth of a whole chat request. We do one SIMD UTF-8 check over the entire body up front and then parse it as an already validated string.
  • Chat requests are mostly tiny objects (a message, a content part, a tool call) with a handful of keys. Hashing every key is overkill so for anything with 8 or fewer keys we just scan them.
  • We only pull out the fields the engine reads, and apply the engine's own request rules (like how vLLM pre-parses tool call arguments) in Rust, so the body is never parsed twice or re-serialized.
  • Text going into toks gets the same SIMD UTF-8 check. The standard library ran at ~3GB/s on CJK heavy prompts which was a fifth of toks' entire encode time.

Not everything worked. We tried parallel JSON parsing and parallel template rendering and ripped both out. Waking up threads cost more than the work they saved.

While Under load

Benchmarking throughput is nice, and you can always make your numbers look many multiples higher than they will actually be in production. As part of building our library we tested how the long tail grew when under load asking ourselves, as a serving instance gets busier how slow are the slow requests?

From 8K to 160K token prompts, on both models, our p99 is lower than the SGLang gateway's median. On Deepseek at ~37K tokens our p99 is 2.6ms while their p50 is 2.9ms (and their p99 is 15.9ms). This is thanks to NIFs being run on dirty CPU schedulers, a million token request never stalls the rest of the BEAM, and the router can keep routing while the tokenizer is chewing through large or many requests.

How does this compare to the engines?

We took vLLM and SGLang's own request preprocessing (applying the chat template, parsing the body and then producing tokens), and benchmarked it in-process against ours.

We're 53-181x faster across every size on both models. Now that we’re sending raw token IDs instead of raw JSON to the inference engines, we wanted to evaluate the impact this has on the inference engines.

Sending IDs takes SGLang from 31 to 1,031 req/s (33x) and vLLM from 43 to 505 req/s (12x). With raw chat each API server process flattens out at about one core, because tokenization runs in Python under that process's GIL. Running 4 processes only takes vLLM from 13 to 43 req/s on an 8 vCPU box, while sending our ids gets the same 4 processes to 505. This is the hot path change from the start of the post, we tokenize once in the router and then inference engine never has to do it again.

Tokenization Accuracy

A core requirement for our tokenizer is identical output with engines. If the token IDs we send don't exactly match what the engine would have produced from the raw chat, the model is seeing a different prompt to the one the customer sent which could cause the model to behave in strange ways - when we dug into this most tokenizers outside of the engines themselves aren’t!

The tricky part is that tokenization isn't just running text through a tokenizer. Each engine renders the chat template and applies its own request rules first; each engine has its own weird quirks which produce slightly different tokens. vLLM for example parses tool call arguments and re-serializes them, so the JSON the model sees isn't byte for byte what the client sent. This is why we tokenize differently based on the engine we’re sending the request to ensuring that the tokens are correct for the engine.

On every benchmark we also compared the tokens against vLLM's and on 100% of requests, at every prompt size, on both models our outputs matched 1-1. The others perform well on short chats but drift once requests have tool calls and longer conversations in them. At ~8K tokens the SGLang gateway only matched 16% of requests (which was expected as SGLang renders the template differently from vLLM), NVIDIA Dynamo 73% and HF transformers 96%. At ~1M tokens with images every other library matched 0%, and on Kimi K3.1 HF transformers renders the wrong prompt entirely because it ignores reasoning_effort.

Additional Tokenizer Comparisons

Baseten wrote a blog post on how they sped up Kimi tokenization with their library, basetenkenizer, so we wanted to compare both libraries on Kimi K3.1. Their Rust tokenizer can't render Kimi chat templates so their production path renders in Python and then encodes in Rust. To keep it fair we compared just the encode step on their own API, with both libraries encoding the exact same pre-rendered Kimi K3.1 segments.

With 8 workers we're 1.9-5.7x faster than their fastest setting from ~200 to ~1.1M tokens.

Agentic workloads and caching

A large percentage of our traffic is agentic, which resend the whole conversation every turn. So we replayed 8 agent conversations of 213 turns each, anywhere from 2K to 160K tokens per turn.

The segment cache is content addressed and shared across every BEAM process, so once a chunk of conversation has been encoded it never gets encoded again. After about turn 30, 98-99% of tokens come straight from the cache. The gain is "only" 1.2-1.8x because even a fully cached turn still has to parse the request and render the whole conversation, and that grows every single turn which is still a bottleneck we want to solve!

What’s next?

Sending token IDs took SGLang from 31 to 1,031 req/s and vLLM from 43 to 505 req/s on long prompts, and ChatTokenizer is 53-181x faster than the engines' own preprocessing while still matching their IDs exactly.

This is the first post in a series, and next we want to write about running 10,000+ concurrent websockets to our GPU workers from a handful of Elixir nodes, how we route inference requests across engines and pools, and what we've learned about getting more prefix cache hits.

If this kind of work sounds fun, we're hiring across engineering. Come work with us! (if there isn’t a perfect fit please email or DM me)

Appendix 1: Benchmark Methodology

All the benchmarks ran on CoreWeave, one Xeon 8562Y+ node with 8 exclusive vCPUs per run. Every library ran on the same node one after the other with cold caches, and each number is the median of two 10 second runs after a 3 second warm-up. The requests come from WildChat, glaive function-calling and SWE-agent trajectories at pinned commits, bucketed by prompt size (XS ~140 tokens, S ~1.4K, M ~8K, L ~37K, XL ~160K, XXL ~1M plus 8 images).

Versions: vLLM 21d9da46b3, SGLang 31e55a5120, basetenkenizer 0.2.9, dynamo-llm 1.6.2, Kimi K3.1 Pro at 6d0a020.

  • Each graph is Deepseek v4.1 or Kimi K3.1 as labelled, with 8 vCPUs.
  • ChatTokenizer, the SGLang gateway, Baseten (basetenkenizer 0.2.9), NVIDIA Dynamo and HF transformers ran one after another on the same machine.
  • vLLM and SGLang were measured on their own in-process request preprocessing in a separate session on a different CoreWeave region.
  • vllm-rs has no in-process mode so it was measured over HTTP at 256 concurrent clients, which includes HTTP overhead the others don't pay.
  • Ours matched vLLM's IDs on 100% of requests. Exceptions for the others: the SGLang gateway matched 98% at ~140 tokens, 87% at ~1.4K, 16% at ~8K and 0% at ~1M (it skips image expansion); Baseten and HF transformers 96% at ~8K; NVIDIA Dynamo 73% at ~8K. Everything else matched 100%. Mismatched requests still count towards their throughput.
  • On Kimi K3.1, "Baseten" is their production K3 path (the model's Python renderer plus basetenkenizer's encoder) since the Rust library alone can't render Kimi chats. NVIDIA Dynamo can't load Kimi K3.1 and HF transformers renders the wrong prompt for it (it ignores reasoning_effort).
  • HF transformers means Python apply_chat_template followed by tokenization.
  • All graphs use linear scales. Where values span several orders of magnitude, each prompt size (or engine) gets its own axis, so compare bar heights within a panel and not across panels.

Appendix 2: vLLM and SGLang tokenization speedups

The 120x in the title is Deepseek v4.1 at ~1M tokens with 8 inline images, where our tokenizer does 162 req/s compared to vLLM's own preprocessing does 1.35 req/s. Below is every prompt size on both models so you can see the full range. All numbers are req/s with 8 vCPUs, in-process, using vLLM and SGLang's own request preprocessing (chat template rendering plus tokenization) at the versions listed above. Across every size we're 61-181x faster than vLLM and 53-112x faster than SGLang, with most sizes landing around 90-120x.

← All articles

What this article says

More in AI & Machine Learning

All →
Context API extracts full context from any web page
Perplexity

Context API extracts full context from any web page

From High-Level Diplomacy to Urban Public Spaces, SenseTime Showcases AI Innovations During APEC Economic Leaders' Meeting Week
SenseTime

From High-Level Diplomacy to Urban Public Spaces, SenseTime Showcases AI Innovations During APEC Economic Leaders' Meeting Week

How to use AI data enrichment and verify the results
Perplexity

How to use AI data enrichment and verify the results

Gemini 4 Neon vs Claude Haiku 5.5: The Ends of One Ladder
OrcaRouter

Gemini 4 Neon vs Claude Haiku 5.5: The Ends of One Ladder

Meet bison-1 and groundhog-1, frontier American decision models
Baseten

Meet bison-1 and groundhog-1, frontier American decision models

Hy4 flash Vision: 74.8 MMMU-Pro, and Why 52.0 Still Stands
OrcaRouter

Hy4 flash Vision: 74.8 MMMU-Pro, and Why 52.0 Still Stands

More from Inference.net

MiniMax M3.1 Flash is live on Inference: Day 0
Inference.net

MiniMax M3.1 Flash is live on Inference: Day 0

Specialized LLMs: The model you need doesn't exist yet
Inference.net

Specialized LLMs: The model you need doesn't exist yet

How Inference.net trains Specialized Language Models that cut AI costs by up to 50x
Inference.net

How Inference.net trains Specialized Language Models that cut AI costs by up to 50x

Introducing Inference platform: Monitor, train, and deploy self-improving AI models
Inference.net

Introducing Inference platform: Monitor, train, and deploy self-improving AI models

Schematron V2: Frontier HTML-to-JSON extraction at a fraction of the cost
Inference.net

Schematron V2: Frontier HTML-to-JSON extraction at a fraction of the cost

Introducing AutoEvals: Automatically find the best model for your task
Inference.net

Introducing AutoEvals: Automatically find the best model for your task