If you search code, you’ve probably asked this: is a code-specific embedding model worth it, or is a strong general one good enough?
Papr’s papr-embed-code-1 is a good test case. It’s Qwen3-Embedding trained further on code retrieval, with open weights in two sizes: 0.6B and 4B. Papr reports top scores on CoIR, a code retrieval benchmark. But it trained on most of CoIR’s training data, and its model card says so. Papr also reports the 4B at #1 on MTEB’s two held-out code leaderboards, RTEB (Code) and CodeRAG.
We asked a narrower question. Does it beat the model it started from on code it never saw? And what does it cost to run in Qdrant?
How We Tested
We compared each papr-embed-code-1 model with the Qwen3-Embedding model it was trained from, with the same query instructions. The tasks come in three groups:
- Code Papr didn’t train on: HumanEval and MBPP match a task description to the Python function that solves it. FreshStack matches a Stack Overflow question to the files of the library it’s about. WikiSQL matches a question about a table to its SQL query.
- Code Papr trained on: three CoIR tasks, though the 4B skipped CodeTransOcean Contest. APPS matches a programming problem to its solution. CodeTransOcean Contest matches a program to the same program in another language. StackOverflowQA matches a question to its answer.
- Prose: four BEIR datasets, NFCorpus, ArguAna, SCIDOCS, and FiQA.
Where Papr’s model cards score its own models, ours match to within 0.001 nDCG@10 on every task except the ones full of identical documents, marked † in the full results. There, the score depends on how ties break, which can change with batch size. On SyntheticText2SQL, we got 0.625 and 0.650 for the two sizes, against Papr’s 0.720 and 0.723.
Better on Code It Never Saw
On code Papr didn’t train on, both sizes beat their base model. The 4B gained 0.034 nDCG@10 on average, and the 0.6B 0.026. On the CoIR tasks, the gains were similar.
Code Search Gains, Prose LossesExact search, nDCG@10. Higher is better.nDCG@10Unseen codeCoIRProse0.00.20.40.60.81.0Qwen3 0.6BPapr 0.6BQwen3 4BPapr 4B
The gain held on every one of those tasks, but its size varied a lot. On HumanEval, the 4B base model already scored 0.98, so there wasn’t much left to gain. On WikiSQL, the 4B gained 0.070.
Prose went the other way. Both sizes scored a little lower than their base model, by 0.003 for the 0.6B and 0.016 for the 4B. On TREC-COVID, a set of COVID-19 research papers, the 0.6B lost 0.065. If most of what you search is prose, keep the base model.
Compressed to 1 Bit, the 4B Beats the 0.6B
Code search usually means a lot of vectors: a function, a file chunk, or a snippet each. The 4B’s vectors have 2,560 dimensions, which is 10 KB per vector in float32.
They compress very well. With 1-bit TurboQuant, each vector takes 328 bytes, 31x less. On our seven code tasks, 1-bit TurboQuant kept 99.7% of the 4B’s float32 score without rescoring, and all of it with rescoring at 2x oversampling.
That changes which model to pick. The 4B compressed to 1 bit scores higher than the 0.6B in float32, and takes 12x less RAM.
RAM per VectorLower is better.05,00010,0000.6Bfloat324096 B0.6B1-bit136 B4Bfloat3210240 B4B1-bit328 BBytes per vector
The 4B still costs more to run, because it embeds about four times slower than the 0.6B on the same GPU. And WikiSQL is the exception, where the 0.6B scored higher.
A Million Snippets in One Collection
Our other code tasks have at most 20,000 documents, which makes compression look easy. So we also loaded CodeSearchNet-CC into one collection: about a million snippets in six languages, from Python to Ruby. Each query is the start of a function, and the right document is the rest of it. A payload filter keeps the search within the query’s language.
Compression held up at this size. At hnsw_ef 128, 1-bit TurboQuant with rescoring kept 99.5% of exact search’s score, against 99.7% for float32. It was faster too, at 1.0 ms per query against 1.8 ms, with the original vectors for rescoring already in memory.
The bigger lever was hnsw_ef, the number of candidates the search explores. Raising it from 128 to 256 added 0.002 to 0.004 nDCG@10 to every setup. That’s enough for rescored 1-bit vectors at 256 to beat float32 at 128. At 256, float32, 2-bit, and rescored 1-bit vectors all kept 99.8% or more, and the 1-bit search took 1.4 ms against 2.9 ms for float32.
Dropping the filter and searching all six languages at once cost 0.005 to 0.007.
What About BM25?
Code is full of exact names, so keyword search seems like a natural partner. Qdrant 1.20 added a code tokenizer for full-text indexes. It splits getUserName and get_user_name into the same words. Qdrant can also score full-text matches with BM25, so a hybrid query can fuse BM25 with the dense search.
On our tasks, BM25 on its own was far behind dense search. On APPS, where the queries are problem statements and the documents are bare solutions, it scored 0.01 nDCG@10. The two share almost no words.
Fusing BM25 with dense search at equal weight made every task worse, by 0.02 to 0.17 for the 4B. Down-weighting BM25 helped, but only up to break-even. With BM25’s RRF weight at 0.1, the 4B averaged 0.0006 below dense search alone, and its best gain on any task was 0.002.
Our queries were questions, descriptions, and code, and for those, dense search alone was the better choice.
So, Does It Pay Off?
For code search, yes. On code it never saw, papr-embed-code-1 beat the model it started from at both sizes, by about 0.03 nDCG@10. On prose it did a little worse, so if most of your corpus is prose, stay with the base model.
The bigger choice is the size. Compressed to 1 bit, the 4B scored higher than the 0.6B in float32, in a twelfth of the RAM. If you can afford to embed with the 4B, we’d store it with 1-bit TurboQuant, keep rescoring on, and raise hnsw_ef before adding bits. Compare against an unquantized collection on your own queries, once with rescoring disabled and once with 2x oversampling. If you index several languages in one collection, filter by language.
We’d leave BM25 out to start. If your users search for exact identifiers, that’s the case to test it on, with a low weight.
The 0.6B is coming to FastEmbed in its next release.
Full Results
Every model got the same query instructions. † Many identical documents, so the score depends on how ties break. Left out of the averages. ‡ Papr trained on its train split. § The 4B didn’t train on it.
Mean over seven code tasks (HumanEval, MBPP, FreshStack, WikiSQL, APPS, CodeTransOcean Contest, and StackOverflowQA), searched with HNSW. Rescoring uses 2x oversampling. Bytes count only the vectors searched in RAM.
papr-embed-code-1-0.6b on CodeSearchNet-CC, 1,005,474 snippets, each query filtered to its language. Exact search scores 0.923. Latency is Qdrant’s server-side time.
papr-embed-code-1-4b. BM25 uses the code tokenizer. RRF uses Qdrant’s default k.
Thanks to Papr for releasing the code models with open weights, and for publishing exactly how they trained and evaluated them.












