Open-Source Model Releases Worth Actually Trying

Any list of specific open-source models is wrong within three months. The release cadence is roughly weekly across the major labs, and a post naming the best 7B model in July is naming a historical artifact by October. So this is not a list. It is the filter, which has a much longer half-life than its outputs.

The filter has five dimensions. None of them is benchmark score, and the omission is deliberate.

1. The license, read to the end

"Open source" and "open weights" are not synonyms and the difference will eventually be a legal conversation. Several of the most widely deployed model families ship under bespoke licenses containing things no OSI-recognized license contains: monthly-active-user thresholds above which you must negotiate separately, acceptable-use policies incorporated by reference and amendable by the licensor at will, naming and attribution requirements on derivative models, and clauses terminating your rights if you initiate patent litigation.

Three things to grep for specifically. Whether the license is revocable, since a revocable grant means the weights you built a product on can become unlicensed by unilateral action. Whether the acceptable-use policy lives at a URL rather than in the license text, which means the terms can change without you agreeing to anything. And whether training-on-outputs is restricted, because if you plan to distill, a clause forbidding use of outputs to improve other models is the whole ballgame.

Apache 2.0 and MIT weights exist and are common. When they are available, the license dimension costs you nothing.

2. Tool-calling and structured output reliability

This is where most models die for production use, and it correlates weakly with reasoning benchmark scores. A model can score respectably on graduate-level reasoning evaluations and fail to emit parseable JSON on the fifteenth call out of a hundred.

The failure modes are specific and repetitive: trailing commas, a fenced code block wrapped around the JSON when you asked for raw JSON, a plausible-looking function name that was not in the tool list, correct schema with a string where the schema said integer, and single-object output when the schema specified an array of one. Each of these is a 500 in your pipeline.

Here is the claim worth testing yourself, because it changes the architecture: constrained decoding makes the syntax question disappear entirely and does nothing for the semantic one. A grammar-constrained sampler (llama.cpp GBNF grammars, or the structured-output backends in vLLM) masks the logits at each step so tokens that would violate the grammar have zero probability. Invalid JSON becomes impossible, not unlikely, and parse-failure rate goes to exactly zero. The model can still pick the wrong tool, hallucinate an argument value, or put a semantically absurd integer in a valid integer field. So: constrained decoding is mandatory infrastructure and is not a substitute for evaluating tool selection accuracy.

3. Usable context versus advertised context

A 128K advertised window does not mean 128K of usable attention. Effective retrieval degrades well before the stated limit, and the degradation is position-dependent rather than uniform: information in the middle of a long context is recovered less reliably than information at either end. Long-context claims are also frequently produced by position-interpolation methods applied after pretraining on much shorter sequences, which extends the addressable range faster than it extends the model's ability to use it.

Test it in twenty minutes. Build documents at 8K, 32K, 64K, and the advertised maximum, each containing one factual needle. Place the needle at 10%, 50%, and 90% depth. Ask for it. You are looking for the length at which the 50% position starts failing, and that number, not the one on the model card, is your architectural budget. For a memory or retrieval system this directly determines how many memories you can inject before recall gets unreliable, which is a design input you cannot get from a spec sheet.

4. Quantization and tooling support

A model with no GGUF conversion and no vLLM support is a research artifact, regardless of quality. The practical questions:

• Does the architecture have a merged implementation in llama.cpp and in vLLM? A novel attention variant or an unusual MoE routing scheme can mean months of lag, and during that lag you are running the reference PyTorch implementation at a fraction of the throughput.
• Do quantizations exist at the sizes you can actually serve? Q4_K_M is the usual sweet spot for local deployment; below Q4 quality falls off noticeably and the drop is worse for smaller base models, which have less redundancy to give up.
• Does it fit? Rough rule for a dense model: parameter count times bytes-per-parameter, plus KV cache. A 7B at Q4 is around 4 GB of weights. KV cache is roughly 2 * layers * kv_heads * head_dim * seq_len * bytes, which at long context can exceed the weights themselves. Grouped-query attention shrinks this substantially, so check the KV head count, not just the parameter count.

5. Documented tokenizer and chat template

The lowest-glamour dimension and the one that produces the most confusing failures. If the chat template is undocumented and you guess the role markers or the BOS handling, the model will still answer. It will just answer noticeably worse, and there is no error to trace. Multi-turn formatting, whether a system role is supported at all, and how tool results are represented are all template-level details that silently change quality.

Look for a chat_template in tokenizer_config.json. If it is present and your runtime applies it, this dimension is handled. If the release ships a template only in a blog post code snippet, expect an afternoon of debugging output that is subtly off.

Why leaderboard rank barely matters

Public benchmark rank is a poor predictor of deployment fit for three structural reasons. Benchmark contamination is pervasive and unmeasurable from outside, since you cannot audit a training set you cannot see. Aggregate scores average across tasks you do not care about, so a model that wins overall may be losing on the one capability your product depends on. And rank compression is severe: the gap between rank 3 and rank 12 is often a couple of points on a metric with a wider confidence interval than the gap, while the gap in tool-calling reliability between those same two models can be 20 percentage points and appears on no leaderboard at all.

The 30-minute protocol

Run this before reading a single benchmark table. Ten minutes of setup, twenty of measurement.

• Minutes 0-3.Open the license. Search for "revoke," "monthly active," "acceptable use," and "derivative." If any clause is a dealbreaker for your deployment, stop. You just saved 27 minutes.
• Minutes 3-6. Confirm architecture support in your serving stack and that a quantization exists at your memory budget. Check tokenizer_config.json for a chat template.
• Minutes 6-16. Tool-calling harness. Twenty prompts against three real tool schemas from your product, five samples each, temperature at whatever you run in production. Score two numbers separately: parse rate and correct-tool rate. Below 95% parse without constrained decoding, or below 90% correct-tool with it, means significant fallback engineering.
• Minutes 16-24. The needle test above. Record the length at which mid-context retrieval degrades.
• Minutes 24-30. Throughput on your hardware, not theirs. Tokens per second at your batch size and your context length. Also record time-to-first-token, because for interactive use TTFT is what users perceive and it is not implied by throughput.

Write the five numbers down: license verdict, parse rate, correct-tool rate, usable context, tokens per second. That is a deployment decision. A leaderboard rank is not.

As an aside from running this filter repeatedly: the tool-calling dimension is why Unimatrix stayed protocol-level rather than model-specific, since a memory layer that any client can call outlives whichever model won this quarter. Pricing across tiers is on the pricing page.