Why We Built SystemOne.dev, and an Open System One Model

Sat, 03 Oct 2026 13:00:00 GMT

Something has been quietly true for a while now: a large share of what teams use large language models for isn’t generation. It’s classification, routing, scoring and gating: decisions dressed up as text.

We noticed it the way most people do, by writing the same defensive code for the fifth time:

res = llm.chat(messages=[...])                    # "ONLY RETURN VALID JSON..."
try:
    parsed = json.loads(strip_fences(res.text))
except json.JSONDecodeError:
    parsed = retry_with_stricter_prompt(text)
if parsed.get("category") not in ALLOWED:
    parsed = {"category": "unknown", "confidence": 0}

None of that is business logic. All of it exists because we asked a text generator a multiple-choice question and got back a paragraph.

The gap

System One models fix that problem directly: they answer typed questions about your program’s state with a probability for every option, in tens of milliseconds, without generating text. The architecture is real and it works.

Two things were missing.

The engineering literature. Search for how to pick a threshold and you find “0.95 is a good starting point”. Search for what calibration drift looks like in production and you find research papers. Every team is working it out alone, and making the same four mistakes.

Open models. The best-known System One model was a hosted service. You couldn’t run it on your own data, inspect how it was trained, or change it.

What we’re launching

SystemOne.dev: the patterns, written down once, with code that runs.
Kenning: an open System One model under Apache-2.0. It’s 435M parameters, runs on a ~2 GB GPU footprint or a CPU, and its training data and licences are documented.
SystemOne Builder: an open toolkit to serve, train, distil and benchmark your own System One models on one GPU, from a dashboard.
systemone-client: one Python client for Kenning, Cloudflare’s Clef and TypeSafe’s Jev. Switch engines by changing a URL.

Two commitments

We’ll stay honest about limits. Kenning v0.4 is faster and much smaller than the big engines, and behind them on subtle phishing. We publish those numbers, not just the flattering ones. “No hallucination” gets a page that spends as much space on what doesn’t disappear as on what does. Calibration is something you verify on your own data, so every page tells you how.

We’ll stay open and engine-neutral. The code, the weights, the benchmark suites and this site are all open. Every engine that speaks the format is welcome here, and pages about other engines are welcome contributions.

There’s a version of this site that’s more exciting to read and less useful to build on. We’re not going to write that one.

Start here

Want code running → Quickstart
New to the idea → System One vs. System Two
Not sure it fits → Is System One right for my problem?
Want to help → Contributing

If you’re already running a decision model in production, we especially want to hear from you. Real numbers on real workloads are the scarcest thing in this whole space.

The Four Mistakes Every System One Codebase Makes First

Sat, 03 Oct 2026 12:00:00 GMT

Four bugs show up over and over in System One integrations. They aren’t subtle once you know to look, and all four are cheap to fix at the start and painful to fix once six months of data has been written under them.

1. Folding uncertainty into the wrong branch

The most common, and the only one on this list that’s a straightforward correctness bug:

# The bug.
if r.nouls["fraud"].noul > 0.95:
    block_transaction()
else:
    allow_transaction()

A fraud probability of 0.94, the model saying “this is very probably fraud”, falls through to allow_transaction().

The else branch has silently become a bucket for two opposite situations: “confidently fine” and “alarmingly uncertain”. In your logs, the second is now indistinguishable from the first.

# The fix: uncertainty is its own outcome, with a threshold on each side.
p = r.nouls["fraud"].noul
if p >= 0.95:
    block_transaction()
elif p <= 0.02:
    allow_transaction()
else:
    manual_review(transaction, p)

Rule: a yes/no answer needs two thresholds, and the middle is a real outcome. More in the fuzzy if-statement.

A close cousin: gating on a choice’s confidence field. In the System One format, confidence measures how far the winner stands above an even guess. It isn’t the winner’s probability. Gate on probabilities[choice] (why).

2. Thresholds chosen because they sound responsible

0.95 appears in almost every codebase, and almost nobody can say why. It isn’t derived from anything. It just sounds careful.

The actual arithmetic is one line:

automate when   p > cost_of_error / (value_of_automating + cost_of_error)

Which gives very different numbers depending on what the decision does:

Action	Cost of being wrong	Threshold
Route a ticket to the wrong queue	1	0.50
Auto-junk an email	20	0.95
Block an IP	100	0.99
Delete user content	500	0.998

Two useful consequences. First, plenty of low-stakes decisions should be automated at 0.6: teams leave easy wins on the table with a blanket 0.95. Second, if the arithmetic demands 0.998 and your model rarely gets there, that decision isn’t automatable yet, and knowing that is worth more than a threshold that pretends otherwise.

3. No escape option

Choice("Which team?", {"billing": None, "technical": None, "account": None})

The model has to return one of these. When a message is none of them (a legal threat, a partnership enquiry, a language you don’t support) it returns the nearest one. Often with high probability, because relative to the other two options it really is the best fit.

Your probability safety net doesn’t catch this. The number is high. The answer is wrong.

Choice("Which team?", {"billing": None, "technical": None, "account": None,
                       "other": "Anything that fits none of the above"})

Then alert on other as a share of traffic. A rising other rate is the earliest signal that your options have drifted out of date: it moves weeks before accuracy visibly degrades.

4. Monitoring uptime instead of the distribution

Almost everyone monitors request count, error rate and p99 latency. Almost nobody monitors the thing that actually fails.

System One models degrade silently. Calibration drifts when your inputs shift: a new market, a new product surface, an adversary adapting. The model keeps returning well-formed answers with plausible probabilities. Your dashboards stay green. The answers get worse.

Four signals worth alerting on:

metrics.histogram("decision.probability", p, tags={"choice": d.choice})
metrics.increment("decision.choice", tags={"choice": d.choice})
metrics.increment("decision.below_floor")
metrics.increment("decision.degraded")        # fallbacks taken

Signal	What a move means
Winning probability falling	Your inputs shifted: re-check calibration
Answer mix shifting	Either the world changed, or your state builder did
`below_floor` rate rising	Real inputs your options don’t cover
`degraded` rate above zero	You’re silently running on fallbacks

That last one deserves its own alert. A timeout fallback that quietly allows every request, while the error rate stays at zero because you handled the exception, is the failure that does the most damage before anyone notices.

Beyond metrics, sample and read the decisions: fifty a week, by hand. Every team that has been burned here says the same thing afterwards: the dashboards looked fine.

The pattern behind all four

Each of these is the same mistake in a different costume: treating the probability as decoration instead of as the primary output.

If you take one thing away, make it this. The chosen option is just the most likely one. The probabilities behind it are what make the model safe to build on. Code that ignores them has thrown away the only thing that separates a decision model from a very fast guess.

Further reading: Understanding calibrated confidence and The fuzzy if-statement.

SystemOne.dev | Blog