Jev: not every decision needs a conversation

Some calls to a language model have a useful result that fits in one word: billing, support, sales. Or in a binary answer. Or in one level of a rubric.

Yet we often treat them as small conversations: we write instructions, ask for a format, receive text, and turn it into a signal the program can use. Structured outputs have reduced much of that work. But an earlier question remains: did we need a conversational model for that decision?

That is the interesting question behind Jev, TypeSafe AI’s model. Not whether it replaces the generalist agent, but which part of the agent’s work could stop passing through it.

Jev receives a state — text or structured data — and typed questions. It returns answers through three primitives: Choice, for selecting among options; Noul, for a binary question expressed as a value between zero and one; and Score, for an ordered rubric.

The response space is defined before the model is called. We are not asking for a paragraph to interpret later, but for a signal the program already knows how to consume.

TypeSafe presents this as a family of “System One” models and publishes large speed and cost advantages in its own workflows. Those are vendor results, with its tasks, comparators, and references. We have not reproduced them.

Nor should the proposal be confused with an exclusive property: an LLM with constrained generation can also produce outputs that respect a schema. What deserves evaluation here is the combination of interface, decision quality, latency, and cost.

We ran 192 experimental requests against the pinned version jev-1.13.0, plus one initial access check. All experimental requests returned successfully at the HTTP level and produced 604 typed answers, with no detected errors in shape, keys, types, or range.

The core of the test was twelve simple tickets, each in English and Spanish. Jev got all 24 routes and all 24 binary billing answers right.

Then we changed option order, added noise, introduced instructions that tried to divert the classification, and tested texts that claimed another system had already assigned the wrong category. We also included controls without an explicit defense against those instructions. We did not find an incorrect route classification in those variants.

This is a favorable observation, not a certification of robustness. The perturbations were in English; the translations, repetitions, and variants came from a small set. They are not hundreds of independent production examples. We also did not evaluate the subjective correctness of each urgency score.

In requests with one, five, and twenty short questions, median end-to-end latencies were 290, 278, and 299 milliseconds, respectively. Eight requests per configuration: enough to describe what happened, not enough to promise a latency tail or an SLA.

The 193 calls, including the initial one, would cost approximately USD 0.00454 at the published rate and according to the recorded tokens. This is an estimate of Jev usage, not an invoice or the cost of the whole investigation. We did not run our own LLM comparator, so we cannot convert that number into a savings multiplier.

One of the most useful examples was not a sophisticated attack. It was this message: “I have a problem.”

With an escape output, Jev chose other. When we changed the instructions to force it to choose among billing, technical support, and sales, it selected technical with confidence = 1.0.

It did not violate the instructions: we had ordered it to choose even if no category fit. Nor is this an isolated proof of miscalibration, because we changed the options and the instruction at the same time.

The lesson is a design lesson: a valid response and high confidence do not compensate for a contract that forces a decision without enough information.

If the system needs to be able to say “I do not have enough information,” that possibility must exist in the decision policy. It is not enough to expect the model to invent it inside a closed set that excludes it.

Here it is important to distinguish three things: the distribution returned by the model, the confidence field that summarizes a property of that distribution, and the frequency with which its decisions turn out to be correct.

They are not interchangeable.

External research by log·entropy found that Choice confidence is compatible with a rescaling of maximum probability that depends on the number of options. Our responses were also compatible with that relationship in a later exploratory check. That does not prove the internal implementation or turn the number into a probability of being right.

The most useful calibration evidence comes from evaluating predictions against labels. Anthus published a study with 8,801 constructed sentiment examples and a separation between calibration and test data. It reported that isotonic regression reduced Noul calibration error from 0.117 to 0.008 on its test set.

That is a promising result: there is a signal that can be used. It also has important limits. It is a single constructed domain, includes a stratum with deliberately arbitrary labels, and works with probabilities, not with the confidence field as if it were probability of being right. It does not establish a universal threshold for other systems.

A second pilot, with 300 examples, found that Jev outperformed GLiNER in news and banking, but achieved 48% accuracy in emotion classification, with worse probabilistic quality than the comparator. Its coverage thresholds were selected and evaluated on the same sample: they describe the pilot, not a validated policy for production.

The conclusion is not that Jev is reliable or unreliable in the abstract. It is that the task, the formulation, and the data are still part of the classifier we are evaluating.

We also observed differences that an integration could overlook.

The billing probability in Choice and the Noul about billing differed by as much as 0.33. Routes remained stable across repetitions, but some Scores changed by as much as 0.06. And sixteen Score answers differed by 0.01 from the expected value calculated with the returned probabilities: compatible with separate rounding, although we did not prove the cause.

We do not present those observations as type errors. Jev’s documentation warns that identities should not be expected between independently evaluated questions. The original numeric flags were preserved in the results.

The practical consequence is simple: if the program needs exact equality, a calculation, or a deterministic rule, it should execute them in code. Do not ask the model for several formulations and assume they will close algebraically.

Jev’s opportunity does not need the promise of replacing the complete agent.

A system can use it to propose a route, grade a text against a rubric, or detect a semantic condition. Then code can check deterministic requirements, decide when to abstain, and limit the allowed actions. Cases that require more context, tools, or interaction can escalate to another component.

That does not remove evaluation. It moves evaluation toward a more concrete unit: model version, question, options, data, threshold, and action policy. Changing one of those pieces may require measuring again.

Our results justify exploring that integration. They do not justify saying that Jev never makes mistakes, that it resists any adversarial instruction, or that its confidence is calibrated for any use.

Not every decision needs a conversation. Every decision that matters needs a suitable contract, evidence about its errors, and limits on its consequences.

The investigation included an initial panel of Gemini, DeepSeek, and Kimi through Command Code. Their objections changed the protocol. The later review of results was completed with GLM-5.3 and GLM-5.2 through ZAI when the initial provider exhausted its quota. These last two belong to the same family. The effective models were verified; agreement among agents was not used as proof.

The main protocol of 120 requests was fixed before the responses. A supplement of 72 was registered after the first run began, but before inspecting its results, to control for number of questions, presence of explicit defense, and misleading self-labeling. We did not add calls after observing the results in order to chase a failure. Requests were serial and had no retries.

The protocols, responses, analyzer, hashes, and debate corrections were preserved as local evidence of the investigation. The third-party measurements cited were reviewed documentarily, not replicated.