Skip to content
Why you're using Jev wrong: four fixes from TypeSafe's docs
Back to Blog
#AI models#Jev#Jev limitations#jev-latest#TypeSafe AI#Jev confidence threshold

Why you're using Jev wrong: four fixes from TypeSafe's docs

Zohaib Masood

Most bad Jev results come from treating it like a chat model: thresholds tuned on a moving alias, broad questions over too much context, arithmetic left to the model. TypeSafe's own documentation gives the fixes. Pin a model version, keep counting, dates and maths in code, ask one literal question per judgment, and set a threshold for each action according to what a wrong answer costs.

Everything below is from TypeSafe's docs and launch post, read on 1 November 2026.

You are calling jev-latest with thresholds you tuned last month

If you tuned your thresholds on jev-latest in September or early October, you tuned them on a different model from the one now answering. TypeSafe released Jev 1.14 on 14 October and said jev-latest would move to it a week later. The models page now shows the alias pointing to jev-1.14.0. TypeSafe calls 1.14 "small, subtle improvements to everything", and warns:

An alias moves when a new release ships, so its answers can change without any change to your code. — TypeSafe AI, Models

The fix is to send a versioned ID, jev-1.13.0 or jev-1.14.0, wherever a threshold depends on the score. Log the model field each response returns. Re-run your labelled cases on 1.14 before you switch. TypeSafe intends to give 60 days' notice before retiring jev-1.13.0. Promptfoo's Jev integration gives the same advice for graders: pin a version when you tune a threshold against it.

You are asking it to calculate

Jev is not a calculator. TypeSafe's list of known failure modes for 1.13 says it "does not count reliably", reads dates "as text, not as ordered quantities", and judges hex colours and other numeric codes worse than plain names. The score levels are also "weak in numerical calibration", so you cannot interpolate an exact figure between two levels.

Split the work. The model makes the judgment and your code does the arithmetic:

  • Counting: loop over the items in code, ask one yes/no question (a Noul) per item, and add up the answers yourself.
  • Dates: extract the day, month and year as a Choice over fixed options, with a "not stated" option. Then build the date and compare it in code.
  • Numbers: convert in code and send the result or a named bucket. Keep the model for the part that is a judgment, such as whether a value is a warning.

Your question is not the one you meant

Jev "answers the question you wrote, not the one you meant". Scoping words and negations are read at face value. Four habits make this worse:

  • Several judgments in one question. Split them into separate questions and combine the answers in code.
  • Criteria that pull against the instruction. TypeSafe's example is a yes/no question where true maps to no. Write instructions and criteria that an ordinary reader would agree on.
  • Option order. In a Choice, 1.13 can lean towards the option listed first. Reorder the options on a test set and check that the answer holds.
  • Too much state. Accuracy falls as unrelated text grows. Filter and retrieve in code, and send only the fields the question needs. The TypeSafe-Danger-Zone-Long-Context header raises the limit to 262k tokens, but the docs say it is "slower and less accurate". The launch post says the danger-zone features, images included, have "too much jaggedness" for TypeSafe to vouch for in production.

You act on every answer the same way

A single cut-off for every action is the wrong standard. Gate each action on what a wrong answer costs. The confidence field on Choice and Score answers shows how concentrated the probabilities are. It is not the chance that the answer is right. A yes/no question (a Noul) has no confidence field. Read its probability directly.

TypeSafe suggests three paths: act automatically when confidence is high, ask for confirmation when it is middling, and send low-confidence cases to a person. Its starting point for a Noul is 0.7 for yes and 0.3 for no, with values in between going to review. Raise the yes threshold when a false yes is expensive, such as issuing a refund. Lower it when missing a true yes is expensive, such as a safety flag. TypeSafe warns against fitting thresholds closely to a handful of test cases, so tune them on a larger set of your own labelled examples, run against the pinned version.

The work is in the wiring around Jev: pinned versions, arithmetic in code, and a review path for uncertain answers. If you want that built into your product or workflow, talk to Vesprr about it, or see what we build.

More on Jev: What is Jev? · Jev pricing · Cloudflare Clef, a Jev-compatible alternative

Sources