~/code.benoitguivarch.fr / posts/2026-09-23-jev-checks-if-agents-are-done.md

The first job I'm giving Jev is checking whether my agents are done

2026-09-23 · 666 words

I run several coding agents in parallel. What slows me down is not their work but checking it: is this one finished, stuck, waiting on me, or saying “done” when it is not? These are small questions with a handful of possible answers, and I ask them dozens of times a day.

TypeSafe released a model built for that kind of question on 15 September 2026.1 Jev does not write text: it takes some state and a list of typed questions, and returns an answer to each with a probability.1 There are three question types: pick one of N options, rate on an ordered scale, or yes/no.2 TypeSafe announces responses in 70 to 500 ms and $0.042 per million input tokens, with output free.1

What the numbers do not tell me

Every figure so far comes from TypeSafe, and nobody outside has tested it yet.3 In TypeSafe’s benchmark the reference answer is the average of two frontier LLMs, and TypeSafe’s own team wrote the workflows.2 On one comparison Jev matched the LLM it replaced, 67.8% against 67.9%, rather than beating it.4 The claim is equal quality at a fraction of the time and cost, measured against other models.

So my question is narrower: does it agree with me, on my agents, often enough to act on?

Where it fits in supervising agents

The founder argues for splitting AI work into small decisions that can be checked one by one, instead of one large prompt.5 Supervising agents already has that shape:

All five come up often, have bounded answers and small inputs, and a wrong answer costs me a second look, not a broken release.

The first one I test

I start with “done, really”. In my setup it needs nothing new to trigger: both agents I use already run a hook when they stop. And the correct answer shows up on its own: either I accept the work, or I send the agent back. A community plugin already checks “done” claims with Jev,6 so the idea is not exotic; I want the numbers on my own work.

It runs in shadow mode. At each stop, a hook saves the task, the agent’s last message and the diff, then Jev’s answer, then what I did next. After two weeks I compare Jev’s answers with my own decisions, repo by repo, and set the threshold from that data rather than from a benchmark. Each call carries its whole state,2 so I can save cases before I have access and replay them once I do.

If agreement is high, the next step is letting it send an agent back on its own when it answers “says done but is not”. If it is low, the logged cases will show where it goes wrong.


  1. Diogo Almeida, “Introducing System One Models & Jev”, TypeSafe AI Blog, 2026-09-15. https://typesafe.ai/blog/introducing-system-one-models-and-jev ↩↩↩

  2. MarkTechPost, “TypeSafe AI Releases Jev: A System One Model That Returns Typed, Calibrated Decisions Instead of Text”, MarkTechPost, 2026-09-19. https://www.marktechpost.com/2026/09/19/typesafe-ai-releases-jev/ ↩↩↩

  3. MindStudio, “RLCD vs RLHF: What Is Typesafe’s Jev Model Actually Claiming?”, MindStudio Blog, 2026-09. https://www.mindstudio.ai/blog/typesafe-jev-rlcd-vs-rlhf ↩

  4. Anthony Maio, “Jev: The Language Model That Won’t Talk”, Substack, 2026-09. https://anthonymaio.substack.com/p/jev-the-language-model-that-wont ↩

  5. Latent Space, “Jev: System One models for Prod, not God — with Diogo Almeida, CEO, TypeSafe AI”, Latent Space, 2026-09. https://www.latent.space/p/jev ↩

  6. APIMaster.AI, “How to Use Jev in Claude Code and Codex: Four Routes That Work”, APIMaster.AI Blog, 2026-09. https://apimaster.ai/blog/jev-claude-code-codex ↩