What I Actually Use Jev For

Photo by Андрей Сизов on Unsplash.

Someone asked me “what is Jev?” after my setup post, where it got one line as my judge model. I gave the short answer, but I realized it might be neat to write up my actual agent usage of Jev over time.

Every Jev call Oh My Pi makes writes a model_usage entry with a purpose field into ~/.omp/agent/sessions. I pulled all of them, from the night I set it up on September 30 through October 11.

What Jev is

Jev is TypeSafe’s first System One model. It doesn’t write text. You send it a state and typed questions (yes/no, pick one, score on a scale), and it returns probabilities your code can branch on. TypeSafe prices it at $42 per billion input tokens.

This was my smoke test that night, through OMP’s eval judge() helper:

const r = await judge("The sky is green.", { factual: { type: "bool", instructions: "Is this statement factually true?" } });
console.log(JSON.stringify(r));

{"factual":{"type":"bool","bool":0.05}}

A 5% chance the sky is green. That’s the whole interface.

Two weeks of calls

Almost all of it is find, OMP’s semantic code search. It’s based on jegrep, and it only turns on when your judge role points at Jev. A grep pass pre-ranks files, then Jev judges the top 128 by filename, excerpts from the best 20 files, and finally full passages. Every score you see is Jev’s probability that the passage matches. My 61 searches made 604 Jev requests, with a median of 757 ms per search.

What it was used for

1. Finding the email helper in my Django SaaS

The agent needed to send a transactional email and didn’t know where the existing helper lived. Here’s the result as it landed in the transcript:

2 hit(s) for "send email helper function transactional email to user" in apps/ (τ 0.20), strongest first

apps/teams/invitations.py  0.94  43 lines judged
  apps/teams/invitations.py:1-43  0.94  from django.conf import settings
apps/users/views.py  0.22  77 lines judged
  apps/users/views.py:1-77  0.22  from allauth.account.models import EmailAddress

listed 155 · judged 128 · read 20 files (11.8KB) · 5 requests · 16K tokens · $0.0007 · 675ms wall / 1.1s api

The real helper scored 0.94, and the near miss in users/views.py barely cleared the 0.20 bar.

2. Explaining an empty /metrics in Ankusa

I was adding a Rust SDK to Ankusa, my Elixir webhook receiver. The live smoke test got a 200 from /metrics with an empty body, so the agent asked find how that endpoint is wired and whether it needs a flag:

5 hit(s) for "how the /metrics endpoint on the operator admin listener is wired, and whether it needs a config flag or a metrics collector enabled" in packages/ankusa/lib/ankusa/ (τ 0.20), strongest first

packages/ankusa/lib/ankusa/admin/router.ex  0.89  211 lines judged, partial
  packages/ankusa/lib/ankusa/admin/router.ex:1-211  0.89  defmodule Ankusa.Admin.Router do
packages/ankusa/lib/ankusa/instance.ex  0.68  229 lines judged
  packages/ankusa/lib/ankusa/instance.ex:1-229  0.68  defmodule Ankusa.Instance do
packages/ankusa/lib/ankusa/metrics.ex  0.63  210 lines judged, partial
  packages/ankusa/lib/ankusa/metrics.ex:1-210  0.63  defmodule Ankusa.Metrics do
packages/ankusa/lib/ankusa/config.ex  0.36  186 lines judged, partial
  packages/ankusa/lib/ankusa/config.ex:1-186  0.36  defmodule Ankusa.Config do
packages/ankusa/lib/ankusa/telemetry.ex  0.23  62 lines judged
  packages/ankusa/lib/ankusa/telemetry.ex:1-62  0.23  defmodule Ankusa.Telemetry do

listed 67 · judged 67 · read 20 files (45.0KB) · 9 requests · 25K tokens · $0.0010 · 791ms wall / 1.9s api

The ranking lined up with the answer. The route in admin/router.ex came first, and config.ex scored only 0.36. There was no flag to find: metrics.ex ships the Prometheus reporter in core for every instance. A fresh node just had no events yet. After the agent sent a demo event through a route, the scrape returned 45 series, and the SDK PR merged with all 94 conformance vectors passing.

3. Four searches behind one design answer

I asked how to bundle limits like rate limits into a plan but still tweak them per account. In 13 seconds the agent sent four searches, which put entitlements.py, feature_gating.py, and teams/models.py at 0.85 to 0.95 for $0.0026 total. From those files it recommended a JSON overrides field on Team, merged over the plan’s limits inside get_entitlements, because every limit check already goes through that function.

4. Five judgments in one eval cell

Early on, I asked my agent what OMP sends Jev. It ran five judgments through judge() at once, four of them modeled on questions OMP can ask. The inputs were made up, and the answers are Jev’s:

Rows 1 and 4 match OMP’s unexpected-stop detection and rules with a question field. Neither has run on Jev in my sessions yet, but 0.94 would clear the 0.7 bar a question rule needs. That cell is 5 of the 6 judge() calls in my table.

5. Picking how hard a subagent should think

When I fanned out 19 subagents on Ankusa with thinking set to auto, Jev read each task’s one-line solutionSpace and picked an effort level:

12 got low, 3 got medium, and the storage state machine got xhigh. Each decision cost about $0.00002.

How it adds up

629 decisions for under nine cents. Most of it is code search my agents run without me asking, and the rest is small calls that would be wasteful to send through a chat model. Low scores like 0.22 show up in the output, so the agent can see a weak match for what it is.

So the next time someone asks what Jev is: it’s the model that answers yes or no with a number attached. Ten Levels of Jev covers many more ways to use it.