Jev for Tasks We Usually Solve with LLMs: A Five-Model Comparison

For comparison, we selected a conventional LLM, a zero-shot classifier, and two models with publicly available weights that emerged after Jev: Laya and the Qwen System One scorer. We tested them on email classification and retrieval. This synthetic example closely resembles one of our real tasks and lets us evaluate operations that are useful far beyond email.

TypeSafe introduces Jev as the first model in its System One family: it accepts data and predefined questions, then returns values of specified types. According to its developers, Jev computes structured decisions without generating free-form text. The interface is designed for direct use in application logic. Jev announcement.

The interface has three main question types:

These types are available in the TypeSafe Python SDK. For our experiments, Noul is sufficient: it can check whether an attribute is present or assess whether a document contains information needed to answer a question.

For example, a request to check a single attribute looks like this:

The application receives a numeric score that it can use in a condition, a sort operation, or a filter. Multiple attributes can be requested at once.

There is an important distinction between the format of a response and its content. A typed interface defines the valid shape of the result. Whether the model understood the email and chose the correct value still needs to be tested. That is what the quality measurements in our experiments address.

The task “here is a text; evaluate a statement about it” is familiar from zero-shot classification. NLI classifiers, for example, assess whether a text supports a proposed hypothesis. Our chosen model, MoritzLaurer/bge-m3-zeroshot-v2.0, explicitly uses an entailment versus not_entailment formulation. Model card.

We can give it an email and the hypothesis “The email states a concrete price.” For another task, we change the hypothesis; no fine-tuning on our email corpus is required.

In this sense, our use of Jev is also zero-shot: we describe the task in words and use an existing model. However, zero-shot describes how a model is used. It does not, by itself, specify the model’s architecture, training, or performance on a particular task.

Comparing Jev with NLI was therefore a natural choice. If an existing classifier already performs an operation well enough, it provides a useful baseline for evaluating a new approach.

We originally planned to compare Jev with an LLM and a zero-shot classifier. After Jev appeared, other projects began positioning themselves as open alternatives—a kind of “open Jev.” We added two of them to see whether similar decision interfaces would produce similar results.

Laya supports noul, choice, and score questions over a supplied state. The implementation we tested uses a bidirectional ModernBERT-large encoder with additional decision heads. It has approximately 421 million parameters. Its author directly compares it with TypeSafe Jev and publishes both code and weights. Laya project.

Qwen System One scorer is built on Qwen3.5-4B-Base, with a LoRA adapter and a trained scalar head. The model computes a score for each combination of data, question, and answer option, then normalizes the option scores. In our case, the options are yes and no. This configuration does not generate free-form text. Qwen scorer model card.

This gave us the following lineup:

We used all models without fine-tuning on our email dataset. In the tables below, “LLM” refers specifically to our chosen gpt-5.6-luna configuration.

In one of our real workflows, a client wants to ask questions about their correspondence with buyers: which products were discussed, what prices were agreed, when delivery was promised, and who is currently expected to take action.

Two operations are useful in this application. First, emails can be labeled in advance to identify those containing prices, delivery times, quantities, and specifications. Then, when a user asks a question, the application needs to select the most useful sources and pass them to an LLM to compose an answer.

We reproduced this scenario in a synthetic corpus. All companies, emails, and transactions are fictional, while the situations themselves are typical of business correspondence.

The corpus contains 100 English-language emails spanning 2024–2026. Of these, 90 belong to correspondence with Vektor: six threads covering different products and transactions. The remaining 10 form a single thread with Orion—five buyer messages and five supplier replies. Both customers discuss the P-17 product, so matching the product name alone is not enough to find the correct answer.

We first defined the transaction histories and facts, then turned them into correspondence. This established in advance what happened and which emails support each answer.

In some threads, a buyer asks about a product but decides not to purchase it. In others, they return to it later and place an order. An initial price may differ from the agreed price. A tentative delivery date is replaced with a confirmed date, while quoted messages still contain the old terms. A buyer’s voltage or temperature requirements do not always match the capabilities of the proposed product.

We prepared a separate set of attribute labels for the emails. For retrieval, we created 36 questions, each with a search scope, a date, expected facts, and supporting emails. The models were not given the reference answers or lists of correct sources.

For each email, the task was to identify seven independent attributes:

The wording affects the meaning of the task. Asking “How much does it cost?” does not mean that the email contains a price. A request for a delivery date does not contain the date itself. An old delivery date in a quotation still counts as a mention of timing, while a request that has already been withdrawn should not count as an active request for action.

Here is an excerpt from one of the emails:

The ground-truth labels for the full email are:

The LLM returned this kind of object through structured output with a JSON Schema. Jev received seven Noul questions in a single request. For this email, for example, it returned the following scores, shown here as the contents of the scores field after normalization by our adapter:

We converted the scores from Jev, NLI, Laya, and Qwen into Boolean attributes using a threshold of 0.5. For this email, both Jev and the LLM got all seven attributes right.

NLI processed seven email–hypothesis pairs in a single internal batch. Laya received seven questions. The Qwen scorer evaluated yes/no for each attribute, giving 14 sequences per email. Each model saw the full original email, including headers and quoted messages.

We measured elapsed time until a validated result was available. Model loading and warm-up were accounted for separately. API costs were calculated from token usage and the rates in effect on the experiment date; for computation on our own hardware, we recorded a cost of $0 in the tables.

Indexing ran sequentially, one email at a time. NLI and Laya ran on a Mac using MPS, Qwen ran on an RTX 3090, and Jev and the LLM were accessed through APIs. For the LLM, we used the standard service tier and default reasoning settings, with a limit of 4,096 output tokens. Reasoning tokens were included in the cost.

There were 700 decisions in total: 100 emails × 7 attributes. Alongside the proportion of correct labels, we counted emails for which all seven fields were correct.

Jev performed close to the LLM in quality and slightly outperformed it on this corpus. It processed the full set of emails 4.9 times faster, with API costs lower by a factor of 8.7.

The fast local models showed a different pattern. NLI and Laya processed the emails faster than Jev but made substantially more errors. Both got all seven fields right for only 15 out of 100 emails. The Qwen scorer performed better: 85.43% of labels were correct, with 36 fully correct emails.

We also calculated macro F1—the mean F1 across the seven attributes, giving each attribute equal weight. The scores were 0.977 for Jev, 0.968 for the LLM, 0.397 for NLI, 0.596 for Laya, and 0.723 for the Qwen scorer. The ordering of the models remained the same on this metric.

Neither Jev nor the LLM made any errors on concrete prices. The action-request attribute was the hardest for Jev: it accounted for 11 of its 16 errors. The LLM made 12 errors on this attribute and 13 on identifying product information.

This illustrates why it is useful to evaluate fields separately. A concrete amount is often stated directly in the text. Detecting an action request requires understanding who is writing to whom, whether the request is still active, and whether it belongs to the current message or an old quotation.

In this run, NLI did not make a single positive prediction for product quantity or tentative delivery timing. Its fast responses therefore did not, by themselves, produce a useful index for these fields.

There is also a subtlety in the labeling itself. In one email, the supplier withdraws a tentative date of November 7 and confirms a new date of November 12. The delivery_is_tentative attribute remains true for that email because a tentative date is mentioned in the text. Identifying the current delivery date is a separate task involving the user’s question and the correspondence history.

In the second experiment, we tested ranking. Given a user question, each model had to score candidate emails and select the three most useful ones.

For example:

This requires finding information about the power supply and motor rating. An email requesting specifications may concern the right product without containing the answer. An email that states the voltage, number of phases, and power rating is a suitable source.

For each question, we prepared a search-intent description and seven Boolean category selectors, shared by all models. A question about price and delivery selects has_price and has_delivery_time.

The filter uses OR: an email qualifies if it contains either a price or delivery timing. These details may appear in different messages. A value of false means the category is not used in the filter. If no category applies, category filtering is skipped.

The filter also respects the customer and date: a historical question must not receive information from future emails. Questions spanning all customers have no restriction to a single buyer.

To select candidates, we used the ground-truth category labels prepared when creating the corpus. We did not use the predicted indexes from the first experiment. All five models therefore ranked the same set of emails for each question.

Jev and Laya evaluated a statement that the email contains concrete information to answer the query. NLI checked the corresponding hypothesis. The Qwen scorer evaluated the yes/no options. The LLM returned JSON containing a number from 0 to 1 according to a specified relevance scale, ranging from no useful information to a direct answer to the query.

The task had the same meaning for every model: assess the email’s usefulness as a source. We did not compare the numeric scales across models. Retrieval used the ordering of emails within each model’s results.

We did not apply the 0.5 threshold at this stage. Every candidate received a score, and the candidates were then sorted in descending order. Ties were resolved using a hash of the question and email ID.

Here is Jev’s actual top three for the question about the P-17’s power supply and motor rating:

All three emails are on the reference list of sources for this question. The LLM also selected three relevant emails, although the selection and ordering differed slightly.

The main metric was Hit@3: whether at least one of the three selected emails belongs to the ground-truth set of relevant sources. Each question is a success or a failure, and the final score is the proportion of successful questions.

The dataset contains 36 questions. For 31, the answer is present in the correspondence; for the other five, the requested information is absent. All 36 were included in the timing and cost measurements. Hit@3 was calculated over the 31 answerable questions.

We also checked evidence sufficiency in the top three: whether the selected emails support all the facts in the reference answer. This is stricter than a hit. A single email may be enough for product specifications, while a history of price changes requires several messages. The annotations include questions whose sufficient evidence set contains more than three emails.

Filtering produced 1,585 question–email pairs per model. Questions were processed sequentially. For the API models, up to five candidate requests ran concurrently within each question; local models processed one candidate at a time.

This experiment ran entirely from the GPU server: NLI, Laya, and Qwen ran sequentially on the same RTX 3090, while Jev and the LLM were called through APIs from that server. NLI and Laya used float32; Qwen used bfloat16. Query time included filtering, processing all candidates, receiving results, and sorting. It excluded loading, preliminary input-length checks, and warm-up.

Jev and the LLM achieved the same result on the main metric: a relevant email appeared in the top three for 30 out of 31 questions. Jev completed the full run 5.2 times faster, with API costs lower by a factor of 8.4.

For an interactive application, it is also useful to look at the time per question. The median was 2.26 seconds for Jev versus 11.62 seconds for the LLM. At the 95th percentile, it was 6.71 versus 37.99 seconds. This is the time spent selecting sources, before generating the user-facing answer.

For evidence sufficiency, Jev succeeded on 19 out of 31 questions, compared with 17 for the LLM. Equal Hit@3 scores therefore did not imply equally useful retrieved context.

The most consistent finding was that Jev performed close to the chosen LLM in quality on both operations, with substantial gains in speed and cost. This held both for identifying a fixed set of attributes and for assessing relevance against a changing question.

The models with publicly available weights produced different results. In indexing, the Qwen scorer outperformed Laya: 85.43% correct labels versus 74.86%. In ranking, the order reversed: Laya achieved 77.42% Hit@3, compared with 61.29% for Qwen. Assuming that performance on one operation would carry over to the other would have been a mistake.

The NLI classifier was the fastest in both experiments. However, in ranking, it found a relevant source for only 17 out of 31 questions. For our task, the speed advantage came with a substantial loss in quality.

Jev and the LLM also failed on the same single question under Hit@3: which customers currently need a follow-up about P-17, and what should be discussed with each? This asks about the current state of relationships and obligations, built up across several messages.

This case exposed the influence of the initial filter. The action-request category retained only three of the ten annotated relevant emails; neither model then placed any of those three in its top three. Correct attributes for individual messages do not, by themselves, reconstruct the history of fulfilled and outstanding obligations. Retrieval needs to account for those relationships.

In the application, the retrieved emails should become context for an LLM. We assign the final answer to the LLM: combining information, comparing terms, explaining changes, and citing sources.

Jev can handle classification and candidate scoring within this system. A practical design looks like this:

The ranking experiment used a ground-truth index so that all participants received identical candidates. In an application, the index can be generated automatically. The numbers above measure classification and selection; final answer generation was not included.

This division of work is particularly useful when many small decisions must be made before producing a single answer. The cost of scoring dozens of candidates, and the time spent waiting for those scores, affect the entire interaction. This is the stage where we measured a substantial improvement.

Email is one example. Similar operations appear in support-ticket processing, document workflows, request routing, and knowledge-base search. In each case, we can identify the attributes, checks, or scores that the application needs from a model and compare ways of producing them.

We started by asking whether Jev could handle common structured tasks that we usually solve with LLMs. For the two operations we tested, the answer was yes: on our corpus, Jev achieved quality close to the selected LLM, doing the work roughly five times faster at one-eighth to one-ninth of the API cost.

The comparison with NLI, Laya, and the Qwen scorer added another part of the picture. Speed, model size, and a similar interface do not determine performance on a particular operation. We had to measure indexing and ranking separately.

For our work, the main result is the option to assign classification and data selection to Jev, then pass the prepared sources to an LLM for the final answer. This gives us a concrete way to reconsider applications in which almost every decision currently requires a separate LLM call.