Chat over 1,000 Emails: What Changes When We Add Jev

In our previous article, we tested Jev on operations we usually assign to LLMs: identifying email features and assessing whether a message matches a query. On those tasks, Jev showed a promising combination of speed, cost and accuracy. Classification, in particular, was approximately 4.9 times faster and 8.7 times cheaper in API costs, with slightly better quality.

The next question was how these advantages would translate into a complete application. The system now needs to understand the user’s question, find documents, check conditions, sometimes perform several connected searches, and produce an answer with sources. For this experiment, we chose chat over an email archive and compared it with a conventional approach based on an LLM and full-text search.

An email archive can contain several prices for the same product, a tentative delivery date and its later confirmation. A replacement proposed by the supplier may be rejected by the buyer. The discussion resumes a month later, while the old terms still appear in quoted text.

Three things matter in this kind of chat: responding quickly, considering all relevant documents and keeping costs reasonable. For an internal tool, speed and completeness often matter more than a small difference in query cost: the employee needs an answer so they can get on with their work.

Response time is tied to the volume of analysis. Reading a few well-chosen emails is relatively easy. Checking dozens of similar documents, ruling out refusals and superseded decisions, and reconstructing the connections between them takes more work. The more sources we are willing to check, the better our chance of producing a complete answer, and the more work the system has to do.

We therefore expanded the archive from our first experiment from 100 to 1,000 emails. It contains 77 threads: 990 messages concern the fictional customer Vektor, and another 10 concern Orion. A single product now appears in several projects, similar models are discussed alongside it, and a proposal, its approval and the actual delivery may be recorded in different messages.

Experiment data

All emails were created specifically for this study. The companies, people, products and transactions are fictional; no customer correspondence was used. Below is a message from this synthetic archive.

Both implementations store the original messages in SQLite. During import, the code parses headers and records ordinary metadata: email and customer IDs, date, thread, reference to the previous message, subject, sender and recipient. Product-code mentions are also extracted locally and stored in emails.metadata.product_ids.

The model then identifies semantic features. Both indexes use the same schema of 15 required boolean fields:

It is useful to distinguish the presence of information from the values themselves. has_accepted_price=true helps locate an email containing an accepted price. To answer the user, the agent still needs to establish the amount, product, order and applicable pricing terms from the source.

The baseline indexer sends gpt-6-luna one email together with descriptions of all 15 features. The response is constrained by JSON Schema and validated with Pydantic. Written out explicitly, the contract looks like this:

In the implementation, the class is created with create_model: each field receives a Field(description=...) from a shared feature dictionary. All fields are required, and extra fields are forbidden. The result is stored both as a complete classification and as individual rows in the flags table, allowing emails to be filtered by feature.

In this implementation, the LLM indexes boolean features. The agent reads prices, manufacturers, countries of origin and relationships between documents from the retrieved originals while preparing its answer. SKUs in ordinary metadata come from local parsing, rather than value extraction by this Pydantic model.

With Jev, the same 15 features become Noul questions. The response is a score indicating whether a condition is true; in our experiment, we accept it at a threshold of 0.5.

Values require an additional step. In the interface we use, Jev checks supplied candidates. Code must therefore extract a value from the text first: we wrote regular expressions, then asked Jev whether each extracted string actually played the proposed role in the email.

The classes describing an extracted candidate and the assembled Jev request look like this. payload contains the email and its set of questions, while candidates links each model response to the original value:

A single request includes the 15 main checks, additional checks for the presence of a product or a service offered for sale, and questions about every extracted candidate. For each value, we specify its proposed field, context from the email and acceptance criteria. Jev sees the subject and the full message body.

Consider email-918. It mentions a standard heat exchanger, another model and different prices in the same message:

For SKUs, we use this pattern:

It finds HX-90 and HX-90B. Matching the format still requires a semantic check: a similar string might be a technical designation or a document number. For HX-90, we therefore ask: is this string used as a product code in this email?

A broad supplier pattern also finds Italy before a full stop. The country-of-origin pattern extracts the same string after country of origin is. This produces two candidates with the same value but different proposed fields. Jev checks each in context:

These are actual scores from the indexing log. Italy is retained as the country of origin and rejected as a supplier. Both product codes are confirmed as model mentions: having HX-90B in the index does not mean that its price applies to HX-90.

Storage is divided into three parts:

• emails.metadata — ordinary metadata and locally extracted product codes;
• flags — the 15 boolean features, plus their scores for Jev;
• entities — values accepted by Jev, with their field, score and source span. The complete result, including rejected candidates, is stored in classifications.result.

Here is a shortened representation of the confirmed values from this email:

This index helps filter documents by specific values and explain where each field came from. However, extraction quality depends on the patterns. If a regular expression misses an unusually formatted SKU or name, Jev never receives that candidate for validation. Extractors need to be extended for new formats and languages. A price mention alone also does not establish which product it applies to: relationships and current terms still need to be checked against the original message.

In this run, the LLM completed indexing faster in wall-clock time. The two tasks have different workloads and concurrency settings: the LLM identifies 15 features, while Jev also validates values. These measurements describe our indexer configurations; comparing model speed under the same workload would require the same execution settings.

Jev confirmed 3,753 values in addition to the 15,000 main boolean decisions. The sum of API attempt durations was approximately 1.91 times lower, but building the index took longer at the chosen concurrency. Indexing cost should be considered separately from the cost of subsequent chat answers.

We prepared three questions of each type — 12 final questions. They test different ways of working with sources, from retrieving a single fact to checking a new semantic condition.

1. Find the Source and Answer Immediately
   Example: what power supply, motor rating and wetted casing material were specified for standard P-31 in the Cedar electrical review?
   The answer is in one email: 400 V AC, three phases, 2.2 kW and AISI 316L. The agent needs to find email-152 and distinguish standard P-31 from the P-31H variant. Both agents gave a complete answer.
    
2. Split the Question into Independent Parts
   Example: how did the accepted P-31 prices change between two orders, who is the manufacturer, and what is the country of origin?
   The two prices and the product’s origin can be retrieved independently and then combined into one answer. The price fell from EUR 1,040 to EUR 980 — a decrease of EUR 60, or approximately 5.77%. The manufacturer is NordWerk, and the country is Finland.
   Both agents answered completely. In this example, the baseline agent took 9.94 s, and the hybrid took 5.33 s.
    
3. Use a Retrieved Fact to Guide the Next Searc
   Example: which product belongs to the approved replacement bundle for Cedar RMA-Q71, and what price was accepted in SQ-2603?
   First, the service-case decision points to bundle KIT-M9. The bundle register then identifies product SK-8R. Its commercial terms can be found once that relationship is established:

Both agents reconstructed the chain and gave the price per set, excluding VAT and carriage. Both implementations produced complete answers to all three questions of this type.

4. Check a Condition That Is Not in the Index

The user first asks for an overview of the P-55 discussions. They then follow up: which of these emails record that Vektor actually authorised continued production before the protective interlock had passed its verification?

The emails are already loaded, but this condition was not indexed in advance. We cannot prepare fields for every possible user interest. Usually, candidates are retrieved through full-text search or embeddings, then read and checked. Our baseline uses full-text search; the hybrid checks the new criterion against the original emails with Jev.

The word “approval” alone is insufficient. It appears in permission, refusal, requests for approval and old quotations. The agent needs to establish who gave permission, whether it applies in the current message and whether verification had already been completed.

The first three scenarios describe a sequence of steps; the fourth introduces a new semantic condition. In a real chat, they can be combined.

Both agents use gpt-6-luna to plan retrieval and generate an answer with sources. They differ in their prepared indexes and how they check documents.

LLM + FTS searches the 15 features and ordinary metadata. For conditions outside the schema, it uses SQLite FTS5 / BM25 to retrieve text matches, then analyses the original messages with the LLM.

LLM + Jev uses an index of features and confirmed values. Jev identifies the query category, checks candidates and scores their usefulness. When a new condition appears, the LLM formulates it for search_jev, and Jev checks documents within the selected scope.

Below are the results for the 12 scenarios. Each group contains three questions; times are averaged over completed scenarios. For the fourth type, both messages are included: the overview and the follow-up.

The Jev agent received 12 out of 12 complete grades. The baseline had seven complete, two partial and two incorrect answers. No automated grade was recorded for #006, so we do not count that question as an error.

The combined elapsed time across all 15 turns was 149.23 s for LLM + FTS and 83.33 s for LLM + Jev — approximately 1.79 times less for the hybrid. The gain depends on the question: for #009, the hybrid took 17.21 s versus 8.28 s, with both answers graded complete.

The hybrid used 155,804 generative LLM tokens, compared with 380,314 for the baseline: a reduction of approximately 59%. However, Jev performs part of the analysis and has its own usage and cost.

The hybrid’s total token count, including Jev, was 545,236. Total retrieval cost in this series was $0.016711 for the baseline and $0.023890 for the hybrid. Broader semantic analysis produced more complete answers and lower combined elapsed time in this experiment, at a higher API cost.

Return to the question about permission to continue production before interlock verification. In the simultaneous UI run, the baseline reported that there were no matching emails. The Jev agent checked all 100 documents in the selected pool and found seven:

The baseline made three LLM calls. The hybrid made two LLM calls and 35 Jev HTTP requests, checking 100 documents. Across the three new conditions, that amounts to 300 checks of whether an email matches a criterion. Being able to check more possibilities in a reasonable time helps preserve completeness.

The follow-up answer took 7.18 s for the baseline and 6.22 s for the hybrid, at costs of $0.001313 and $0.004689. This is a separate paired measurement and is not included in the full-series table above.

The other two questions involving a new criterion showed a similar pattern: the baseline named two of the seven required emails in #011 and reported no supporting evidence in #012. Jev returned the complete set in both cases. For the simple question #002, the baseline answer was partial because it omitted a carriage condition. Most answers from both agents were complete on independent subtasks and dependent chains.

The Jev agent was also checked against another synthetic archive: seven out of seven questions received complete grades. That check included a new semantic criterion, different products and names, time-period handling, and cases where the documents contained no answer.

Below are recorded answers and their source emails. You can switch scenarios and replay the appearance of each answer using its recorded duration. Questions #001, #004 and #007 use the full runs; #010 uses the separate paired measurement. Viewing the examples makes no new model requests.

Our first Jev experiments showed advantages in individual operations. In an application, the result also depends on the index, how candidates are retrieved and the volume of analysis.

In our email chat, Jev made it possible to check more documents against a semantic condition while keeping the generative LLM’s context small. The clearest difference appeared on questions that could not be represented by predefined index fields: checking the full selected pool found supporting evidence that the baseline search missed.

For the practical development of this kind of chat, we would leave value extraction and indexing to the LLM, and use Jev during retrieval. An LLM can extract manufacturers, countries, prices, quantities and document references from free text into a defined structured schema. This reduces dependence on a collection of regular expressions. Values still need to be validated against the contract and stored together with supporting spans.

In the measured baseline indexer, the LLM returned only 15 boolean features. Extending its contract to extract values is the next step in this architecture. The example above illustrates the reason for this division of work: Jev indexing required us to extract candidates separately and check roles such as those proposed for Italy; during retrieval, Jev already receives the original documents and can quickly check the user’s condition.

The combination with embedding-based retrieval is particularly interesting. It can find semantically related documents even when their wording differs from the question. However, semantic similarity does not guarantee the required fact: a request for approval, a refusal, an old proposal and a current authorisation may all appear together. A similar separation between broad candidate retrieval and subsequent assessment is used in the retrieve & rerank approach.

Jev can perform the next step: check retrieved sources against a specific condition, remove unsuitable documents and rank the remaining ones. We can then retrieve a wider candidate pool through embeddings and check more documents before passing supporting evidence to the LLM for its answer. Embeddings broaden coverage, while Jev checks whether sources satisfy the question. This combination could improve completeness within an acceptable response time.

We did not measure the combination with embeddings in this experiment; it is a direction for the next implementation. Its completeness also depends on the initial retrieval step: Jev can only check sources that enter the candidate pool.

API costs increased in our series. For an internal chat, this can be a reasonable trade-off if complete answers and shorter waits matter more than a small difference in expenditure. The practical choice depends on the questions in the workflow: which documents need to be considered, how critical an omission would be, and how long the user is willing to wait.

Our conclusion: the LLM prepares structured data and formulates the answer; Jev checks a broad set of sources during retrieval. This division of work lets us use the strengths of both approaches in one email chat.