Document Q&A: Yes for Two Question Types, No for the Third
A freight forwarding group with 4,000 active carrier and customer contracts wanted an LLM assistant for contract questions; their internal proof of concept was "amazing in the demo and wrong often enough that nobody uses it." Envion refused to prototype for the first two weeks and built a 400-question evaluation set instead — which produced a taxonomy: lookup questions (61%, 94% accuracy — deploy), interpretation (27%, 88% with clause-level citation and mandatory human confirmation — deploy as assisted), and aggregation (12%, 34% and undetectably wrong — do not deploy; extract fields and answer with SQL instead). The assistant refuses aggregation questions on purpose — and that single behaviour rebuilt trust with the ops floor.
The challenge
Operations and commercial teams asked contract questions constantly — what's the detention free time for this customer, does this carrier agreement allow fuel surcharge pass-through, what's our liability cap on this lane. Answering meant finding the right contract among 4,000, then the right clause among forty pages of amendments.
They had already run a two-week internal proof of concept, described as "amazing in the demo and wrong often enough that nobody uses it."
That's a very common place to be, and the reason is almost always the same: nobody defined what correct meant before building. A fluent wrong answer about a liability cap is considerably worse than no answer.
Decision path
No prototyping for the first two weeks. Instead, an evaluation set — because without one you cannot distinguish a system that works from a system that sounds like it works. 400 real questions from ticketing history and Slack, answered by contracts specialists with the source clause cited, verified by a second specialist: ground truth, and something more useful — a taxonomy.
Type A — Lookup (61%): "What's the detention free time in the Maersk 2024 agreement?" One clause, one contract, factual. Retrieval finds it, the model reads it out. Type B — Interpretation (27%): "Does this contract let us pass through the new emissions surcharge?" Requires reading several clauses together; the failure mode is subtle — a defensible-sounding reading a lawyer would disagree with. Type C — Aggregation (12%): "How many of our carrier contracts have a liability cap below €100K?" This is where these systems quietly break, and it's the type executives ask most.
Retrieval is a similarity search — it returns the most relevant chunks, not all matching chunks. Ask it to count across 4,000 contracts and it will confidently answer based on the twenty documents it happened to retrieve. The answer is wrong, wrong by an unknowable margin, and nothing in the output signals that. The internal proof of concept had been doing exactly this — which is why trust collapsed.
Envion contribution
With the eval set in place, the prototype could report real numbers per question type rather than an overall impression. Type A: 94% accuracy — deploy. Type B: 71%, rising to 88% with one design change — forcing the answer to quote and cite the specific clauses it relied on, framed as "here are the relevant clauses and my reading" rather than as an answer; users verify in seconds instead of trusting blindly, and accuracy of the system including the human is what matters. Type C: 34%, unreliable in a way the user can't detect — do not deploy.
For aggregation questions the correct engineering answer isn't a better prompt, it's a different architecture: a one-time extraction pass over all 4,000 contracts pulling a defined field set — liability cap, renewal date, auto-renewal flag, payment terms, detention terms, surcharge provisions — into a structured database, a hand-verified sample, and aggregation answered with SQL. Boring, deterministic, correct, auditable.
Envion also insisted the assistant refuse Type C questions rather than attempt them, with a message pointing to the structured query tool. Getting a system to decline is unglamorous — and it's the thing that preserved trust after the previous attempt had burned it.
Delivery
Production requirements: permission-aware retrieval — a commercial manager must not surface a contract they're not cleared to see, enforced at retrieval, not by filtering output. Re-indexing on contract amendment, since a stale index answering from a superseded clause is the worst failure available here. The eval set run as a regression suite on every model or prompt change. And a logged escalation path to the contracts team for anything the system declines or flags low-confidence.
Outcome and evidence
Ten months on: median time to a contract answer fell from 25 minutes to 40 seconds, questions routed to contracts specialists fell from ~900 to ~180 per month, 88% of Type A and B questions are answered without escalation, aggregation questions moved from unreliable to deterministic structured queries, and weekly active adoption among ops users went from ~5% on the prior proof of concept to 79%.
The advice that generalizes: build the evaluation set before the system — two hundred real questions with verified answers converts the entire project from opinion to measurement. Categorize the questions, because the answer is almost never uniform: most document-assistant projects are highly feasible for the majority of what people ask and structurally infeasible for a minority — and the infeasible minority is usually the part leadership is most excited about. And design refusal deliberately: a system that declines 12% of questions gets trusted with the other 88%; a system that attempts everything gets abandoned.
| Metric | Before | After |
|---|---|---|
| Median time to a contract answer | 25 min | 40 sec |
| Questions routed to contracts specialists | ~900/mo | ~180/mo |
| Type A + B answered without escalation | 0% | 88% |
| Aggregation questions | Unreliable | Deterministic (structured queries) |
| Adoption (weekly active ops users) | ~5% (prior PoC) | 79% |
Client feedback
What the client says about this engagement
“Our first attempt failed on trust, not on technology. It answered everything and was wrong sometimes, so people stopped using it entirely. Stanislav's version answers less and gets used more. He spent the first two weeks building a test set instead of building the product, which I questioned at the time — and it's the reason we could see that twelve percent of our questions were fundamentally not a retrieval problem.
The system now says 'I can't answer that, use the contract database' and that single behaviour is what rebuilt confidence with the ops floor.”
From the engagement lead
What I’d tell anyone considering this
“Build the evaluation set before the system. Two hundred real questions with verified answers from your own experts. It's a week of unglamorous work and it converts the entire project from opinion to measurement — you'll know what accuracy you have rather than guessing from demos.
Then categorize the questions, because the answer is almost never uniform. Most document-assistant projects are highly feasible for the majority of what people ask and structurally infeasible for a minority — and the infeasible minority is usually the part leadership is most excited about. Finding that boundary early is the whole job.
And design refusal deliberately. A system that declines 12% of questions gets trusted with the other 88%. A system that attempts everything gets abandoned.”
Evidence gate. This page publishes only what Envion's project records and client disclosure permissions support. Outcomes are added once verified against a baseline, a measurement period, and an approved source.
FAQ
Questions about this case
Facing a similar challenge?
Building an assistant over your document set? Define "correct" before you build — Envion starts with the evaluation set, not the demo.
Discuss a Similar ChallengeKeep exploring
Similar case studies
Executive Technology Leadership
Support for high-stakes product and AI decisions
Bring senior technology leadership into the business when the roadmap is unclear, delivery is at risk, an AI initiative needs stronger ownership, or the company needs an experienced technical voice before hiring a permanent CTO.
Discuss Interim CTO SupportCore responsibilities
- Align product and technology priorities with business goals and measurable outcomes.
- Review architecture, delivery risks, data foundations, security needs, and AI readiness.
- Lead internal teams and external partners through a practical execution plan.
- Clarify team structure, ownership, decision rights, and delivery cadence.
- Support investor, board, partner, and due-diligence conversations with credible technical judgment.
New experience
Prompt-to-Page — try it right here
Describe the landing page you want, in your own words. We turn it into a finished page and email you a private link in 5–10 minutes — no briefs, no calls, $0 to see the result.
- Describe what you want to create.
- We structure, write, and compose the page.
- You receive a private link when it is ready.
Start with a sentence — the interactive builder takes it from there.
Generate My PageSafe, respectful content only. No obligation.