Decision Support: Discovering the Model Was Right and the Decision Was Wrong
A digital insurance provider with 480,000 policyholders had a healthy churn model — 0.83 AUC, validated, monitored — driving automatic 12% renewal discounts at €5.6M a year, for a 1.4-point retention improvement Finance could not justify. Envion’s finding: the model was not broken; it was answering a different question from the one the business was acting on. Uplift modelling, a natural experiment and a permanent randomized holdout cut spend to €2.4M while improving retention uplift to +2.1 points.
The challenge
A churn model predicts who is likely to leave. The retention decision requires knowing who will stay because of the offer — a causal question, not a predictive one.
Envion partitioned the customer base into four groups: persuadables — leave without the offer, stay with it, the only group where spend creates value; sure things — stay either way, so discounting them is pure margin loss; lost causes — leave either way, so discounting buys nothing; and sleeping dogs — the counterintuitive group where the offer itself increases churn, typically by drawing attention to a renewal the customer wasn't thinking about or signalling the listed price was never real.
A high churn score identifies risk; it does not distinguish persuadables from lost causes — and lost causes tend to score highest, because they are genuinely leaving. The programme was systematically spending most heavily on the customers least influenceable.
Decision path
Data readiness: eighteen months of offer history existed, but with no control group — every high-scoring customer had received the offer. Without variation, causal effect is not identifiable, and Envion was direct that no amount of modelling could extract the answer from that dataset.
A natural experiment: Envion found one — a 6-week technical fault in 2023 had silently suppressed offer delivery for a subset of eligible customers, effectively at random. Around 9,400 customers formed an accidental control group. It indicated roughly 31% of offer spend went to sure things and around 11% to sleeping dogs where the offer appeared to increase churn — both presented with explicit confidence intervals, with the plain caveat that the sleeping-dog effect needed a designed experiment to confirm.
Uplift prototype: models targeting incremental retention rather than churn probability, built on the natural-experiment data, produced comparable retention at approximately 40% of the discount spend when targeting the top uplift decile instead of the top churn decile.
Envion contribution
The central recommendation was not a model — it was a permanent randomized holdout: 5% of eligible customers withheld from offers on an ongoing basis, so causal effect could be measured continuously rather than inferred from an accident. Envion built the design, sizing and stopping rules, and worked through the internal objection — "we're deliberately letting customers churn" — with a quantified estimate of the holdout's cost against the value of knowing.
Decision design: Envion also mapped where the decision should not be automatic — vulnerable-customer flags, open complaints, recent claims disputes, and any case where a discount could constitute unfair pricing under the client's conduct obligations route to a human retention specialist regardless of score. Envion advised against uplift-based price differentiation entirely — technically feasible, and a conduct risk the client did not need.
Delivery
Production requirements were defined explicitly: the permanent randomized holdout with governance sign-off and quarterly reporting; the uplift model retrained quarterly on holdout-derived data, monitored on incremental retention per euro rather than AUC; conduct-risk exclusion rules enforced before scoring, not after; a decision log linking every offer to score, treatment, outcome and any human override; and a 3-month build plus a mandatory 6-month measurement period before spend reallocation.
Practical rules from the project: distinguish prediction from causation before you act — "who is at risk" and "who will respond to what we do" are different questions; if you never withhold, you can never know — a holdout group is not lost revenue, it is the only instrument that measures whether the programme works at all; look for natural experiments in your own history — outages, phased rollouts, eligibility cut-offs; and consider that the intervention may be actively harmful to some segment — sleeping dogs are real and almost never looked for.
Outcome and evidence
Twelve months after rollout: annual retention discount spend fell from €5.6M to €2.4M, retention uplift versus the no-offer baseline improved from +1.4 to +2.1 percentage points, incremental retentions per €100K of spend rose from 61 to 197, customers offered discounts fell from 84,000 to 29,000, the sleeping-dog segment was excluded after holdout confirmation, and roughly 4% of eligible cases now route to a human specialist.
The designed holdout confirmed the sleeping-dog effect at a smaller magnitude than the natural experiment suggested — a useful reminder, which Envion had flagged in advance, that accidental controls overstate.
Client feedback
What the client says about this engagement
“We had a good model and a bad decision, and for eighteen months nobody separated the two — because the model's metrics were healthy and everyone kept pointing at them. Envion asked one question in the kickoff that reframed everything: how do you know the customers you retained wouldn't have stayed anyway? We had no answer. We couldn't have an answer, because we'd never held anyone back.
The holdout was a fight internally — deliberately not trying to save 5% of at-risk customers feels indefensible until you see the number attached to not knowing. I'd also note they talked us out of a piece of work. Uplift-based pricing was technically on the table and they said don't, on conduct grounds. That's the advice you're paying for.”
Evidence gate. This page publishes only what Envion's project records and client disclosure permissions support. Outcomes are added once verified against a baseline, a measurement period, and an approved source.
FAQ
Questions about this case
Facing a similar challenge?
If a healthy-looking model is attached to spend Finance cannot justify, discuss the decision layer with Envion — the first question is whether the customers you retained would have stayed anyway.
Discuss a Similar ChallengeKeep exploring
Similar case studies
Executive Technology Leadership
Support for high-stakes product and AI decisions
Bring senior technology leadership into the business when the roadmap is unclear, delivery is at risk, an AI initiative needs stronger ownership, or the company needs an experienced technical voice before hiring a permanent CTO.
Discuss Interim CTO SupportCore responsibilities
- Align product and technology priorities with business goals and measurable outcomes.
- Review architecture, delivery risks, data foundations, security needs, and AI readiness.
- Lead internal teams and external partners through a practical execution plan.
- Clarify team structure, ownership, decision rights, and delivery cadence.
- Support investor, board, partner, and due-diligence conversations with credible technical judgment.
New experience
Prompt-to-Page — try it right here
Describe the landing page you want, in your own words. We turn it into a finished page and email you a private link in 5–10 minutes — no briefs, no calls, $0 to see the result.
- Describe what you want to create.
- We structure, write, and compose the page.
- You receive a private link when it is ready.
Start with a sentence — the interactive builder takes it from there.
Generate My PageSafe, respectful content only. No obligation.