The Challenge
A regional credit union serving approximately 85,000 members across Ontario was operating a fraud review process that had become unsustainable. The compliance team — six investigators — was responsible for reviewing every transaction flagged by the institution's rules-based fraud detection system. On paper, the system was performing: it caught genuine fraud. In practice, it was generating more work than the team could handle, most of it unnecessary.
The false positive rate was running at 60%. For every ten transactions flagged for review, six were legitimate — ordinary member activity that had tripped a detection rule because it was unusual in some way: a large transfer, an out-of-province purchase, a transaction at an unusual hour. Each of those six required a manual investigator review to clear. Each review took an average of 4.2 hours, including pulling transaction history, contacting the member if warranted, documenting the outcome, and logging the case in the compliance system.
The caseload math was brutal. On a typical week, the system flagged 140–180 transactions. With a 60% false positive rate, that meant 84–108 false positives requiring manual clearance, consuming the equivalent of 2.5 to 3 full-time investigator positions just to process cases that should have been cleared automatically. The remaining investigator capacity for genuine fraud cases — cases that required substantive analytical work — was chronically squeezed.
The compliance team's morale reflected this reality. Investigators had joined the institution to investigate fraud. They were spending the majority of their time confirming that legitimate transactions were legitimate. Staff turnover in the compliance unit had run above 30% annually for two consecutive years.
The institution was also facing a regulatory dimension. The Office of the Superintendent of Financial Institutions (OSFI) had flagged the credit union's fraud detection framework as a review area in its next scheduled audit. The combination of high false positive rates, manual-heavy processes, and investigator capacity constraints created audit exposure. The institution needed to demonstrate that its fraud risk management framework was proportionate, systematic, and effective — not that it relied on investigators working through large queues of low-probability alerts.
The Approach
Audit (3 weeks). We conducted a detailed assessment of the existing fraud detection infrastructure — the rules-based system's 34 detection rules, the transaction data architecture, the case management workflow, and the compliance reporting framework. We interviewed all six investigators, the compliance manager, and the Chief Risk Officer.
The audit produced a specific diagnosis: the rules-based system had been built to maximize recall (catching all fraud) without any constraint on precision (minimizing false positives). The rules were conservative by design — no one wanted to miss a genuine fraud case — but the conservatism was miscalibrated. Many rules were flagging transaction patterns that historical data showed to be associated with genuine fraud in less than 3% of cases. The AI opportunity was not to replace the rules-based system but to add an intelligent triage layer that assessed each flagged transaction before it entered the investigator queue.
Strategy (4 weeks). We designed a two-layer architecture: an AI anomaly detection model operating upstream of the rules-based system, and an automated triage engine that assessed rules-flagged transactions against member behavioral history, network-level patterns, and real-time risk signals before routing them to investigators.
The design specified three routing outcomes for each flagged transaction:
- Auto-clear: Transactions where the AI assessed fraud probability below a defined threshold, based on member history and transaction context, were automatically cleared with a documented rationale — no investigator involvement required.
- Priority review: Transactions with elevated fraud probability indicators were routed to investigators with an AI-generated case summary, priority ranking, and recommended investigation path.
- Standard review: Transactions requiring investigator assessment but not meeting priority criteria were queued with AI-generated context summaries.
The auto-clear threshold was calibrated conservatively to ensure that no transaction with meaningful fraud indicators was cleared without investigator review. The threshold was set based on analysis of three years of historical case data, validated against known fraud outcomes.
The design was reviewed and approved by the institution's Risk Committee and Chief Compliance Officer before implementation began.
Implement (3 months). Implementation proceeded in two phases. The first month deployed the AI anomaly detection model — trained on 36 months of the institution's own transaction data plus anonymized consortium data from other credit unions operating within the same regulatory framework. The model identified 47 distinct behavioral features that, in combination, differentiated genuine fraud from false positive patterns with significantly greater precision than the existing rules.
The second and third months deployed the triage engine and integrated it with the existing case management system. Investigators continued to work within their familiar case management interface; the AI layer enriched each case with a structured summary, probability assessment, supporting evidence, and recommended action. The investigator made the final determination on every case that reached them.
The auto-clear logic was subject to a human override for the first 60 days of deployment: a senior investigator reviewed a random 20% sample of auto-cleared transactions each week. No errors were identified in 12 weeks of sampling. The override was subsequently moved to a monthly sampling protocol.
Empower (parallel). Training focused on three areas: working with AI-generated case summaries (understanding what the model is assessing and why, rather than treating its output as a black box), using the feedback mechanism to flag cases where the AI assessment appeared incorrect, and producing the compliance documentation that would demonstrate the framework's effectiveness to OSFI.
The Results
- 87% reduction in false positive rate. The false positive rate dropped from 60% to 7.8% within 90 days of full deployment. Investigators' caseload shifted dramatically toward genuine fraud cases — the work the institution was paying them to do.
- Average case resolution time dropped from 4.2 hours to 45 minutes. AI-generated case summaries, pre-populated with transaction context and member history, eliminated the investigator time previously spent assembling case information manually. Investigators could focus their time on analysis and judgment rather than data gathering.
- OSFI audit passed. The institution's next scheduled OSFI review occurred four months after full deployment. The audit assessed the fraud detection framework and found it to be proportionate and effective. The documented auto-clear logic, sampling protocols, investigator oversight design, and case outcome tracking gave auditors a systematic framework to evaluate — rather than a manual process with limited documentation.
- Investigator capacity released for complex cases. With false positives reduced by 87%, the compliance team's effective capacity for substantive fraud investigation increased by approximately 230%. Two investigators who had been primarily processing false positives were redeployed to a newly established financial crime intelligence function — proactive pattern analysis rather than reactive case processing.
- Staff satisfaction materially improved. Compliance team attrition dropped from above 30% annually to zero in the 12 months following deployment. Investigators cited the change in work quality — substantive investigation rather than false positive clearance — as the primary driver.
Key Lessons
1. Precision matters as much as recall in fraud detection. Maximizing recall — catching every fraud case — is the instinct that drives most fraud detection systems toward high false positive rates. But false positives are not free: they consume investigator capacity, degrade morale, and obscure genuine fraud signals in a queue of noise. An AI layer that improves precision without degrading recall produces better investigative outcomes and a more sustainable compliance operation.
2. Regulatory compliance requires systematic documentation, not just manual oversight. OSFI auditors are not looking for evidence that people are working hard. They are looking for evidence that the institution has a systematic, documented, and proportionate framework for managing fraud risk. The AI-augmented system produced that evidence automatically — every auto-clear decision was documented with the rationale, every priority case was tracked through outcome, and the sampling protocol provided ongoing validation. Manual processes rarely produce documentation of equivalent quality.
3. Augmented investigators outperform both pure AI and pure manual review. The institution briefly considered a fully automated fraud clearing system. The risk assessment did not support it — for a federally regulated financial institution, eliminating human judgment from fraud disposition introduced regulatory and liability risk that was not acceptable. The hybrid model — AI triage, human decision — produced a better outcome than either approach alone: the speed and consistency of AI processing combined with the judgment and accountability of investigator oversight.
For financial institutions looking to modernize fraud detection and compliance operations, see Remolda's financial services AI capabilities and our workflow automation services.