From 30 Minutes to Five: Detecting Payment Incidents Faster with ML

A payment service provider approached me with a business-critical monitoring problem. The company was connected to hundreds of payment providers. When one began failing, its monitoring team needed to detect the incident, investigate it, reroute transactions where possible, and notify affected merchants.

Every minute mattered. A delayed response could mean lost transactions, lost revenue, and dissatisfied customers.

The company already had a mature rule-based monitoring system. After several years of improvements, however, it was approaching its practical limits. Some incidents were detected too late, while false alarms created additional work for the monitoring team. At the 95th percentile, incident detection took approximately 30 minutes. To meet the SLA expectations of tier-one customers, the target was less than five.

Why machine learning was the next step

The CTO contacted me after seeing one of my conference talks on anomaly detection. During our first half-day together, we mapped the monitoring process, examined the existing solution, defined success, and identified the simplest technical integration points.

I did not promise that ML would solve the problem. My job is not to convince every company to adopt it. It is to run a disciplined, time-bounded process that produces an honest answer.

Machine learning should be an evolutionary step beyond a mature rule-based system—not a revolution for its own sake.

We agreed to evaluate incident recall, detection time, and false-positive frequency. Detecting anomalies sooner would not help if unnecessary alerts overwhelmed the people investigating them.

Testing safely in production

Several years of monitoring history gave us time-series metrics, recorded incidents, and the outcomes of manual investigations. Once a model showed promising offline results, we deployed it alongside the existing rule-based system in shadow mode. It generated predictions, but its alerts did not yet reach the operational team.

This parallel setup let us compare recall, false positives, and detection time without disrupting a critical payment process. It took about one month to reach the first production deployment.

Why the first two approaches did not work

The first model analyzed changes in payment-provider response latency. It looked excellent offline: after only a few dozen transactions, it could see behavior moving in the wrong direction.

Within several production days, the weakness became clear. It generated far more false positives than expected, so we stopped it before exposing those alerts to the monitoring team.

Offline evaluation tells you whether a model is worth testing. Only production evaluation tells you whether it is worth using.

We then focused on payment acceptance rate, a signal the monitoring team already understood. A simple univariate forecasting model still did not improve the metrics enough. The final approach combined multiple correlated time series and accounted for the distinct operating profile of each payment provider. It forecast expected acceptance rate and treated a sufficiently large negative deviation from the observation as a potential incident.

The third production iteration

Each production evaluation ran for roughly two weeks to collect enough evidence. While one candidate was running, I developed the next, allowing us to keep learning without pausing the development cycle.

The initial engagement was time-boxed to three months. We had not reached the target by then, but the metrics were moving consistently in the right direction, so the CTO approved a further three months. If several genuinely different approaches had produced no movement, my recommendation would have been to stop.

The third production iteration delivered the required result: the 95th-percentile detection time fell from approximately 30 minutes to less than five, while recall and false-positive frequency remained within the required bounds. Feedback from the monitoring team confirmed that the model had highlighted several important situations earlier. The CTO then approved its integration into the main monitoring process.

From one use case to an internal ML capability

Meeting the incident-detection SLA removed an important obstacle to working with tier-one customers. It also gave the company production evidence that ML could create measurable value. Our work later expanded into fraud detection, and the company began building an internal data science team.

I helped interview its data scientists and team lead. As the internal team became more capable, my involvement decreased to occasional consultation. That is one of the outcomes I value most: leaving a client with a working system, an effective team, and the processes to continue independently.