How a Big Four MSSP went from an hour per alert to 5.3 minutes

The managed security arm of a Big Four professional services firm runs security operations for a large base of enterprise clients, and it does so on a stack most security teams would envy. Even so, until recently, an alert that deserved a proper look was costing one of its analysts somewhere between 45 minutes and an hour.

With Armor OS agents working the queue across 20+ of its customer environments, that same investigation now takes 5.3 minutes on average, and every alert gets one as soon as it fires. Nothing about the tooling changed. The investigation work moved to agents, and the analysts moved to the alerts that actually needed them.

The setup

An MSSP carries the enterprise SOC's problems many times over. Each client comes with its own log sources, naming conventions, allowlists and escalation paths, and an analyst who moves from one client's queue to another's is effectively switching environments every time. In this case the client stacks ran Google SecOps, Microsoft Sentinel and CrowdStrike EDR, and across them the firm was handling more than 100,000 alerts.

The expensive part was not the volume by itself. It was what each investigation involved. For any alert worth looking into, an analyst had to work out which queries to run for that particular client's stack, pull the relevant logs from each source, line the results up against each other, and then write a verdict that could be defended to the client. Done properly, that took most of an hour, and in an MSSP the hour comes straight out of the margin on the contract.

The firm had tried the usual responses. Adding analysts helped for a while, until the next client onboarded. Playbooks helped with the alert types the team had seen before and did little for the rest. Tuning down noisy rules reduced the queue at the cost of coverage. None of it shortened the investigation itself, which is where the time was going.

What changed

The firm put Armor OS investigation agents on the live queue. For each alert, the agents work through the same steps an analyst would, without waiting for someone to pick the alert up.

Plan. The agents build the investigation steps from the alert type and the specific customer environment it fired in. A new-device sign-in at one client and the same alert at another get different plans, because the environments and the data behind them are different.

Execute. The steps run at runtime. The agents write the queries, pull the logs from Google SecOps, Sentinel and CrowdStrike, and correlate across them. Nothing is pre-wired per client; the investigation happens against that client's actual data at the time the alert fires.

Verdict. The alert comes back as a true positive, a false positive, or suspicious, with the evidence that led there attached. An analyst can open any closed alert and see what was checked and what was found.

For an MSSP, that evidence trail was a condition of using the system at all. Closing an alert on a client's behalf means being answerable for the closure, so a fast verdict with nothing behind it would not have been usable no matter how fast it arrived.

The results

Across the queue, 78% of alerts came back as false positives, 12% as suspicious and 10% as true positives.

The 78% is where most of the hour had been going: alerts that needed a real investigation just to be confirmed as noise. Those now close with the evidence attached and without an analyst involved. The 10% true positives arrive already investigated, so response starts from the full picture rather than from a hunch. The 12% flagged as suspicious are the alerts the agents deliberately hand to a person, because the evidence was ambiguous enough that a human call was the right next step. The analysts still get those, but with the investigation already done rather than still to do.

In practice that 12% is what the analysts' day now looks like. Roughly one alert in eight reaches a person, and it reaches them with the work attached.

"The intelligent prioritization is game-changing. We went from drowning in 10,000 alerts to focusing on the 50 that actually matter." Director, Large MSSP

What other MSSPs can take from this

Most MSSPs with a serious stack are sitting on the same hour, for the same reason. The investigation is manual, and it is manual separately for every client, so the cost scales with the client base rather than with the tooling.

A few things from this deployment are worth noting for anyone in a similar position. The first is that the stack was not the constraint. This customer already had three of the strongest platforms on the market and was still spending an hour per alert, and a fourth tool would have added another console to query rather than removed any of the work. The second is that the per-client variation, which is usually the reason automation projects stall at an MSSP, turned out to be something the agents handle as a matter of course, because they plan each investigation for the environment the alert came from. The third is that the evidence trail is what made the speed usable in a managed service context. A five-minute verdict a client cannot see the reasoning for is a liability. A five-minute verdict with the reasoning attached is something a client can be shown.

Across those 20+ environments, the firm's analysts remain the people who decide what happens on the hard alerts. They no longer spend the hour getting there.

Download the two-page case study: From 45-minute investigations to 5.3 minutes, across 20+ MSSP customers