Skip to content
AI Workshop

AI SRE Agents 2026: The Best Model Finds the Root Cause in 56% of Incidents — AIOps, Benchmarks, Costs and How to Roll Out

When production breaks at 3 a.m., the first 30 minutes usually go to the same routine: open the dashboards, read the alerts, check what was deployed, grep the logs. In 2026 that routine is being handed to AI. AWS made its DevOps Agent generally available in March, Microsoft, Datadog and PagerDuty sell their own AI SRE agents, and many teams wire Claude Code or OpenAI Codex straight into Grafana and Kubernetes. But how good are these agents? On the independent ITBench-AA benchmark from Artificial Analysis and IBM, the best model finds every root cause in only 56.2% of Kubernetes incidents, and Anthropic's newest flagship, Claude Opus 5.5, scores 38.2%. This guide shows what AI SRE agents do, which models lead, what they cost and how companies in Croatia, Germany and Austria can introduce them without creating new outages.

56.2%
best ITBench-AA score: share of Kubernetes incidents where the model finds all root causes (GPT-5.6 Sol)
25.3%
best root-cause accuracy of five frontier agents on realistic ORCA-bench oncall tasks
$2M
median cost of one hour of high-impact outage (New Relic 2025)

What an AI SRE agent actually does

AIOps used to mean anomaly detection and alert correlation with classic machine learning. The 2026 version is agentic: a language model with read access to your observability stack that investigates an incident step by step, the way an on-call engineer would. It queries metrics, reads logs and traces, looks at the latest deployments and configuration changes, forms a hypothesis, tests it and writes up what it found. More advanced setups let the agent propose or execute a remediation from an approved runbook.

TaskWhat the agent doesTypical tools in 2026Risk level
Alert triageGroups related alerts, suppresses noise, summarises impactPagerDuty, Datadog, Rootly, incident.ioLow
Root-cause investigationReads metrics, logs, traces and deployments; proposes the root cause with evidenceAWS DevOps Agent, Azure SRE Agent, Datadog Bits AI SRE, NeuBird, Traversal; Claude Code or Codex with MCP serversLow (read-only)
RemediationRolls back a deployment, scales a service, restarts pods from an approved runbookRunbook automation, Kubernetes operators, custom agentsHigh
Post-incident reviewDrafts the timeline and the postmortem from chat, tickets and telemetryIncident platforms, Claude, ChatGPTLow
PreventionReviews changes and configurations for reliability risks before releaseAWS DevOps Agent evaluations, coding agents in CILow to medium

Classification by AI Workshop based on vendor documentation. Tool lists are examples, not recommendations.

Why operations teams want help

The pressure is real. NeuBird's 2026 State of Production Reliability report, a survey of more than 1,000 SRE, DevOps and IT professionals, found that engineering teams spend around 40% of their time on incident management. 78% have had an incident where no alert fired at all, 44% had an outage linked to an ignored or suppressed alert, and almost 40% of incidents are first noticed by customers. 93% pull in at least three engineers for a major incident, working across four to seven tools. New Relic's 2025 Observability Forecast (about 1,700 respondents) puts the median cost of a high-impact outage at $2 million per hour, and use of AI monitoring rose from 42% to 54% within a year.

Production Reliability Pain Points (% of organisations, NeuBird 2026)

Source: NeuBird, 2026 State of Production Reliability and AI Adoption Report (1,000+ SRE, DevOps and IT professionals). "Alerts that need no action" and "incidents first discovered by customers" are typical shares reported in the study, not shares of organisations. NeuBird sells an AI SRE product.

The same survey shows a gap between what leaders believe and what engineers experience. 74% of C-suite respondents say their organisation actively uses AI for incident management; only 39% of practitioners say the same. 57% of executives call their runbooks comprehensive and widely used, against 34% of practitioners. Buying an AI feature is not the same as having AI in the on-call workflow.

How good are the models? The ITBench-AA leaderboard

ITBench-AA is the most useful independent yardstick today. Artificial Analysis built it with IBM Research from IBM's ITBench: 59 Kubernetes incidents (40 public, 19 held out) with alerts, events, traces, metrics, logs and topology. The agent investigates through a shell with up to 100 turns and must name the minimal set of root-cause entities. Scoring is strict: missing any true root cause scores zero, and naming extra suspects lowers the score. Every model runs in the same open-source harness, so the comparison is fair.

ITBench-AA: Share of Kubernetes Incidents Where the Model Finds All Root Causes (selected models)

Source: Artificial Analysis, ITBench-AA leaderboard (accessed 17 November 2026). Average precision at full recall over 59 tasks × 3 repeats, highest effort setting per model, Stirrup harness. 16 of 27 evaluated models shown. Mistral models are not on the leaderboard.

Three findings stand out:

  • No model is reliable yet. OpenAI's GPT-5.6 Sol leads at 56.2%. Even the best agent misses at least one root cause in almost half of the incidents.
  • Newest is not best. Claude Fable 5.1 (49.5%) and Claude Opus 4.7 (46.7%) beat Anthropic's newer flagship Claude Opus 5.5 (38.2%), and OpenAI's GPT-5.6 Sol beats its successor GPT-6 Sol (49.4%). General benchmark rankings do not transfer to your incidents.
  • Open-weight models have caught up. StepFun's Step 5 Preview (55.6%) and Z AI's GLM-5.3-Flash (51.2%, at $0.15/$0.50 per million tokens) are within a few points of the best closed model. That matters for teams that cannot send logs to a US cloud. See our comparison of open-weight and closed models.
ITBench-AA Score by Release Date: Best Closed vs Best Open-Weight Model

Source: Artificial Analysis ITBench-AA leaderboard (accessed 17 November 2026), 27 models released February to September 2026; lines show the running best score. When ITBench was first published in February 2025, the best agent resolved 13.8% of SRE scenarios under a different scoring method.

Progress is fast. Between February and September 2026 the best closed-model score rose from 39.8% to 56.2%, and the best open-weight score from 35.5% to 55.6%. The gap between closed and open models shrank from about four points to less than one.

A harder test: ORCA-bench

ITBench-AA hands the agent a prepared snapshot. ORCA-bench (Gong et al., arXiv 2607.28545) is closer to real on-call work: 1,079 tasks over six days of recorded metrics, logs and traces from a microservice system, starting from vague user reports and investigated through Prometheus, Jaeger and OpenSearch. Across five frontier agents, the best root-cause accuracy was 25.3% on realistic tasks and 10.0% on hard ones, a gap that remained even with Claude Fable 5. The weakest model invented an implausible root cause in 40% of reports, and removing access to source code made every model worse. The authors note that real production systems are far larger than their 50 GB testbed.

Fast or thorough? Turns, cost and accuracy

ITBench-AA: Score vs Average Agent Turns per Task

Source: Artificial Analysis ITBench-AA leaderboard (accessed 17 November 2026). 100-turn cap per task. Colours as in the leaderboard chart.

Longer investigations do not produce better answers. GPT-6 Astra needs only 12.9 turns per task for 48.6%, while Gemini 3.8 Flash uses 82.5 turns for 52.5%. Artificial Analysis observed that models which over-investigate tend to report co-occurring symptoms as root causes. Cost per task varies even more: at launch in May, Claude Opus 4.7 cost $5.38 per task for 46.7%, while Google's open Gemma 4 31B cost $0.14 for 37%. For an agent that runs on every alert, that difference adds up quickly. Our article on AI agent pricing models explains how vendors charge.

The vendors: AWS, Microsoft, Datadog, PagerDuty and build-your-own

AWS DevOps Agent became generally available on 31 March 2026. It investigates incidents, runs preventive evaluations and answers on-demand SRE questions, and connects to CloudWatch, Datadog, Dynatrace, New Relic, Splunk, Grafana, GitHub, GitLab and Azure DevOps. Azure SRE Agent plays the same role in Microsoft's cloud. Datadog Bits AI SRE and PagerDuty's agents sit inside the observability and on-call tools teams already use. PagerDuty's own survey of 1,500 executives found that 81% trust AI agents to act on the company's behalf during a crisis such as an outage, a level of trust the benchmarks do not justify yet.

The alternative is to build: Claude Code or the Claude Agent SDK, or OpenAI Codex, connected through the Model Context Protocol to Prometheus, Grafana, Kubernetes and your ticketing system. This gives you control over the model, the data path and the permissions, and fits teams that already use AI coding agents. Mistral's open models can run on your own servers when logs must stay on-premises, although they have not been tested on ITBench-AA. General-purpose agents such as Manus are useful for research and reporting, but should not get credentials to production systems.

What Gartner expects, and what it warns about

Agentic AI in IT Operations: Today vs Gartner Forecast

Sources: Gartner, Top AI Trends in Infrastructure and IT Operations for 2027 (26 October 2026; 50% minimal human involvement by 2030, 25% NOC/SOC merger by 2030); Gartner Hype Cycle for AI in IT Operations 2026 via The Register (60% deployment and 20% human approval by 2029, 40% disruption by 2028). The disruption forecast applies only to organisations running agentic I&O at scale in production.

Gartner expects 60% of enterprises to use agentic AI in infrastructure operations by 2029, up from fewer than 10% today, and half of organisations to run AI agents in IT operations with minimal human involvement by 2030. Human approval would gate only 20% of AI-suggested actions in 2029, down from 80% in 2025, thanks to deterministic guardrails. The warning comes with it: by 2028, 40% of I&O organisations running agentic operations at scale will suffer a business-critical disruption, up from less than 1% in 2026. Gartner also predicts that, through 2027, companies paying for IT operations agents with uncapped token-based pricing will exceed their initial cost expectations by more than 50%, and recommends a "no telemetry, no deployment" rule.

The economics: cheap agents, expensive outages

AI Agent Cost vs Outage Cost (USD, logarithmic scale)

Sources: AWS DevOps Agent pricing page ($0.0083 per agent-second; AWS examples with 8-minute investigations), New Relic 2025 Observability Forecast (median high-impact outage cost $2M per hour, $1M with full-stack observability). Agent costs exclude charges for queried AWS services such as CloudWatch Logs Insights.

At $0.0083 per agent-second, an 8-minute AWS DevOps Agent investigation costs about $4, and AWS's enterprise example (500 incidents, 40 evaluations and 30 custom agent runs a month) costs $2,365.50. One minute of a high-impact outage costs a median $33,333. If an agent shortens even one major incident by a few minutes, it pays for a year. The risk is on the other side of the ledger: an agent with write access that makes the wrong call can cause the outage. That is why the rollout order matters more than the vendor choice.

A safe rollout ladder: Level 1, read-only summaries of alerts and dashboards. Level 2, root-cause investigation with evidence, reviewed by the on-call engineer. Level 3, the agent proposes a remediation from an approved runbook and a human clicks "run". Level 4, automatic remediation only for a short allow-list of reversible actions (rollback, scale-out), enforced by deterministic policy outside the model, with a full audit log. Move up one level only when the agent's suggestions were right on your last 50 real incidents. Our guide to AI guardrails in production covers the controls.

The EU angle: NIS2, DORA and the AI Act

  • NIS2 incident reporting. Essential and important entities must send an early warning within 24 hours of becoming aware of a significant incident and a notification within 72 hours. An AI agent that assembles the timeline and impact quickly helps meet those deadlines, but a human must own the report.
  • DORA for financial services. Banks, insurers and their ICT providers must classify major ICT incidents and send an initial notification within 4 hours of classification and no later than 24 hours after detection. Keep the agent's findings and actions in the incident record so they can be audited.
  • EU AI Act. AI systems used as safety components in the management and operation of critical digital infrastructure are listed as high-risk in Annex III. A read-only assistant for engineers is usually not, but an agent that autonomously operates critical infrastructure may be. Check before you reach Level 4. See our EU AI Act compliance guide.
  • GDPR and logs. Logs and traces often contain personal data. Mask them before they reach a model, prefer EU-hosted or self-hosted models for sensitive systems and sign a data processing agreement with every vendor.

What this means for companies in Croatia and DACH

  • Start with investigation, not automation. A read-only agent that writes a first root-cause hypothesis saves the most time with the least risk.
  • Test models on your own incidents. Replay ten past incidents and compare models. ITBench-AA shows rankings change by task, and the newest model is not always the best.
  • Fix the telemetry first. Agents are only as good as the logs, traces and runbooks they read. If no alert fired, the agent will not see the problem either.
  • Cap the spend. Prefer predictable pricing or hard budgets, and track cost per investigation from day one.
  • Treat the agent like a new team member. Give it the least privilege it needs, log everything it does and review its work, just as you would for a junior engineer. Our article on AI incidents and risk management shows what happens when this is skipped.

The bottom line

AI SRE agents are already good at the tedious part of incident response: collecting evidence across tools and proposing where to look. They are not yet good enough to be trusted with the fix on their own. The best model gets the full root cause right in 56% of benchmark incidents, and only about a quarter on harder, more realistic tasks. Companies that use agents as fast, well-supervised investigators will cut response times; those that hand them production write access too early risk becoming part of Gartner's 40%.

Want AI to shorten your incidents without causing new ones?

AI Workshop builds AI SRE assistants with Claude Code, MCP and your observability stack, tests models on your own past incidents and sets up the guardrails and audit trails NIS2, DORA and the EU AI Act expect. Our Claude Certified Architects keep your logs in the EU.

Talk to an AI consultant

Boris Agatić

Founder of AI Workshop and Claude Certified Architect. Over 10 years of building software, guiding companies from the first strategy call to production AI agents.

LinkedIn (opens in a new tab)