AI-Assisted Detection Rule Builder

An AI-assisted way to create detection rules in OpenSearch Security Analytics. Instead of stepping through configuration, analysts start from the intent and iterate on top of agent's work.

Some visuals and product details have been recreated, simplified, or omitted to respect confidentiality and intellectual-property requirements. The case study focuses on my design process, decisions, and contributions rather than confidential product information.

The problem

Writing a detection rule had quietly turned into a full-time job. To get one running you had to hand-author a Sigma rule or query, point it at the right telemetry source, set up suppression logic, and map MITRE ATT&CK techniques — all before you got to find out whether it caught anything real without flooding the queue with noise. Every step ahead of that first signal is friction, and friction is where people give up.

The challenge

This was the second half of 2025, and where the technology stood at the time really shaped the problem. The question analysts brought to an agentic interface back then was "can I trust what this thing is showing me?" — trust in the artifact itself: is this query right, are these findings real, did it actually understand what I asked for? Everything below comes out of designing for that question.

Agent quality wasn't reliable enough for full autonomy. Analysts needed real control over everything the agent produced — queries, metadata, suppressions — and the architecture had to let us drop those checkpoints later, as models improved, without starting over.

The agent couldn't feel bolted on. Only rule creation was becoming agentic — findings triage, suppressions, and rule management all stayed traditional UI. So the interaction had to feel native to the product rather than a chat panel stapled onto an existing page, and analysts needed an easy way back to doing it themselves.

Two UIs meant double duty. Our obvious first move was side-by-side: the agentic interface next to the traditional form. Testing put an end to that quickly. Participants couldn't tell where their own judgment was needed and where they were just watching an agent fill in the same fields, slightly worse than they would have. That reframed the whole problem for us: how do you let the agent own the workflow while keeping the person meaningfully in control?

Intent first, not form first

It starts with a plain-language description of what you're trying to catch. From there the recommendations get progressively sharper — broad at first, drawn from your connected telemetry and MITRE coverage gaps, then narrowing as your intent gets specific. It flips the wizard around: you say what to detect, and the system figures out how to configure it.

I also pushed back on an inherited habit: asking for all the metadata upfront, a leftover from API design and form-based UI. Severity, MITRE mappings, observables, and descriptions all get inferred from the first prompt and refined in conversation instead. Nobody should be stuck at the door filling in fields the agent can work out on its own.

The agent gets the primary real estate

Instead of adding a chat panel to the existing form, we let the agentic interaction own the interface. Manual creation — starting from a query or a Sigma spec — is a full alternative mode rather than a second track running alongside. One mode at a time, so it's always clear where you are and whose judgment is driving.

Fast-track to the testing loop

The agent turns that first intent into a complete draft rule and gets you to testing as fast as it can. Research was very clear about why: analysts decide whether the agent understood them by looking at findings from historical data, not by reading an explanation. Real findings and real signal-to-noise build confidence far faster than anything the agent could say to reassure you.

The query stays visible and editable the entire way through. The agent drafts a broad template and suggests refinements, but the query itself stays deterministic and open to inspection. This was the single strongest thing we heard: analysts will not give up sight of the query behind the rule.

What testing taught us

Beyond ruling out the side-by-side model, the sessions surfaced a need the rule builder alone couldn't answer. Before writing anything, analysts want to know what coverage they already have and which threats are emerging, so they can decide what's worth writing and tuning next. Rule creation sits downstream of that judgment — which is why the entry point opens with recommendations grounded in your telemetry and coverage gaps instead of an empty prompt.

Outcome

• Design partners — real security teams — were enthusiastic about having threat intel and query building in one continuous flow instead of two separate tools.
• Rule creation got faster, mostly by taking form-filling out of the path to a testable rule.
• An interaction model with room to grow: the human checkpoints can collapse as the agent improves, without redesigning the experience.

What I'd do differently

I'd work out a trust calibration framework for the whole product before shipping the first agentic feature. Rule creation happened to go first, but findings triage, suppression management, and the rest are heading the same way. One consistent pattern language — for the moments human judgment is needed, for telling agent-generated work apart from human-authored work, for how autonomy levels get communicated — would give analysts a single coherent mental model as more of the product turns agentic. Without it, every feature invents its own trust patterns and the whole thing starts to feel incoherent.

That need has only gotten sharper since. The question has moved from "can I trust what this interface is showing me?" to "can I trust this system to act for me?" — from checking an artifact to handing over a decision. Our progressive autonomy architecture saw the shift coming, but it was built to collapse human checkpoints, not to answer the harder question sitting underneath them: what does an analyst need to see, once, to stop checking? That's the problem I'd love to take on next.

Progressive autonomy without a rebuild

Autonomy here is a spectrum, not a setting. Today the agent recommends and the analyst confirms at a few key moments. As models get better, those moments collapse — fewer confirmations, more agent authority — using exactly the same interaction primitives. The fast-track path (intent → full draft → test right away) is already the high-autonomy version of the flow that step-by-step conversation gives you at the low end.