About
Daniel Begimher is a Senior AI Security Engineer at AWS who led development of AWS's first AI security agent from proof-of-concept to production GA. With 13+ years in cloud security and incident response, he has red teamed Amazon Q Developer, co-authored RAG poisoning research, and presented at AWS re:Invent and re:Inforce. He holds CISSP, OSCP, and all 12 AWS certifications.
Sessions
SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agents
What you will learn:
•Investigation vs. Classification: Learn the critical difference between an AI that correctly triages alerts (97.1%) and one that conducts genuine forensic investigation (41.9% novel finding coverage)—and why both metrics matter for production deployment •Adversarial Evaluation Design: Implement an LLM-as-Judge that inverts the burden of proof, preventing the confirmation bias that accepts alert repetition as valid investigation •Realistic Benchmark Generation: Use the OUAT methodology to create measurable ground truth from real incident patterns without exposing sensitive production data •Performance by Attack Category: Understand why Unauthorized Access investigations yield deep findings (47.9% hit 7+ novel discoveries) while •Malicious File Execution struggles (1.9%)—and what this means for agent deployment decisions •Production Readiness Framework: Apply the M1/M2/M3 metric framework to evaluate whether your AI security tools are performing genuine investigation or sophisticated pattern matching