Learn
Learn: training and testing AI agents
Plain explanations of reinforcement learning (RL) environments, agent testing and grading, written by the people who build AutoStack's environments. Every example comes from a real episode in our insurance claims environment.
Guides · step by step
Guides
Methods you can follow: how to test an agent before deployment, and how to evaluate a claims agent before a pilot.
- 01How to test AI agents before deploymentHow to test an AI agent before deployment: a working copy of its systems, tasks from your rules, grading on final state, repeated runs and replays.Riya Arora · 2026-10-10
- 02How to evaluate a claims AI agentHow to evaluate an insurance claims AI agent before a pilot: what to test, how to grade the claim file and audit log, and what insurers should ask vendors.Riya Arora · 2026-10-10
Articles · explainers
Articles
The ideas behind the method: what an RL environment is, and why we grade state instead of text.
- 01What is an RL environment for AI agents?An RL environment is a simulated system where an AI agent acts and gets a verifiable score. How environments, benchmarks, evals and sandboxes differ.Bhavya Bakshi · 2026-10-10
- 02LLM as a judge vs deterministic gradingLLM-as-a-judge vs deterministic grading for AI agents: what each checks, where each fails, and when to use state checks on the final system state.Bhavya Bakshi · 2026-10-10
Glossary
Terms, defined.
RL environmentA simulated system where an AI agent takes actions and receives a score for completing a task correctly.Verifiable rewardA reward computed by checking the final state of a system against written rules, so anyone rerunning the same seed gets the same score.Agent evaluation sandboxAn isolated system where an AI agent can act without touching real data or money.AI agent test environmentAn environment used to test an agent before deployment: a working copy of the systems it will touch, tasks built from the workflow's rules, and a grader that checks what the agent changed.EpisodeOne attempt by an agent at one task: the environment resets to a seeded state, the agent acts through tools, and the grader scores the result.TaskA piece of work the agent must complete, written against an operating manual, with a known correct outcome the grader can check.Seeded resetRebuilding the environment's starting state from a seed number.GraderThe code that scores an episode.