Learn · updated 2026-10-10

Glossary: terms for training and testing AI agents

Short definitions of the terms used when training and testing AI agents in environments, from RL environment and verifiable reward to grader, seeded reset and audit log. Each term has a stable link you can share.

RL environment: A simulated system where an AI agent takes actions and receives a score for completing a task correctly. Labs use the score as a reward in training; teams use the same environment as a test before deployment. More →
Verifiable reward: A reward computed by checking the final state of a system against written rules, so anyone rerunning the same seed gets the same score. Contrasts with a score from an LLM judge reading the transcript. More →
Agent evaluation sandbox: An isolated system where an AI agent can act without touching real data or money. An environment is a sandbox with tasks and graders inside. More →
AI agent test environment: An environment used to test an agent before deployment: a working copy of the systems it will touch, tasks built from the workflow's rules, and a grader that checks what the agent changed. More →
Episode: One attempt by an agent at one task: the environment resets to a seeded state, the agent acts through tools, and the grader scores the result.
Task: A piece of work the agent must complete, written against an operating manual, with a known correct outcome the grader can check.
Seeded reset: Rebuilding the environment's starting state from a seed number. The same seed always produces the same state; different seeds vary the details. AutoStack episodes are built from a seed in about 6 ms. More →
Grader: The code that scores an episode. AutoStack graders are deterministic: they read the final system state and the audit log and check them against written rules. More →
State check: A rule the grader tests against the final state of the system, such as a reserve being set before a payment. Grading on state checks scores what the agent did, not what it said. More →
Audit log: The record of every action the agent attempted during an episode, including attempts the system rejected.
Rejected attempt: An action the system refused, such as a payment over the agent's authority. AutoStack graders count rejected attempts against the agent even when no money moved.
LLM judge: A language model asked to score an agent's transcript or output. Quick to set up, but it can be persuaded by an agent describing an action it never took. More →
Benchmark: A fixed set of tasks run the same way for every model so results can be compared. A benchmark is usually built on an environment. More →
Shortcut agent: An agent that tries to score without doing the task, for example by doing nothing or escalating everything. Good graders fail shortcut agents; AutoStack's fail them on every task.
Act-or-escalate twin: Two near-identical tasks where the right answer differs: in one the agent should act, in the other it must hand the decision to a human.
MCP (Model Context Protocol): An open protocol for serving tools to AI agents. AutoStack environments serve their tools from an MCP server backed by a real database. More →
Tool call: One request from the agent to a tool. Read tools return information; write tools change the system's state, which is what the grader checks. More →
Payment authority: The largest amount an agent or adjuster may pay without approval. Above it, the correct action is to escalate. More →
Reserve: The amount an insurer sets aside for the expected cost of a claim. In the claims environment, a reserve must be set before a payment and stay consistent with what is paid. More →
SIU referral: Sending a claim with fraud indicators to the Special Investigations Unit. Nothing irreversible should happen to the claim before the referral. More →
Evidence pack: What an AutoStack evaluation delivers for each failure: the rule broken, expected vs actual, the dollar impact, severity and a replay command. More →
Replay: Rerunning a recorded episode from the same seed to reproduce a failure exactly.