Glossary of RL environment terms

The set of actions an agent can take in an environment. For LLM agents it is usually a list of tools with typed arguments.

See also Tool call, RL environment

A structured attempt to make a verifier accept work that should fail, run before the verifier is used for training or scoring. Each strategy that gets through is logged and fixed.

See also Verifier, Reward hacking

A fixed set of tasks with a scoring rule, used to compare systems. Its value depends on the tasks staying unseen by the systems it measures.

See also Held-out split, Saturation

A piece of evidence a program can check quickly, such as a feasible solution or a proof that none exists. Used to grade optimisation answers.

See also Verifiable reward

When evaluation tasks, or their answers, leak into a model’s training data, so a score measures memory rather than skill.

See also Held-out split

A way of scoring without doing the task: taking no action, reusing stale evidence, exhausting the budget, or producing output that only looks correct. Used to test verifiers.

See also Adversarial audit

One attempt at a task, from the environment’s reset to the point where it ends. Training runs many episodes.

See also Rollout, Trajectory

A recorded episode performed by a human professional, including decisions, rationales, mistakes and corrections. Used as training data and as a reference.

See also Trajectory

When a measure becomes a target, it ceases to be a good measure. The economic statement of the reward-hacking problem.

See also Reward hacking

The standard Python interface for RL environments, maintained by the Farama Foundation: reset() starts an episode and step(action) returns an observation, a reward and termination flags.

See also RL environment

Tasks kept out of training and out of public view, used only for evaluation.

See also Contamination, Benchmark

The score the best human strategy reaches on a task, measured the same way as the model.

See also Human baseline

The score the best human achieves on the same task, measured the same way. The most informative reference point for a model’s score.

See also Benchmark

Using a language model to grade another model’s output. Flexible, but a learned judge can be influenced by the text it grades.

See also Verifier, Reward model

A task that takes many steps and tool calls, where early decisions constrain later ones and errors compound.

See also Episode

A verifier that grades by executing the rules of the domain, such as a matching engine, an accounting identity or a proof checker, rather than by opinion.

See also Verifier, Settlement

The part of the environment’s state that the agent is allowed to see at a given step.

See also State

The record of what happened when agents acted in an environment: decisions, mistakes, recoveries and verified results.

See also Trajectory

The strategy an agent follows to choose actions. For an LLM agent, the model plus its prompt and tools.

See also Reinforcement learning

A rule that must hold at every step of a workflow, not just at the end. Catches agents that reach the right final state the wrong way.

See also Verifier

Training an agent by letting it act, observing the reward that follows, and updating its policy to earn more reward.

See also Policy, Reward

The number an environment returns to say how well the agent did. What the agent learns to maximise.

See also Verifier

Scoring well by exploiting a flaw in the grader instead of doing the intended task. Also called specification gaming.

See also Goodhart’s law, Adversarial audit

A model trained, usually on human preferences, to predict a reward. Used in RLHF.

See also RLHF, LLM-as-judge

A world an agent acts in and is graded by: state, observations, actions, transition rules and a reward.

See also Gymnasium API, Verifier

Reinforcement learning from human feedback: training against a reward model learned from human preference judgments.

See also Reward model, RLVR

Reinforcement learning with verifiable rewards: training against rewards computed by programmatic checks such as answer keys or test suites.

See also Verifiable reward

Running a policy through an environment to produce an episode, usually many in parallel during training.

See also Episode

When the best systems score at or near the maximum, so a benchmark can no longer tell them apart.

See also Benchmark

Deciding an outcome by the real rules of the domain, as an exchange settles a trade. A settled outcome needs no judge.

See also Mechanistic verifier

DeepMind’s term for an agent satisfying the literal specification of an objective without achieving the intended outcome.

See also Reward hacking

Everything that is true about the environment at a moment in time, whether or not the agent can see it.

See also Observation

A temporary account for amounts not yet explained. It should close at zero; a balance left there hides an unresolved difference.

See also Trial balance, Process invariant

A structured action an LLM agent takes, such as running a query or posting a journal entry, with arguments the environment can validate.

See also Action space

The full sequence of observations, actions and rewards in an episode.

See also Episode, Expert trajectory

A list of every ledger account with its debit or credit balance. Debits equal credits in a balanced book, which proves arithmetic, not that the work was done.

See also Suspense account

A reward that a deterministic check can compute without asking another model for an opinion.

See also RLVR, Verifier

The component that decides the reward. The part of an environment most worth attacking before use.

See also Mechanistic verifier, Adversarial audit

SimReal’s word for an RL environment built around one area of real work, together with its verifier, tasks and human baseline.

See also RL environment, Verifier