Environments, runs and results - testworld

An environment is a set of simulated tools seeded with data. Every run starts from the same saved state. List them with GET /v1/environments. A run is one agent doing one task in one environment. It is created, runs while the agent calls the tools, ends once, and is then evaluated.

Status	Meaning
`created`	No call has reached the tools yet.
`running`	The first call has been answered.
`evaluating`	The run has ended and the checks are being evaluated.
`done`	The evaluation is on the run.
`failed`	Scoring itself broke. The trace is still there.

The trace is the ordered record of every call the tools answered, reads included. Nothing from inside the agent is in it. The evaluation compares the end state and the agent’s output with the run’s checks. The score is the share of checks that passed, from 0 to 1.

How a run ends

ended_by says how:

finish: you called POST /v1/runs/{id}/finish.
idle_timeout: no call for idle_timeout_seconds (default 600).
timeout: timeout_seconds have passed since creation (default 3600).
max_calls: the run reached max_calls (default 500).

Every ending is scored, including a run that never made a call.