Checks

A check is one assertion about the end state or the agent’s output. A task’s template checks always run, unless the request sets replace_template_checks to true. Add your own in checks on POST /v1/runs. The API gives a custom check an id and sets its source to custom.

Kind	Needs	Passes when
`record_exists`	`collection`, `where`	A record in the collection matches `where` at the end.
`record_absent`	`collection`, `where`	No record matches.
`field_equals`	`collection`, `field`, `value`	A matching record has `field` exactly equal to `value`.
`field_contains`	`collection`, `field`, `value`	A matching record has `value` in `field`, case-insensitive.
`write_count`	`collection`, `min`, `max` (all optional)	The number of records created, updated and removed is within the range.
`no_writes`	`collection` (optional)	That number is zero.
`action_called`	`action`	A call in the trace has that operation.
`action_not_called`	`action`	No call has that operation.
`output_contains`	`value`	The output sent to finish contains `value`, case-insensitive.
`output_equals`	`value`	The output equals `value`.
`rubric`	`text`	A language model answers yes or no to `text`, given the task, the trace, the state diff and the output.

where is an object of field to value. A string matches as a case-insensitive substring. Other values must be equal. Fields can be dotted paths such as start.date. Collection and action names are the environment’s own. A template task’s checks show the names in use. For offsite-planning the collections are events, slack_messages, messages, drafts, labels, channels, users and calendars. Use rubric only for what a fixed check cannot decide. If no judge model is available, a rubric check is undecided: passed is null, it counts as not passed, and the run still ends done.

Get started

Concepts

More