205: Testing with Agents
What does a test suite prove when an agent writes it and a model runs in it? This chapter tests at project scale, where agents write and run much of the suite and a language model may sit inside the product under test. It teaches four habits for that suite. A suite reaches past its assertions, so force the isolated, deterministic setting in the test helper and choose the tests from what the change can reach. A model that plays or grades the product is an instrument with sampling error, so every report prints its sample and every experiment's design is committed before any output exists. A guarantee about model output belongs where the model cannot get past it, in a schema or in code at the seam. And a model's version is configuration with one home, while its cost is a measurement that names its convention.
Materials
This Chapter's Insights
- ACI-044: The Suite Has a Blast Radius
- ACI-045: Measuring with a Stochastic Instrument
- ACI-046: The Seam That Takes Model Output
- ACI-047: Model and Cost Are Configuration and Measurement
Browse every insight on the course map.