Create AI evaluations

An introduction to AI Evals: why we need them, and how to create them.

What to expect from this series, and what you should know before you start.

Mapping your web testing knowledge to the world of large language models.

Define what good and bad looks like for your AI application.

Automate the basics. Use code to catch simple errors.

Get your subjective evaluations running with a basic judge model.

Finish setting up your basic judge model to get your subjective evaluations running.

Applied engineering tips to build your AI testing pipeline.

Structure your testing into layers.

A non-exhaustive list of sources used in this course and eval tools that can help you.