Eval

An eval is a test that measures an AI system's quality on a specific task — running the system against a labeled dataset and scoring its outputs against expected results.

Evals are how you know an AI system is getting better, not just changing. Without evals, every prompt edit is a gamble. With evals, you A/B test prompts, models, and tools the same way you'd unit-test code. Most production agentic systems run nightly evals against pinned scenarios.

Example

Acme's AI Support eval has 200 historical tickets with the 'right' resolution. After every prompt change, the eval runs and reports % match — a regression below 90% blocks ship.

How OpenLabor uses it

OpenLabor employees come with starter eval suites and let you add custom evals from your own data.

How many eval cases do I need?

50 well-chosen examples beats 5,000 noisy ones. Start small, expand when you find failure modes the small set misses.

Related: llm, agentic-workflow.

AI Labor Glossary