AI Evaluation, Safety & Reliability Engineering Bootcamp Certificate

Checkpoint 1: Define the system success contract, explicit acceptance criteria, failure taxonomy and a representative gold-set dataset before changing prompts or models.
Checkpoint 2: Implement deterministic assertions plus model-based graders; calibrate automated scoring against a human-labeled sample and document disagreement cases.
Checkpoint 3: Evaluate retrieval separately from generation; add groundedness, answer-quality, segment and edge-case analysis with traces that locate the failure stage.
Checkpoint 4: Evaluate agent plans, handoffs and tool calls; add adversarial cases for prompt injection, excessive authority and unsafe actions with explicit approval boundaries.
Checkpoint 5: Build a regression suite comparing model/prompt/tool configurations on quality, safety, latency and cost; define pass/fail thresholds and launch gates.
Checkpoint 6: Instrument production traces and online quality signals; simulate a regression, investigate it, roll back safely and capture the incident plus improvement loop.

Day 1: Choose a real AI workflow; define users, task-success criteria, safety boundaries, prohibited behavior, uncertainty and the decision the evaluation must support.
Day 2: Create a versioned eval dataset with normal, edge, ambiguous and adversarial cases; manually label a calibration subset and document dataset limitations.
Day 3: Build deterministic checks and model graders; measure grader agreement, false positives/negatives and explain where human review remains necessary.
Day 4: Add RAG or agent/tool evaluation: trace retrieval, reasoning and actions separately; create a failure taxonomy and diagnose representative failures.
Day 5: Run red-team and robustness tests including injection/jailbreak/tool-abuse cases; implement mitigations and re-run the same benchmark to prove improvement.
Day 6: Wire evals into CI/release gates; compare two configurations on quality, safety, latency and cost and trigger a simulated rollback from a failed regression.
Day 7: Publish repo, eval dataset/sample labels, grader design, regression report, red-team evidence, dashboard, demo, safety/reliability case and operations runbook.