AI Evals & Monitoring
Not Started0 / 10 complete
Open a video or resource link, watch it, then check its box to track your progress and unlock what comes next.
10 lessons3 levels1 capstone project
Measure and improve probabilistic AI systems using golden datasets, human review, LLM-as-judge, retrieval and agent metrics, tracing, regression testing, online evaluation, drift detection and production observability.
BASIC
- Instrumenting & Evaluating LLMs — Hamel Husain (30 min selected)
World-class applied evals mental model: error analysis, unit tests, human eval, LLM-as-judge, traces and agent evaluation. - LangSmith Tutorial: LLM Evaluation for Beginners — Dave Ebbelaar (25 min selected)
Practical introduction to tracing, datasets, experiments and evaluation workflows with a production tool. - AI Observability Overview — Arize AI (10 min)
Connects evaluation to production monitoring, issue detection, diagnosis, drift and operational feedback loops.
INTERMEDIATE
- LLM Observability, Evaluation & Experimentation — Arize
Apply AI Evals & Monitoring through the adjacent capability: LLMOps. - Confidently Iterate on LLM Apps — W&B Weave
Apply AI Evals & Monitoring through the adjacent capability: LLMOps. - Complete Guide — n8n
Apply AI Evals & Monitoring through the adjacent capability: Workflow Automation.
ADVANCED
- Debug, Evaluate & Ship Reliable AI Agents — LangChain
Extend AI Evals & Monitoring into senior/production practice through LLMOps. - Beginner Automation Tutorial — n8n
Extend AI Evals & Monitoring into senior/production practice through Workflow Automation. - AI Infrastructure on Kubernetes: Run:ai & KAITO — Microsoft / NVIDIA
Extend AI Evals & Monitoring into senior/production practice through AI Factory.