AI Inference / Performance Engineer
Open a video or resource link, watch it, then check its box to track your progress and unlock what comes next.
- Understand inference mechanics and establish a defensible performance baseline
- Profile bottlenecks and tune the inference stack systematically
- Scale distributed inference and push the performance frontier
Benchmark: OpenAI + Anthropic + NVIDIA inference/performance roles (Sep 2026): optimize frontier-model serving end to end; profile bottlenecks across applications, models, kernels, accelerators, networking and fleet; tune batching, KV cache, quantization, parallelism and serving runtimes; deliver reproducible latency, throughput, utilization and cost gains with correctness and reliability.
Who’s Hiring Right Now
US openings posted in the last 7 days at Anthropic, OpenAI — this role’s own benchmark employers — refreshed daily, as of 2026-11-01. Pulled directly from each company’s public job board; not exhaustive, and postings close fast — worth confirming the details on the employer’s own site before applying.
Understand inference mechanics and establish a defensible performance baseline
Model-serving architecture, prefill/decode + latency/throughput economics
Coverage: AI Compute, Inference & Performance / Model Serving / MLOps
GPU execution, CUDA memory hierarchy + hardware-aware optimization
Coverage: CUDA / GPU Architecture / HPC / AI Accelerators
Profile bottlenecks and tune the inference stack systematically
Profiling, tracing + kernel/runtime bottleneck analysis
Coverage: Nsight / Profiling / Observability / Performance Engineering
KV cache, batching, quantization + serving-engine optimization
Coverage: TensorRT-LLM / vLLM / SGLang / Quantization / KV Cache
Scale distributed inference and push the performance frontier
Multi-GPU/multi-node parallelism, NCCL + production serving
Coverage: Distributed Systems / NCCL / NVLink / Networking / Serving Infrastructure
Kernel optimization, scheduling + reproducible performance economics
Coverage: CUDA Kernels / Compilers / Benchmarks / Capacity / Cost / Energy
Role-Readiness Proof / Capstone
Take a real LLM or VLM from baseline serving to an optimized production reference: define TTFT, inter-token latency, throughput and cost SLOs; benchmark a single-GPU baseline; profile with Nsight; quantify compute, memory and network bottlenecks; tune batching, KV cache and quantization; scale across GPUs; test a kernel/runtime optimization; add autoscaling and observability; validate correctness; publish a reproducible performance report with cost-per-token tradeoffs.
Proof Rubric
- Use controlled, reproducible benchmarks
- distinguish compute-, memory- and network-bound behavior
- connect profiler evidence to code/config changes
- improve latency, throughput and utilization without correctness regressions
- reason about tail latency and SLOs
- quantify cost and energy implications
- document when an optimization stops paying off