Technical Program Manager, AI Inference
Open a video or resource link, watch it, then check its box to track your progress and unlock what comes next.
- Architecture + execution
- Capacity + launch
- Operate + scale
Benchmark: Anthropic Technical Program Manager, Cloud Inference + OpenAI compute/inference program signals (current Sep 2026) Market refresh: inference/model-serving architecture; model onboarding/release pipelines; cloud partner execution; accelerator/capacity readiness; latency-throughput-cost SLOs; dependency/roadmap management; launch readiness; incidents/rollback; engineering coordination and executive cadence.
Who’s Hiring Right Now
US openings posted in the last 7 days at OpenAI — this role’s own benchmark employers — refreshed daily, as of 2026-11-01. Pulled directly from each company’s public job board; not exhaustive, and postings close fast — worth confirming the details on the employer’s own site before applying.
Architecture + execution
Inference architecture + serving
Coverage: Prefill/decode / GPU memory / model servers / routing / autoscaling
Program plan + model onboarding
Coverage: Critical path / dependencies / RACI / release criteria
Capacity + launch
Performance + capacity program
Coverage: TTFT / ITL / throughput / GPU utilization / cost / headroom
Cloud deployment + partner coordination
Coverage: KServe / deployment automation / compatibility / blockers
Operate + scale
Readiness + incidents
Coverage: Go/no-go / load tests / rollback / incident response
Executive risk + roadmap
Coverage: RAID / capacity scenarios / source-of-truth dashboards
Role-Readiness Proof / Capstone
Run a managed-inference program from model onboarding to launch: architecture/dependency map, TTFT/ITL/throughput/cost SLOs, capacity scenarios, integrated roadmap, launch review, simulated shortfall/incident, dashboard and executive decision memo.
Proof Rubric
- Inference fluency
- dependency-complete plan
- measurable gates
- cloud/model readiness
- RAID discipline
- incident/rollback
- quantified latency, throughput, utilization, headroom and cost
- crisp executive communication