Model Serving, Inference Ops & Observability

This course covers the operation of production inference services on a commissioned GPU cluster, distinct from model development or training. Students deploy a model behind an inference server (e.g., NVIDIA Triton or a vLLM-based server), configure dynamic batching and concurrent model execution to raise throughput, and load-test a serving endpoint to characterize latency versus throughput trade-offs at varying batch sizes. The course covers observability for inference workloads: collecting and interpreting tokens-per-second, time-to-first-token, and request-queue-depth metrics; building Grafana dashboards fed by DCGM Exporter and application-level metrics; and configuring alerting thresholds for latency SLO breaches and GPU memory exhaustion. Students practice canary and blue-green rollout patterns for model version updates, roll back a failed deployment, and diagnose a simulated production incident (e.g., request queue backing up under load) using the observability stack built during the course. The course emphasizes operating and monitoring inference services reliably, not model training or fine-tuning.