Synchronous model loads run inside the shared workflows thread pool, stalling all concurrent workflow runs (latency collapse with idle GPU/CPU)

[imbgar-roboflow]

Summary

On a GPU inference server (roboflow-inference-server-gpu, build of 2026-06-05 / v1.3.0 era) serving multi-model Workflows under sustained production traffic, we observed a latency collapse mode where p50 latency of all workflow endpoints rises 3–10x while accelerator duty cycle stays ≤13% and CPU stays under 25% of limit. Raw /infer/* endpoints on the same pods remain fast. Root cause analysis points at a structural issue: synchronous model loads (3–19s each) execute inside the shared workflows ThreadPoolExecutor, starving step execution for every concurrent workflow run in the process.

Environment

• gpu_http:app via uvicorn, 1 worker, defaults: HTTP_API_SHARED_WORKFLOWS_THREAD_POOL_ENABLED=True (16 workers), WORKFLOWS_MAX_CONCURRENT_STEPS=4, MAX_ACTIVE_MODELS=20, MEMORY_FREE_THRESHOLD=0.20
• Workload: ~6 workflow variants chaining 3–5 models each; working set ~12 model/version combos (~600MB VRAM each) on NVIDIA L4
• Horizontal autoscaling on request rate; replica churn redistributes traffic across pods regularly

Observed

• Model loaded: ... load_time= events: p50 ≈ 5s, p90 ≈ 11s, occurring continuously in steady state (~200–700/hr fleet-wide) because the model working set + MEMORY_FREE_THRESHOLD VRAM guard keeps the WithFixedSizeCache manager evicting (reason=memory_pressure) and re-loading
• When autoscaler churn raises per-pod cache-miss rates, workflow p50/p90 explodes (0.8s → 4–8s p90 across all workflow endpoints uniformly) while GPU duty cycle ≤13% and CPU well under limit — classic heavy-tailed-service queueing on a software bottleneck (E[S²] dominated by inline loads), not a hardware one
• /infer/* on the same pods degrades only mildly (~2x p90 at worst) — consistent with it not using the shared workflows pool, only the model-manager lock

Code path

• inference/core/interfaces/http/http_api.py (~L1193): one process-global self.shared_thread_pool_executor = ThreadPoolExecutor(max_workers=16) is created and passed to every ExecutionEngine.init(...) (~L1293)
• Model blocks execute as steps in that pool; a cold model triggers add_model → download + deserialize + VRAM upload (3–19s) inside a step worker, while also holding the WithFixedSizeCache queue lock for eviction bookkeeping (inference/core/managers/decorators/fixed_size_cache.py), so concurrent runs both lose a pool worker and contend on the lock
• Under churn, several in-flight loads consume a large fraction of the 16 workers; queued steps back up; every workflow run slows; in-flight requests then pile up to whatever the ASGI concurrency limit is, producing 503 load shedding while the GPU is idle

Asks

1. Isolate model loading from step execution — run loads on a dedicated loader executor (or async path) so a cold model only blocks requests that need that model, never the shared step pool
2. First-class model pre-warming: a way to declare the deployment's model set to be loaded at startup (and pinned), with readiness gated on warm-up, so steady-state inline loads approach zero (the _pinned_models machinery in WithFixedSizeCache looks close but isn't exposed/documented for this)
3. Hysteresis / actively-used protection on memory_pressure evictions — under the VRAM threshold, LRU eviction of models that are in active rotation creates a permanent evict↔reload cycle
4. Observability: counters/histograms for model load duration, cache miss rate, and shared-pool queue depth would have made this diagnosable from metrics instead of log archaeology

Happy to provide more detail on the analysis.

[imbgar-roboflow]

Additional finding on why eviction churn was chronic in this setup: memory_pressure_detected() (fixed_size_cache.py:283) only calls torch.cuda.empty_cache() and re-checks free memory when USE_INFERENCE_MODELS=True. With USE_INFERENCE_MODELS=False (our config), torch.cuda.mem_get_info() counts the torch caching allocator's reclaimable blocks as used — so a pod whose actual model working set was ~7GB on a 24GB L4 still read <20% free and cycled evict↔reload continuously (200–700 loads/hr fleet-wide). Suggest calling empty_cache() (or consulting torch.cuda.memory_reserved() - memory_allocated()) before the eviction decision regardless of USE_INFERENCE_MODELS, since fragmentation/cached blocks otherwise masquerade as memory pressure.

[imbgar-roboflow]

Correction to my previous comment after pulling actual GPU memory telemetry: the eviction pressure in our case was real, not a misread. kubernetes_io:container_accelerator_memory_used shows 18.3–20.0GB used on the 24GB card throughout — the deployment's model working set (~15 model/version combos; per-model vram_bytes from unload logs ranging 0.6–13.5GB) genuinely cannot co-reside, so the evict↔reload cycle is structural oversubscription, not allocator-cache misattribution. The empty_cache gating note stands as a design observation, but per-backend it also wouldn't have helped here (legacy mode VRAM is largely ONNX Runtime arena-held, which torch.cuda.empty_cache() can't release).\n\nThis strengthens asks 1 and 2 though: when a working set is larger than VRAM, steady-state inline loads are guaranteed, so isolating loads from the shared step pool and supporting working-set-aware admission/pinning is what separates 'slower tail' from 'whole-pod stalls'.