vLLM V1 Scheduler TTFT Regression with High logprobs and n_completions

Vulnerability Overview This vulnerability involves an extreme TTFT (Time To First Token) regression issue in the V1 engine when requests have high (completion count) and . Specifically, in the current V1 scheduler, requests are batched based on token and sequence counts, but the computational overhead of the sampling phase is ignored. For models with large vocabularies (e.g., Qwen2.5-151k), requests with and require extensive Top-K sorting across the entire vocabulary, leading to a significant increase in TTFT for all requests in each decoding iteration. Since batch execution is synchronous, a "heavy" sampling request forces other co-scheduled requests to wait for its completion, resulting in TTFT regression. Impact Scope Models: Models with large vocabularies, such as Qwen2.5-151k. Request Parameters: Requests with high (completion count) and . Performance Impact: Significant increase in TTFT, leading to higher request latency and degraded user experience. Fix Solution 1. Introduce Budget: Introduce a budget in the and V1 scheduler to track the cumulative computational cost of logprobs across all requests in a single batch. 2. Isolate Heavy Sampling Tasks: Isolate heavy sampling tasks by deferring requests that exceed the budget to the next batch. 3. Prevent Request Stagnation: Ensure that normal requests are not stalled by a single computationally intensive "amplifier" request. Test Plan Environment: NVIDIA RTX A40, CUDA 12.2, Model: Qwen/Qwen2.5-6B-Instruct. Reproduction: Run the fuzzing script with . Comparison: Compare the TTFT of victim requests (r9) and overall batch performance before and after the fix, across 10 independent runs. Test Results Metric Comparison: - Batch p99 TTFT: Reduced from 9838.8 ms to 295.2 ms, a 33.3x improvement. - Victim r9 Mean TTFT: Reduced from 1019.0 ms to 53.7 ms, an 18.9x improvement. - Victim r9 p99 TTFT: Reduced from 9761.9 ms to 65.2 ms, a 149.7x improvement. Summary: The fix restores the TTFT of victim requests from ~9.7s to a stable ~65ms, effectively neutralizing the computational amplification effect of "noisy neighbors" within the same batch. Key Design Highlights Pure Python Implementation: The fix is entirely contained within the scheduling and configuration layers, with no modifications to CUDA kernels or C++ model executors. Minimal Core Logic Changes: Redesigns the sampling process by introducing a "computational budget" heuristic, seamlessly integrating with the existing continuous batching mechanism. Backward Compatibility: The budget is disabled by default (set to 0 or a very high value), ensuring zero impact on existing user workflows while providing "opt-in" safety for latency-sensitive production environments. Low-Risk Coverage: Changes are localized and high-level, allowing safe validation without risking hardware-level or model-specific regressions. POC Code

Goal Reached Thanks to every supporter — we hit 100%!

vLLM V1 Scheduler TTFT Regression with High logprobs and n_completions

More from this source