FastSwitch：一種具有公平意識的 LLM 服務系統，可解決情境切換中的低效率問題

大型語言模型 (LLM) 改變了人工智慧應用程序，為語言翻譯、虛擬助理和程式碼生成等任務提供支援。

Large language models (LLMs) have revolutionized AI applications, powering diverse tasks such as language translation, virtual assistants, and code generation. These models are deployed on resource-intensive infrastructure, typically comprising GPUs with high-bandwidth memory, to handle their immense computational demands. However, delivering high-quality service to numerous users concurrently poses significant challenges. Efficiently allocating these limited resources is critical to meeting service level objectives (SLOs) for time-sensitive metrics, ensuring the system can cater to more users without compromising performance.

大型語言模型 (LLM) 徹底改變了人工智慧應用程序，為語言翻譯、虛擬助理和程式碼生成等多種任務提供支援。這些模型部署在資源密集型基礎設施上，通常包括具有高頻寬記憶體的 GPU，以滿足其龐大的運算需求。然而，同時為眾多用戶提供高品質的服務帶來了巨大的挑戰。有效分配這些有限的資源對於滿足時間敏感指標的服務等級目標 (SLO) 至關重要，從而確保系統能夠在不影響效能的情況下滿足更多使用者的需求。

A persistent issue in LLM serving systems is achieving fair resource distribution while maintaining efficiency. Existing systems often prioritize throughput, neglecting fairness requirements such as balancing latency among users. Preemptive scheduling mechanisms, which dynamically adjust request priorities, address this. However, these mechanisms introduce context-switching overheads, such as GPU idleness and inefficient I/O utilization, which degrade key performance indicators like Time to First Token (TTFT) and Time Between Tokens (TBT). For instance, the stall time caused by preemption in high-stress scenarios can reach up to 59.9% of P99 latency, leading to a significant decline in user experience.

LLM服務系統中一個持續存在的問題是在維持效率的同時實現公平的資源分配。現有系統通常優先考慮吞吐量，而忽略公平性要求，例如平衡用戶之間的延遲。動態調整請求優先權的搶佔式調度機制可以解決這個問題。然而，這些機制引入了上下文切換開銷，例如 GPU 空閒和低效 I/O 利用率，從而降低了關鍵效能指標，例如首次令牌時間 (TTFT) 和令牌之間時間 (TBT)。例如，高壓力場景下搶佔造成的卡頓時間可達P99延遲的59.9%，導致使用者體驗大幅下降。

Current solutions, such as vLLM, rely on paging-based memory management to address GPU memory constraints by swapping data between GPU and CPU memory. While these approaches improve throughput, they face limitations. Issues such as fragmented memory allocation, low I/O bandwidth utilization, and redundant data transfers during multi-turn conversations persist, undermining their effectiveness. For example, vLLM’s fixed block size of 16 tokens results in suboptimal granularity, which reduces PCIe bandwidth efficiency and increases latency during preemptive context switching.

目前的解決方案（例如 vLLM）依賴基於分頁的記憶體管理，透過在 GPU 和 CPU 記憶體之間交換資料來解決 GPU 記憶體限制。雖然這些方法提高了吞吐量，但它們也面臨限制。諸如記憶體分配碎片、I/O 頻寬利用率低以及多輪對話期間的冗餘資料傳輸等問題仍然存在，從而削弱了其有效性。例如，vLLM 的固定區塊大小為 16 個令牌，導致粒度不理想，從而降低了 PCIe 頻寬效率並增加了搶佔式上下文切換期間的延遲。

To address these inefficiencies, researchers from Purdue University, Shanghai Qi Zhi Institute, and Tsinghua University developed FastSwitch, a fairness-aware LLM serving system that introduces three core optimizations: a dynamic block group manager, a multithreading swap manager, and a KV cache reuse mechanism. These innovations synergize to improve I/O utilization, reduce GPU idleness, and minimize redundant data transfers. The system’s design builds on vLLM but focuses on coarse-grained memory allocation and asynchronous operations to enhance resource management.

為了解決這些低效率問題，來自普渡大學、上海啟智研究院和清華大學的研究人員開發了FastSwitch，這是一個公平感知的LLM 服務系統，引入了三個核心優化：動態區塊組管理器、多線程交換管理器和KV 快取重複使用機制。這些創新協同作用，可提高 I/O 利用率、減少 GPU 閒置率並最大限度地減少冗餘資料傳輸。該系統的設計基於 vLLM，但側重於粗粒度記憶體分配和非同步操作，以增強資源管理。

FastSwitch’s dynamic block group manager optimizes memory allocation by grouping contiguous blocks, increasing transfer granularity. This approach reduces latency by up to 3.11x compared to existing methods. The multithreading swap manager enhances token generation efficiency by enabling asynchronous swapping, mitigating GPU idle time. It incorporates fine-grained synchronization to avoid conflicts between ongoing and new requests, ensuring seamless operation during overlapping processes. Meanwhile, the KV cache reuse mechanism preserves partially valid data in CPU memory, reducing preemption latency by avoiding redundant KV cache transfers. These components collectively address key challenges and improve the overall performance of LLM serving systems.

FastSwitch 的動態區塊組管理器透過對連續區塊進行分組來最佳化記憶體分配，從而提高傳輸粒度。與現有方法相比，此方法可將延遲減少高達 3.11 倍。多執行緒交換管理器透過啟用非同步交換來提高令牌產生效率，減少 GPU 空閒時間。它結合了細粒度的同步，以避免正在進行的請求和新請求之間的衝突，確保重疊流程期間的無縫操作。同時，KV快取重用機制將部分有效資料保留在CPU記憶體中，透過避免冗餘的KV快取傳輸來減少搶佔延遲。這些組件共同解決關鍵挑戰並提高 LLM 服務系統的整體效能。

The researchers evaluated FastSwitch using the LLaMA-8B and Qwen-32B models on GPUs such as NVIDIA A10 and A100. Testing scenarios included high-frequency priority updates and multi-turn conversations derived from the ShareGPT dataset, which averages 5.5 turns per conversation. FastSwitch outperformed vLLM across various metrics. It achieved speedups of 4.3-5.8x in P95 TTFT and 3.6-11.2x in P99.9 TBT for different models and workloads. Furthermore, FastSwitch improved throughput by up to 1.44x, demonstrating its ability to handle complex workloads efficiently. The system also substantially reduced context-switching overhead, improving I/O utilization by 1.3x and GPU by 1.42x compared to vLLM.

研究人員在 NVIDIA A10 和 A100 等 GPU 上使用 LLaMA-8B 和 Qwen-32B 模型評估了 FastSwitch。測試場景包括高頻優先權更新和源自 ShareGPT 資料集的多輪對話，平均每次對話 5.5 輪。 FastSwitch 在各種指標上均優於 vLLM。對於不同的模型和工作負載，它在 P95 TTFT 中實現了 4.3-5.8 倍的加速，在 P99.9 TBT 中實現了 3.6-11.2 倍的加速。此外，FastSwitch 將吞吐量提高了 1.44 倍，證明了其高效處理複雜工作負載的能力。與 vLLM 相比，該系統還大幅減少了上下文切換開銷，I/O 利用率提高了 1.3 倍，GPU 利用率提高了 1.42 倍。

FastSwitch’s optimizations resulted in tangible benefits. For example, its KV cache reuse mechanism reduced swap-out blocks by 53%, significantly lowering latency. The multithreading swap manager enhanced token generation efficiency, achieving a 21.8% improvement at P99 latency compared to baseline systems. The dynamic block group manager maintained granularity by allocating memory in larger chunks, balancing efficiency and utilization. These advancements highlight FastSwitch’s capacity to maintain fairness and efficiency in high-demand environments.

FastSwitch 的優化帶來了實實在在的好處。例如，其 KV 快取重用機制將換出區塊減少了 53%，從而顯著降低了延遲。多執行緒交換管理器提高了代幣產生效率，與基準系統相比，P99 延遲提高了 21.8%。動態區塊組管理器透過以更大的區塊分配記憶體來保持粒度，平衡效率和利用率。這些進步凸顯了 FastSwitch 在高要求環境中保持公平和效率的能力。

The research team’s key takeaways include the following:

研究小組的主要結論包括：

In conclusion, FastSwitch addresses fundamental inefficiencies in LLM serving by introducing innovative optimizations that balance fairness and efficiency. Reducing context-switching overheads and enhancing resource utilization ensure scalable, high-quality service delivery for multi-user environments. These advancements make it a transformative solution for modern LLM deployments.

總之，FastSwitch 透過引入平衡公平和效率的創新優化，從根本上解決了 LLM 服務效率低下的問題。減少情境切換開銷並提高資源利用率可確保為多用戶環境提供可擴展的高品質服務。這些進步使其成為現代法學碩士部署的變革性解決方案。