FastSwitch：一种具有公平意识的 LLM 服务系统，可解决上下文切换中的低效率问题

大型语言模型 (LLM) 改变了人工智能应用程序，为语言翻译、虚拟助理和代码生成等任务提供支持。

Large language models (LLMs) have revolutionized AI applications, powering diverse tasks such as language translation, virtual assistants, and code generation. These models are deployed on resource-intensive infrastructure, typically comprising GPUs with high-bandwidth memory, to handle their immense computational demands. However, delivering high-quality service to numerous users concurrently poses significant challenges. Efficiently allocating these limited resources is critical to meeting service level objectives (SLOs) for time-sensitive metrics, ensuring the system can cater to more users without compromising performance.

大型语言模型 (LLM) 彻底改变了人工智能应用程序，为语言翻译、虚拟助理和代码生成等多种任务提供支持。这些模型部署在资源密集型基础设施上，通常包括具有高带宽内存的 GPU，以满足其巨大的计算需求。然而，同时向众多用户提供高质量的服务带来了巨大的挑战。有效地分配这些有限的资源对于满足时间敏感指标的服务级别目标 (SLO) 至关重要，从而确保系统能够在不影响性能的情况下满足更多用户的需求。

A persistent issue in LLM serving systems is achieving fair resource distribution while maintaining efficiency. Existing systems often prioritize throughput, neglecting fairness requirements such as balancing latency among users. Preemptive scheduling mechanisms, which dynamically adjust request priorities, address this. However, these mechanisms introduce context-switching overheads, such as GPU idleness and inefficient I/O utilization, which degrade key performance indicators like Time to First Token (TTFT) and Time Between Tokens (TBT). For instance, the stall time caused by preemption in high-stress scenarios can reach up to 59.9% of P99 latency, leading to a significant decline in user experience.

LLM服务系统中一个持续存在的问题是在保持效率的同时实现公平的资源分配。现有系统通常优先考虑吞吐量，而忽略公平性要求，例如平衡用户之间的延迟。动态调整请求优先级的抢占式调度机制可以解决这个问题。然而，这些机制引入了上下文切换开销，例如 GPU 空闲和低效 I/O 利用率，从而降低了关键性能指标，例如首次令牌时间 (TTFT) 和令牌之间时间 (TBT)。例如，高压力场景下抢占造成的卡顿时间可达P99时延的59.9%，导致用户体验大幅下降。

Current solutions, such as vLLM, rely on paging-based memory management to address GPU memory constraints by swapping data between GPU and CPU memory. While these approaches improve throughput, they face limitations. Issues such as fragmented memory allocation, low I/O bandwidth utilization, and redundant data transfers during multi-turn conversations persist, undermining their effectiveness. For example, vLLM’s fixed block size of 16 tokens results in suboptimal granularity, which reduces PCIe bandwidth efficiency and increases latency during preemptive context switching.

当前的解决方案（例如 vLLM）依赖基于分页的内存管理，通过在 GPU 和 CPU 内存之间交换数据来解决 GPU 内存限制。虽然这些方法提高了吞吐量，但它们也面临局限性。诸如内存分配碎片、I/O 带宽利用率低以及多轮对话期间的冗余数据传输等问题仍然存在，从而削弱了其有效性。例如，vLLM 的固定块大小为 16 个令牌，导致粒度不理想，从而降低了 PCIe 带宽效率并增加了抢占式上下文切换期间的延迟。

To address these inefficiencies, researchers from Purdue University, Shanghai Qi Zhi Institute, and Tsinghua University developed FastSwitch, a fairness-aware LLM serving system that introduces three core optimizations: a dynamic block group manager, a multithreading swap manager, and a KV cache reuse mechanism. These innovations synergize to improve I/O utilization, reduce GPU idleness, and minimize redundant data transfers. The system’s design builds on vLLM but focuses on coarse-grained memory allocation and asynchronous operations to enhance resource management.

为了解决这些低效率问题，来自普渡大学、上海启智研究院和清华大学的研究人员开发了 FastSwitch，这是一个公平感知的 LLM 服务系统，引入了三个核心优化：动态块组管理器、多线程交换管理器和 KV 缓存重用机制。这些创新协同作用，可提高 I/O 利用率、减少 GPU 闲置率并最大限度地减少冗余数据传输。该系统的设计基于 vLLM，但侧重于粗粒度内存分配和异步操作，以增强资源管理。

FastSwitch’s dynamic block group manager optimizes memory allocation by grouping contiguous blocks, increasing transfer granularity. This approach reduces latency by up to 3.11x compared to existing methods. The multithreading swap manager enhances token generation efficiency by enabling asynchronous swapping, mitigating GPU idle time. It incorporates fine-grained synchronization to avoid conflicts between ongoing and new requests, ensuring seamless operation during overlapping processes. Meanwhile, the KV cache reuse mechanism preserves partially valid data in CPU memory, reducing preemption latency by avoiding redundant KV cache transfers. These components collectively address key challenges and improve the overall performance of LLM serving systems.

FastSwitch 的动态块组管理器通过对连续块进行分组来优化内存分配，从而提高传输粒度。与现有方法相比，此方法可将延迟减少高达 3.11 倍。多线程交换管理器通过启用异步交换来提高令牌生成效率，减少 GPU 空闲时间。它结合了细粒度的同步，以避免正在进行的请求和新请求之间的冲突，确保重叠流程期间的无缝操作。同时，KV缓存重用机制将部分有效数据保留在CPU内存中，通过避免冗余的KV缓存传输来减少抢占延迟。这些组件共同解决关键挑战并提高 LLM 服务系统的整体性能。

The researchers evaluated FastSwitch using the LLaMA-8B and Qwen-32B models on GPUs such as NVIDIA A10 and A100. Testing scenarios included high-frequency priority updates and multi-turn conversations derived from the ShareGPT dataset, which averages 5.5 turns per conversation. FastSwitch outperformed vLLM across various metrics. It achieved speedups of 4.3-5.8x in P95 TTFT and 3.6-11.2x in P99.9 TBT for different models and workloads. Furthermore, FastSwitch improved throughput by up to 1.44x, demonstrating its ability to handle complex workloads efficiently. The system also substantially reduced context-switching overhead, improving I/O utilization by 1.3x and GPU by 1.42x compared to vLLM.

研究人员在 NVIDIA A10 和 A100 等 GPU 上使用 LLaMA-8B 和 Qwen-32B 模型评估了 FastSwitch。测试场景包括高频优先级更新和源自 ShareGPT 数据集的多轮对话，平均每次对话 5.5 轮。 FastSwitch 在各种指标上均优于 vLLM。对于不同的模型和工作负载，它在 P95 TTFT 中实现了 4.3-5.8 倍的加速，在 P99.9 TBT 中实现了 3.6-11.2 倍的加速。此外，FastSwitch 将吞吐量提高了 1.44 倍，证明了其高效处理复杂工作负载的能力。与 vLLM 相比，该系统还大幅减少了上下文切换开销，I/O 利用率提高了 1.3 倍，GPU 利用率提高了 1.42 倍。

FastSwitch’s optimizations resulted in tangible benefits. For example, its KV cache reuse mechanism reduced swap-out blocks by 53%, significantly lowering latency. The multithreading swap manager enhanced token generation efficiency, achieving a 21.8% improvement at P99 latency compared to baseline systems. The dynamic block group manager maintained granularity by allocating memory in larger chunks, balancing efficiency and utilization. These advancements highlight FastSwitch’s capacity to maintain fairness and efficiency in high-demand environments.

FastSwitch 的优化带来了实实在在的好处。例如，其 KV 缓存重用机制将换出块减少了 53%，从而显着降低了延迟。多线程交换管理器提高了代币生成效率，与基准系统相比，P99 延迟提高了 21.8%。动态块组管理器通过以更大的块分配内存来保持粒度，平衡效率和利用率。这些进步凸显了 FastSwitch 在高要求环境中保持公平和效率的能力。

The research team’s key takeaways include the following:

研究小组的主要结论包括：

In conclusion, FastSwitch addresses fundamental inefficiencies in LLM serving by introducing innovative optimizations that balance fairness and efficiency. Reducing context-switching overheads and enhancing resource utilization ensure scalable, high-quality service delivery for multi-user environments. These advancements make it a transformative solution for modern LLM deployments.

总之，FastSwitch 通过引入平衡公平和效率的创新优化，从根本上解决了 LLM 服务效率低下的问题。减少上下文切换开销并提高资源利用率可确保为多用户环境提供可扩展的高质量服务。这些进步使其成为现代法学硕士部署的变革性解决方案。