AWS Improves vLLM with Bi-directional KV Cache Transfer

TL;DR: AWS has contributed a bi-directional KV cache transfer feature to vLLM that allows prefill and decode nodes to reuse previously computed context across conversation turns. In benchmarks on Amazon EC2 P5en instances, the feature reduced time to first token by up to 1.6x for Qwen3-32B and 3.3x for GLM-5-FP8 models.

What's New

AWS has released a bi-directional key-value (KV) cache transfer capability for vLLM that significantly improves time to first token (TTFT) for multi-turn agentic workloads. The feature addresses a fundamental inefficiency in conversational AI systems: with each new turn in a conversation, the system must reprocess the entire conversation history, including system instructions, prior prompts, tool results, and model responses.

Traditional prefill-decode (P-D) disaggregation splits requests across specialized nodes—prefill nodes process prompts and build KV caches, while decode nodes generate responses. However, conventional systems only transfer KV cache in one direction, from prefill to decode. This means when processing the next turn of a conversation, the prefill node must recompute KV projections for previous response tokens that the decode node already holds.

The new bi-directional transfer mechanism allows decode nodes to return previously computed KV cache back to prefill nodes on subsequent conversation turns, avoiding redundant computation. AWS originally built and validated this feature on AWS Trainium instance clusters and contributed it to vLLM as an accelerator-agnostic capability.

In benchmarks on a 1P1D deployment using Amazon EC2 P5en instances with AWS Elastic Fabric Adapter (EFA), the feature reduced TTFT by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8. The instances use EFA, a low-latency RDMA network that supports GPUDirect RDMA and Neuron Direct RDMA to move KV blocks directly between GPU or AWS Trainium device memory.

How It Works

The implementation extends the NVIDIA Inference Xfer Library (NIXL) KV connector to let prefill nodes consume remote KV blocks. The system introduces a KV-cache-aware router that maintains an in-memory mapping from conversation IDs to KV block metadata—importantly, the router stores only metadata, not the KV tensors themselves.

When a conversation turn completes, the decode node retains eligible KV blocks for a configurable lifetime (default 480 seconds) and publishes their location. On the next turn, the router retrieves this metadata so the prefill node can reuse the KV from decode through a D→P NIXL READ operation, avoiding recomputation of shared context.

The feature is governed by a configurable KV recompute threshold that determines when reusing remote KV blocks is worthwhile compared to local computation.

Pricing and Availability

The bi-directional KV cache transfer feature is available now in vLLM. Pricing details were not specified in the release, as this is an open-source contribution to the vLLM project. Users will incur standard AWS compute costs for running Amazon EC2 P5en instances or AWS Trainium instances with EFA networking.

How to Try It

The feature is available in vLLM and supports Amazon EC2 P5en GPU instances and AWS Trainium instance clusters connected with EFA. AWS has provided an example proxy implementation on the vLLM GitHub repository that demonstrates the KV-cache-aware router flow. Developers can adapt this example for their specific environments.

The vLLM GitHub repository contains documentation and example code for implementing the bi-directional KV transfer in production systems.

Who It's For

This enhancement is designed for developers and organizations building conversational AI assistants and agentic applications that involve multi-turn interactions. It's particularly valuable for applications where conversation histories grow long, as the TTFT improvements become more pronounced with longer contexts. Teams running large language models at scale on AWS infrastructure who need to optimize accelerator utilization and reduce latency will benefit most from this capability.

Alternatives

Other approaches to improving multi-turn inference performance include prompt caching strategies, speculative decoding, and alternative serving frameworks. However, the bi-directional KV transfer specifically addresses the inefficiency in disaggregated prefill-decode architectures. Results vary with workload and configuration, and the optimal approach depends on specific deployment requirements and infrastructure.