Amir Noohi

In long-context LLM serving, the key–value (KV) cache outgrows GPU memory, so production systems keep it across DRAM, CXL memory and NVMe disk, and a tiering policy decides which blocks stay in fast memory. Deployed policies predict future reads from past ones, which fits the KV cache poorly. The blocks a node needs change with the router's mix of requests, so such a policy notices each change only after requests have waited on disk, although the engine already knows which blocks the queued requests' prefills will read. We measure 17 state-of-the-art tiering policies inside vLLM on a four-tier host over 10 workloads near capacity. No single policy has the best median time to first token (TTFT) on every workload, and at the same request rate the best and the worst policy differ by 1.12–6.17× in median TTFT within a workload, although waiting on KV loads makes up at most 74% of TTFT.

Tenure takes a counted lease on a request's blocks when the request enters the queue, which keeps them off disk until it finishes while letting them move among the fast tiers. Because KV blocks never change, dropping one to disk moves zero bytes, and a block read from CXL moves up to DRAM only when a queued request will read it, so a promotion never pushes down a block about to be read. Unleased blocks follow the classic order that has cost least on recent reads. Tenure has the best median TTFT on 6 of the 10 workloads, and on its worst it is within 1.30× of the best, while every baseline is at least 1.98× from the best on some workload.

Hardware-assisted virtualization relies on two-dimensional (2D) nested address translation, trading the costly software traps of Shadow Page Tables for high access-time latency. This overhead is heavily concentrated at the leaf of the page walk: the final host page-table traversal required to resolve the data page. To eliminate this bottleneck, we introduce SagePTE, a hardware-software co-design that bridges the gap between nested and shadow paging by shadowing only the leaf PTEs, directly in-place within the guest page table. By expanding the guest page table layout on-demand in host memory, SagePTE co-locates guest and host mappings into a unified 128-bit extended entry. To maintain strict isolation while guaranteeing trap-free leaf synchronization, our design seamlessly repurposes existing silicon primitives, such as 128-bit compare-and-swap logic. Zero-delay hardware access mediation within the core's memory execution path safely masks guest reads and upgrades guest writes, triggering automatic hardware synchronization. This allows the page walker to fetch both translations simultaneously and bypass the final host walk entirely. SagePTE operates purely beneath the virtualization boundary, supporting unmodified guests without para-virtualization. We formally verify SagePTE's isolation guarantees using Alloy and evaluate it on ARM, demonstrating average end-to-end speedups over nested translation of 1.30× with 4 KB pages and 1.22× with Transparent Huge Pages (THP).

Disaggregated LLM inference separates prefill and decode across nodes, relying on cross-node KV cache reuse (prefix caching) to reduce time-to-first-token (TTFT). Yet existing systems manage this cache entirely in user space, atop the very kernel that already handles memory mapping, page tables, and RDMA registration. We identify a management scaling paradox: user-space overhead from coordination RPCs, per-chunk locking, redundant memory copies, and RDMA transfer fragmentation grows to 79% of achievable TTFT as the cache warms, nearly doubling latency when caching should help most. We formalize this paradox as a TTFT decomposition model and validate it on the state-of-the-art LMCache + Mooncake stack atop vLLM. We then present RMC (Remote Memory for Cache), an in-kernel framework that extends OS-provided shared memory (/dev/shm) to cluster-wide scope via content-addressed Virtual Memory Regions (VMRs), where any node computes the address of any cached chunk by hashing its token content, requiring no coordination. Across four workloads, RMC achieves 1.1–1.3× TTFT reduction over LMCache + Mooncake and up to 2.1× over full recompute.

The rapid scaling of large language models—from hundreds of billions to trillions of parameters—has rendered distributed memory management the critical bottleneck in modern AI systems. Existing frameworks manage cross-node memory entirely in user space, incurring serialization overhead that consumes up to 76.7% of data transfer latency, memory amplification reaching up to 6.7× from intermediate buffers, and escalating coordination costs that compound with cluster size. These overheads are fundamental: commodity operating systems lack abstractions for TB-scale tensors, cross-node sharing, and the sequential, layer-granular access patterns characteristic of transformer workloads. We present NTSM, a kernel-space middleware that recasts distributed memory management as a first-class OS primitive. Implemented as a Linux kernel module with a minimal kernel modification (one exported symbol on x86), NTSM exposes a symmetric virtual address space across cluster nodes where applications access remote data through ordinary memory operations. Page faults trigger zero-copy RDMA transfers with automatic coherence enforcement via a directory-based M/S/I protocol operating at 32KB–2MB Virtual Memory Region granularity—matching the natural access grain of transformer layers while amortizing fault overhead. By executing in kernel space, NTSM directly manipulates page tables, pins memory for DMA, and manages TLB state, eliminating the serialization, buffer copies, and user-kernel transitions inherent to user-space approaches. NTSM integrates transparently with existing frameworks: 152 lines modified for PyTorch FSDP, 89 lines for Ray Parameter Server. Evaluation on a 34-node RDMA cluster with models up to 175B parameters demonstrates up to 6.5× speedup in distributed training memory operations, 4.4× reduction in time-to-first-token for inference, and 47% reduction in parameter server iteration time—establishing kernel-level memory management as a viable foundation for next-generation distributed AI infrastructure.

As AI models grow exponentially in size, memory has emerged as a critical bottleneck for inference at scale. While hardware solutions like Compute Express Link (CXL) promises to solve the problem of memory capacity and sharing, they require capital investment, and are not widely available. This paper presents RMAI, an in-kernel remote shared memory framework tailored for AI inference workloads, offering a transparent, scalable, and cost-effective software alternative to hardware-based memory expansion and sharing solutions.

Network programmability has always been a topic of interest to researchers. After introducing the concept of SDN, control-plane programmability is achieved. However, data-plane programmability remains a challenge in the context of network programmability. We propose an autonomous SDN-based framework that achieves flexible parsing of arbitrary protocol header fields in real-time using software switches based on the NFV concept.