LLM inference is a stateful, memory-bandwidth-bound, and hardware-topology-sensitive operation. The primary driver of this complexity is in transformer architectures, which generate intermediate multidimensional tensors during the evaluation of input sequences. These tensors are preserved in the Key-Value (KV) cache. As context windows expand from a few thousand tokens to multi-million-token spans, and as multi-turn conversational agents become omnipresent, the management of this KV cache has emerged as a bottleneck for inference scalability and cluster throughput.
When standard orchestration frameworks attempt to route inference traffic, they rely on static infrastructure metrics such as raw CPU utilization, generic network load, round-robin, or memory consumption limits. These traditional load balancers lack visibility into the dynamic internal state of a GPU's Video Random Access Memory (VRAM). They possess no awareness of which specific Low-Rank Adaptation (LoRA) weight matrices are currently loaded into a given accelerator, nor do they track the exact token sequences that currently reside within a node's localized KV cache.
Consequently, requests sharing identical system prompts, overlapping context documents, or extensive chat histories are routed across a fleet of inference servers. This stateless routing forces the underlying compute engines to continuously recompute identical prompt prefixes from scratch. The computational penalty for this redundancy manifests as wasted GPU cycles, degraded Time to First Token (TTFT) metrics, and a reduction in the overall concurrency limits of the cluster.
The following architecture uses a routing engine centered around an Endpoint Picker (EPP). By integrating with Envoy proxy, consuming real-time ZeroMQ (ZMQ) event meshes from inference engines like vLLM and SGLang, and leveraging hierarchical tiered storage representations, the system achieves deterministic, mathematically precise request placement.
Note: I drew inspiration from llm-d, a key CNCF initiative in distributed LLM serving, but did not adopt it for inference. My development roadmap focuses on distinct performance and energy optimization goals that require a different architectural path.
The Endpoint Picker and Envoy Proxy Integration
At the absolute core of the intelligent routing layer is the Endpoint Picker (EPP). The EPP functions as an independent, stateless microservice that operates an algorithmic brain for request placement, overriding the default round-robin or least-connections load-balancing topologies. To execute this logic without introducing latency or requiring invasive modifications to the underlying model servers, the architecture relies heavily on Envoy's External Processing (ext_proc) filter paradigm.
The Mechanics of External Processing
When a client application submits an inference request, typically delivered via an HTTP POST to a standard completion endpoint, the request first reaches the Envoy proxy. Rather than immediately forwarding this payload to a Pod, Envoy suspends the request lifecycle. Utilizing the ext_proc filter, Envoy opens a high-performance, bidirectional gRPC stream to the EPP, transmitting the request headers, metadata, and the full body for inspection.
This architectural decoupling ensures that the data plane (managed by Envoy) remains optimized for high-throughput raw byte transfer and connection management, while the control plane (the EPP) assumes full responsibility for the computationally intensive algorithmic scoring. Upon receiving the payload, the EPP executes a rapid parsing sequence. It extracts the target model identifier, tokenizes the input prompt to generate a structural map of the requested context, and evaluates the resulting token sequence against its internal, continuously updated global memory index.
Dynamic Routing and Metadata Injection
Once the EPP's internal scoring matrix identifies the most optimal physical compute node for the request, it formulates a response to the Envoy ext_proc stream. It attaches targeted identifiers, representing the physical IP address or logical DNS name of the winning inference Pod. The fusion of Envoy's robust data plane with the EPP's complex evaluation logic creates a resilient, intelligent entry point capable of managing extreme enterprise workloads.
Global Memory Map: ZeroMQ Eventing
For the Endpoint Picker to accurately route incoming prompts to the specific Pod holding the highest percentage of relevant KV cache, it must maintain a consistent, microsecond-accurate map of global GPU memory across the entire infrastructure. Attempting to query model servers synchronously via HTTP or gRPC during the routing path would introduce unacceptable network round-trip delays, violating strict latency budgets. To circumvent this, the architecture utilizes a high-throughput, asynchronous event-driven mesh powered by ZeroMQ (ZMQ).
The Publish-Subscribe Network Topology
Modern high-performance inference engines, particularly vLLM and SGLang, possess the capability to emit detailed lifecycle events regarding their internal cache memory management. These engines operate as ZMQ publishers, broadcasting on dedicated TCP ports. The EPP, in turn, deploys a highly optimized ZMQ subscriber client that establishes persistent connections to every active inference Pod within the physical cluster.
The network topology relies on a unidirectional Publish/Subscribe (PUB/SUB) pattern. Because PUB/SUB is inherently an unacknowledged, "fire and forget" protocol, the sockets are configured with substantial High Water Marks (HWM). By setting the HWM to an elevated threshold, the architecture provides deep memory buffers capable of absorbing sudden bursts in token generation or eviction without dropping critical telemetry packets.
Normalizing Disparate Payloads using Adapters
As the inference pods continuously process requests, they emit notifications corresponding to memory mutations. However, different inference providers (vLLM, SGLang) serialize and structure these payloads uniquely, requiring the EPP to employ a suite of polymorphic adapter patterns to normalize the data streams into a cohesive internal format. Through this continuous ingestion and normalization of ZMQ notifications, the EPP is provided with a relentless, normalized stream of exact memory state changes, allowing it to "see" inside the VRAM of every GPU in the cluster.
KV Cache Token Information and The Radix Trie
The raw stream of ZMQ events represents a massive volume of unstructured data. The critical engineering challenge lies in transforming this continuous influx of token block notifications into a queryable, low-latency index that the EPP can search during the lifespan of a live Envoy ext_proc request. Currently, I am using a Radix Trie, though I plan to move to a multi-tier Cache for optimization under load in the future based on load testing results.
The Sharded Radix Trie
The system stores data in a Radix Trie (or prefix tree). Each node in the tree holds a group of cached tokens. As a sequence of tokens is processed, the path we take through the tree matches the growing context of the prompt. To support the massive concurrency required by enterprise environments, the Trie is divided into multiple independent shards. When an operation targets a specific sequence of tokens, the root hash is mapped directly to the shard responsible for that sequence. This sharding strategy virtually eliminates thread contention across the data structure, permitting dozens of simultaneous Envoy ext_proc threads to query the index concurrently without constantly acquiring and releasing global system locks.
Read-Copy-Update (RCU) Concurrency Mechanics
Because ZMQ events dictating the storage and eviction of memory blocks flood the system at a rate of thousands of messages per second, the Trie must be continuously written to without ever stalling the reader threads evaluating active prompts. The architecture achieves this via a Read-Copy-Update (RCU) paradigm. Consequently, reader threads executing routing logic never encounter mutex locks, race conditions, or blocked states. By mapping the token hashes of an incoming prompt through this lock-free structure, the EPP calculates the absolute maximum depth of prefix matches available on every single inference pod across the cluster in microseconds.
Multi-Dimensional Scoring Criteria and Pod Selection
Routing LLM inference requests is not a binary decision; it is a complex, multi-dimensional optimization problem. The presence of a KV cache prefix on a specific GPU does not automatically guarantee that it is the optimal endpoint. When the EPP queries the Sharded Trie, it retrieves a list of candidate Pods accompanied by the precise depth of their KV cache match. The system's algorithmic scoring engine then processes these candidates through a heuristic weighting framework to determine the definitive victor.
Mathematical Weighting and Penalties
The final numerical score assigned to any given candidate Pod is a mathematical combination of several continuously updating parameters, each calibrated to prioritize overall cluster throughput and responsiveness.
- KV Cache Affinity: The foundation of the score is derived from cache locality. Pods that successfully hold a high percentage of the prompt's required prefix in memory receive a massive baseline score amplification. The longer the contiguous prefix match found in the Radix Trie, the greater the point allocation.
- Queue Pressure and Circuit Breaking: Cache affinity is entirely negated if the target Pod is severely backlogged. Queue depth is assigned one of the highest mathematical weights in the scoring formula, operating as a definitive circuit breaker against tail latency degradation. As a specific inference Pod approaches its predefined maximum queue depth, a steep geometric penalty curve is applied to its score. This aggressive penalty forces the EPP to spill over excess traffic to alternative, idle Pods.
- Compute Utilization: General GPU compute utilization and available VRAM metrics serve as stabilizing parameters. Minor penalties are assigned based on total resource consumption to ensure that no single piece of hardware is pushed toward thermal or electrical failure, promoting uniform load distribution across the cluster.
- LoRA Saturation: Penalizes Pods that have reached their maximum limit of concurrently loaded adapters to prevent adapter thrashing.
- Hardware Degradation: Applies severe penalties based on thermal throttling or VRAM error metrics to protect physical infrastructure; automatically routes traffic away from failing hardware.
- Storage Tier Latency: Deducts points depending on whether cache resides in VRAM, RAM, NVMe, or network. Accounts for bus transfer speeds; prioritizes immediate GPU memory over slower disk cache.
Rendezvous Hashing for Tie-Breaking
In highly symmetric cluster states, such as immediately after a scale-out event where a fleet of newly provisioned replicas boots up without an established KV cache, multiple Pods can evaluate to identical scores. In naive routing systems, this leads to the "thundering herd" problem, wherein the router funnels all initial traffic to the sequentially first Pod, instantly overwhelming it.
To circumvent this, the EPP employs Rendezvous Hashing, also known as Highest Random Weight (HRW) hashing. By executing a cryptographic hash of the unique request identifier against the string IP addresses of the tied candidate Pods, the scoring engine establishes a deterministic uniform distribution sequence.
Granular Telemetry via Endpoints
While the asynchronous ZMQ mesh handles the microsecond-level synchronization of specific KV cache tokens, intelligent macro routing requires robust, system-level observation. The scheduling architecture achieves this macroscopic awareness by operating a continuous, highly concurrent background pipeline that actively scrapes the local /metrics endpoints of all participating inference Pods.
Normalizing Disparate Inference Schemas
To support a vast array of inference engines, each exposing highly proprietary and non-standardized metric schemas, the scheduling system rectifies fragmentation by deploying a suite of interface-driven Metric Adapters to parse, extract, and standardize this data into a unified, zero-allocation internal structure known as Engine Telemetry.
- vLLM Telemetry: The engine explicitly exposes vital operational data. The parsing adapter extracts metrics such as
num_requests_waiting,num_requests_running, and the criticalgpu_cache_usage_perc. This provides an instant snapshot of the engine's internal pressure. - SGLang Telemetry: SGLang provides similar data but structures its queues uniquely, often separating the running queue from the waiting queue. The specific SGLang adapter natively ingests these distinct metrics and mathematically sums them to form a single, normalized queue depth metric. This normalization ensures the macro scheduler evaluates load uniformly, completely agnostic to the underlying inference engine.
- Triton / TensorRT-LLM Telemetry: Certain high-performance frameworks do not natively publish a straightforward cache utilization percentage. To overcome this, the Triton adapter employs a mathematical heuristic. It scrapes the
nv_cache_num_hits_per_modeland the corresponding cache misses, dividing the misses by the total requests to inversely derive the precise cache pressure impacting the node.
Performance Impact, Scalability, and Availability of the EPP
The introduction of intelligent, state-aware routing has a profound, measurable impact on the operational economics of LLM inference. These optimizations directly translate to reduced energy consumption and the ability to serve significantly higher concurrency on identical hardware footprints.
Dynamic Auto-Scaling
As the enterprise cluster expands to handle thousands of simultaneous users, the EPP scales horizontally. Because of the optimizations applied to the codebase, specifically the zero-allocation token hashing and the lock-free Read-Copy-Update (RCU) algorithms utilized by the Radix Trie, a single EPP instance utilizes minuscule CPU overhead to evaluate complex routing equations.
When computational load eventually necessitates expansion, standard Kubernetes Horizontal Pod Autoscalers (HPA) effortlessly provision additional EPP replicas. The Envoy proxy layer seamlessly balances the incoming ext_proc streams across this expanded fleet, ensuring the routing layer remains universally scalable and endlessly available regardless of traffic spikes.
Conclusion
The architecture examined represents a fundamental shift in the orchestration of planetary-scale artificial intelligence. By evolving beyond the limitations of naive, stateless HTTP load balancing, the integration of an advanced Endpoint Picker with Envoy's ext_proc filter allows the routing layer to achieve awareness of the physical infrastructure.
By concurrently ingesting millions of ZeroMQ lifecycle events and continuously parsing disparate metrics, the system constructs a real-time map of global GPU memory. Leveraging the extreme efficiency of zero-allocation token hashing and lock-free Radix Tries, the EPP computes optimal request placement in microseconds, accounting for LoRA adapter saturation, thermal degradation, and physical storage medium latencies. As the industry marches inevitably toward full prefill and decode disaggregation, schedulers capable of this profound level of hardware and memory intelligence will be the differentiating factor in realizing economically viable, high-throughput AI inference at scale.