AI Inference Routing: Importance and How NVIDIA PAIR Works

What It Is, Why It Matters, and How NVIDIA PAIR Puts It Into Practice

When several people hit an AI system at once, a basic load balancer chokes. It spreads requests evenly but doesn’t know that one query might run for two seconds and another for twenty. AI inference routing sends each request to the machine best equipped to handle it right now. It considers model availability, GPU memory pressure, queue depth, and current workload before deciding where to send a request.

This article explains what inference routing is, why plain round-robin load balancing falls apart for LLMs, and how NVIDIA Personal AI Router (PAIR) implements routing across local machines to serve more concurrent users without sending anything to the cloud.

Why standard load balancing fails for LLMs

Traditional load balancers were built for stateless web traffic. They assume requests are fast, roughly the same size, and independent. LLM inference breaks all three assumptions.

LLM requests vary wildly. A 50-token prompt asking for a quick summary is nothing like a 4,000-token prompt that needs multi-step reasoning, and outputs can range from a handful of tokens to thousands. A round-robin balancer still hands out requests one by one, so one node ends up processing two long generations while another sits mostly idle. Red Hat’s llm-d team demonstrated this clearly: with the same 16 GPUs, inference-aware routing served roughly twice as many concurrent users at their service-level objective compared to baseline Kubernetes round-robin.

The problem gets worse because LLM decoding is barrier-synchronized. When a model runs across multiple data-parallel workers, every decode step finishes only when the slowest worker finishes. Imbalance therefore doesn’t just shift work around; it wastes GPU cycles directly arXiv:2605.06113. Google’s DLB system, running in production for 22 months, cut median latency by 17% and p95 tail latency by 13% by replacing generic heuristics with latency-aware routing. AWS SageMaker HyperPod Inference Gateway achieved up to 82% lower first-token latency and 97–98% lower p99 TTFT by routing based on KV-cache utilization, queue depth, LoRA adapter residency, and predicted latency.

Standard load balancing optimizes for the wrong metric: request count, not actual work.

What AI inference routing actually does

Inference routing makes a placement decision per request using real-time signals. The goal isn’t just to spread requests evenly; it’s to put each request where it can finish fastest given current conditions. Relevant signals include:

Model availability: Can the node serve the exact model requested?
Queue depth: How many requests are already waiting or running?
KV-cache state: Does the node already hold cached computations for similar prompts?
GPU utilization: How saturated is the node’s memory bandwidth and compute?
Pending workload: How much work has the router already sent to this node?

Not every system uses all of these signals. PAIR, for example, does not consider KV-cache state or whether a model is already warm in memory when routing.

Routing also needs to handle failures gracefully. If a node advertises a model but then returns an error, the router must retry a different node without penalizing the client. RouteBalance fuses model selection and load balancing into one decision, while llm-d additionally ranks nodes using KV-cache state.

Introducing NVIDIA Personal AI Router (PAIR)

NVIDIA Personal AI Router (PAIR) brings this kind of routing to a local, personal-scale cluster NVIDIA PAIR Overview. Instead of routing across a cloud data center, PAIR turns several machines on your home or office network into one place to send inference requests. You point your application at a local endpoint, and PAIR decides which machine actually serves each request.

The project is open source under the Apache 2.0 license. It supports Windows 11, Linux, and macOS, and nodes can mix operating systems and architectures. GPU and driver requirements depend on the inference engine and model you choose, not on PAIR itself.

What PAIR is not

PAIR has clear boundaries. It does not pool GPU memory across machines, create a larger virtual GPU, shard one model across nodes, or split a single inference request between systems. Each request runs entirely on one node. Adding machines increases how many requests you can run concurrently; it does not make an individual request faster.

How PAIR works

The cluster model

PAIR defines four levels: cluster, node, engine, and model.

A cluster is a set of paired nodes. There is no primary or controller; membership is symmetric.
A node is one machine running PAIR. Every node runs the same software and can both serve and route requests.
An engine is an inference server on a node: Ollama or LM Studio.
A model belongs to an engine on a specific node. Nodes do not share models; the same model on two nodes is two independent copies.

A node is eligible to serve a request only when it is reachable, running a compatible engine, and advertising the requested model.

The request path

When an application sends an Ollama-compatible or OpenAI-compatible HTTP request to PAIR’s local endpoint, the proxy on that machine works through these steps NVIDIA PAIR Architecture:

Takes a request-local copy of the latest discovery snapshot.
Applies a model eligibility gate: only nodes whose engine inventory advertises the requested model remain candidates.
Produces an ordered failover list:
- A manually selected node, if pinned.
- The scheduler’s priority order, which is the normal case.
- A deterministic fallback by stable node ID.

The scheduler ranks nodes by total pending work across both engines plus a coarse GPU utilization pressure score. It also counts requests the proxy itself just dispatched, so a burst of concurrent requests spreads out immediately instead of piling onto an idle node that hasn’t yet reported its workload.

The proxy forwards the entire request to the chosen node over mutual TLS. The node’s engine performs inference and streams the response back through the proxy. The client sees an ordinary reply and never learns which machine served it.

What happens on failure

If an advertised owner returns a 404 for a model, the proxy treats it as retryable and moves to the next candidate. It never fails over to an unknown or missing node. Genuine client errors like 400 or 422 are not retried because they would fail identically everywhere. If no advertised owner is routable, the proxy returns a local 502 rather than guessing.

Discovery and identity

Nodes announce themselves on the local network using multicast DNS. The nvpair-node-scanner advertises one consolidated record per host that carries the ports for node info, proxies, pairing, workload propagation, and remote engine control. Each node is identified by a stable UUID, never by hostname, so renaming a machine doesn’t break the cluster.

Pairing is the trust decision. A six-digit PIN bootstraps certificate exchange, after which all cluster traffic runs over mutual TLS restricted to paired nodes. An unpaired machine on the same subnet can see discovery announcements but cannot read model inventories or drive remote engines. Local applications reach the proxy over loopback HTTP, which requires no certificates.

Ports and compatibility

PAIR inverts the usual port assignment so existing clients work unchanged NVIDIA PAIR Getting Started. The proxy takes the standard Ollama or LM Studio port, and the engine moves behind it. For example, Ollama moves from port 11434 to 11435 or higher, while the proxy listens on 11434. A client already configured for Ollama therefore gains cluster routing with zero reconfiguration. One deliberate trade-off: host and GPU telemetry are intentionally plaintext on the LAN, while inference traffic between nodes stays encrypted over mutual TLS.

What makes the routing “workload-informed”

The scheduler tracks pending work per node, counting both locally originated and peer-relayed workloads. GPU telemetry is sampled at the same healthy cadence and mapped to three coarse pressure thresholds. Nodes sort by pending plus pressure, then by pressure alone, then by node ID for deterministic cold-start behavior.

This is conservative by design. The scheduler doesn’t know which engine uses which GPU, doesn’t model available VRAM, doesn’t predict output length, and doesn’t consider whether a model is already warm in memory. Instead, it balances work among the nodes that can serve the request. The result is useful load feedback without a complete capacity model.

Why PAIR is good for multi-user systems

When several users share an AI workflow, whether through a multi-agent application, a local team tool, or parallel development sessions, the bottleneck is usually a single machine’s GPU and memory. PAIR addresses this in concrete ways.

More aggregate throughput from existing hardware

If one node holds a 70B model and another holds a 7B model, or if both nodes hold the same model, PAIR routes requests across both. A cluster with two RTX 4090s can handle more concurrent inference jobs than a single 4090 because the second machine absorbs overflow when the first is saturated. Because PAIR’s proxy presents the same familiar endpoint, applications don’t need to know the cluster exists.

Better hardware utilization

Round-robin balancing leaves GPUs idle while other nodes queue work. PAIR’s scheduler considers pending jobs and GPU pressure together, so a node running one long generation doesn’t hide another node that could immediately serve a short request. Each proxy also tracks its own recent dispatches to prevent burst traffic from overwhelming a just-discovered idle node before workload reports arrive.

Privacy-preserving local scale

PAIR keeps prompts, files, and agent context on the local network. For teams or individuals who can’t send data to cloud APIs, this is the only way to serve multiple concurrent users without exposing sensitive material. Mutual TLS between nodes ensures that even routed traffic stays encrypted.

No client-side rewrites

PAIR speaks Ollama-compatible and OpenAI-compatible protocols. Tools like Claude Code, Cursor, custom agents, and standard OpenAI SDKs work unchanged. The routing decision happens transparently behind the proxy.

Observable routing

The Jobs view shows which node served each request. This makes it possible to audit routing decisions, verify that model placement is working as intended, and diagnose why a particular request slowed down.

Limitations and trade-offs

PAIR is open about its limits, and understanding them matters for anyone planning a multi-user deployment.

No GPU pooling: Nodes do not share VRAM. A model must fit entirely on one machine.
No request splitting: One request cannot span multiple GPUs or machines.
Scheduler sees pressure, not capacity: The same utilization percentage maps to the same pressure on a small and large GPU. A mixed cluster can still favor a slower machine.
Multi-GPU nodes use the busiest device: The telemetry feed reports maximum GPU utilization, which can overlook idle capacity elsewhere on the same node.
Every workload counts equally: A three-token completion and a long generation are both one unit of pending work, so “fewest jobs” is not “least busy.”
Model load state is not considered: Routing doesn’t know whether a model is already warm in memory, so a request can be sent to a node that must cold-load it.
Only PAIR-routed work counts: Inference sent directly to an engine’s own port is invisible to the scheduler.

The team acknowledges these gaps openly and has them on the roadmap, including KV cache-aware scheduling, request queuing, requester-only clients, and support for additional engines like vLLM and llama.cpp.

The bigger picture

Inference routing is becoming a first-class concern across the industry. Google, AWS, and Red Hat have deployed production systems that replace round-robin with request-aware placement, and Microsoft Research has published research on workload-aware routing. The results are consistent: better throughput, lower tail latency, and more concurrent users on the same hardware.

PAIR brings that same idea to the edge. It doesn’t replace cloud inference for massive scale, but for teams, developers, and households running local AI agents and applications, it turns a collection of idle or underused machines into a coherent cluster. The architecture is peer-to-peer, the security model is explicit, and the client experience is invisible.

If you’re building a system where multiple users or agents will issue concurrent AI requests from the same local environment, PAIR is a concrete reference for how inference routing can make that system faster, more private, and more efficient, without requiring you to write a custom scheduler from scratch.

References

Jimatech: Your partner in Data and AI