Skip to main content
Documentation

Install with Gateway API Inference Extension

This guide provides step-by-step instructions for integrating vLLM Semantic Router (vSR) with a Gateway API Inference Extension (GIE) conformant inference gateway. GIE lets you manage self-hosted, OpenAI-compatible models with Kubernetes-native APIs such as InferencePool, while vSR adds prompt-aware model routing through the gateway's ExtProc integration.

Docs indexEdit page
Version: Latest

Install with Gateway API Inference Extension

This guide provides step-by-step instructions for integrating vLLM Semantic Router (vSR) with a Gateway API Inference Extension (GIE) conformant inference gateway. GIE lets you manage self-hosted, OpenAI-compatible models with Kubernetes-native APIs such as InferencePool, while vSR adds prompt-aware model routing through the gateway's ExtProc integration.

Architecture Overview​

The deployment consists of three main components:

  • vLLM Semantic Router: Classifies incoming requests and selects the target model.
  • GIE-conformant inference gateway: Provides the Kubernetes Gateway API data plane. This guide includes tabs for Istio and agentgateway.
  • Gateway API Inference Extension (GIE): Provides Kubernetes-native inference APIs such as InferencePool for load-aware backend selection.

Benefits of Integration​

Integrating vSR with GIE provides a robust, Kubernetes-native solution for serving LLMs with several key benefits:

1. Kubernetes-Native LLM Management​

Manage your models, routing, and scaling policies directly through kubectl using familiar Custom Resource Definitions (CRDs).

2. Intelligent Model and Replica Routing​

Combine vSR's prompt-based model routing with GIE's smart, load-aware replica selection. This ensures requests are sent not only to the right model but also to the healthiest replica, all in a single, efficient hop.

3. Protect Your Models from Overload​

The built-in scheduler tracks backend load and request queues, automatically shedding traffic to prevent your model servers from crashing under high demand.

4. Deep Observability​

Gain insights from both high-level Gateway metrics and detailed vSR performance data (like token usage and classification accuracy) to monitor and troubleshoot your entire AI stack.

5. Secure Multi-Tenancy​

Isolate tenant workloads using standard Kubernetes namespaces and HTTPRoutes. Apply rate limits and other policies while sharing a common, secure gateway infrastructure.

Supported Backend Models and APIs​

The demo models in this guide use the llm-d inference simulator to emulate Llama3 and Phi-4 through an OpenAI-compatible API. The simulator keeps the walkthrough runnable on a local kind cluster without GPUs or model downloads. You can replace it with your own model servers as long as the Semantic Router configuration, gateway routing resources, and GIE backend configuration agree on the request format and backend target.

OpenAI-compatible APIs are not the only supported option. agentgateway custom providers can declare provider-native formats such as OpenAI chat completions, Anthropic messages, responses, embeddings, token counting, and realtime APIs, and can route those providers to a host, Kubernetes Service, or InferencePool. GIE endpoint picker implementations can also support additional request formats through parser configuration; for example, the llm-d EPP parser framework includes OpenAI, Anthropic, vLLM HTTP, vLLM gRPC, Vertex AI, and passthrough parsers.

For details, see the agentgateway custom providers guide and the llm-d EPP request parser documentation.

Prerequisites​

Before starting, ensure you have the following tools installed:

  • Docker or another container runtime.
  • kind v0.22+ or any Kubernetes 1.29+ cluster.
  • kubectl v1.30+.
  • Helm v3.14+.
  • istioctl v1.28+ if you choose the Istio tab.

You can validate your toolchain versions with the following commands:

kind version
kubectl version --client --short
helm version --short
istioctl version --remote=false

Step 1: Create a Kind Cluster (Optional)​

If you don't have a Kubernetes cluster, create a local one for testing:

kind create cluster --name vsr-gie

# Verify the cluster is ready
kubectl wait --for=condition=Ready nodes --all --timeout=300s

Step 2: Install Gateway API and GIE CRDs​

Install the shared CRDs for Gateway API and GIE before installing a gateway implementation:

export GATEWAY_API_VERSION=v1.5.0
export GIE_VERSION=v1.5.0

# Install Gateway API CRDs
kubectl apply --server-side --force-conflicts \
  -f "https://github.com/kubernetes-sigs/gateway-api/releases/download/${GATEWAY_API_VERSION}/standard-install.yaml"

# Install Gateway API Inference Extension CRDs
kubectl apply -f "https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/${GIE_VERSION}/manifests.yaml"

# Verify CRDs are installed
kubectl get crd | grep 'gateway.networking.k8s.io'
kubectl get crd | grep 'inference.networking.k8s.io'

Step 3: Install an Inference Gateway​

Choose the inference gateway you want to use. Each tab only contains gateway-specific installation steps.

Install Istio with support for Gateway API and external processing:

# Download and install Istio
export ISTIO_VERSION=1.29.0
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=$ISTIO_VERSION sh -
export PATH="$PWD/istio-$ISTIO_VERSION/bin:$PATH"
istioctl install -y --set profile=minimal --set values.pilot.env.ENABLE_GATEWAY_API=true

# Verify Istio is ready
kubectl wait --for=condition=Available deployment/istiod \
  -n istio-system \
  --timeout=300s

Step 4: Deploy Demo LLM Servers​

Deploy two lightweight inference simulator instances, Llama3 and Phi-4, to act as backends. These simulator deployments do not require GPUs or a Hugging Face token, and their labels match the InferencePool selectors used later in this guide.

# Deploy the simulator model servers
kubectl apply -f https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/kubernetes/llmd-base/inference-sim.yaml

# Wait for simulator models to be ready
kubectl wait --for=condition=Available deployment/vllm-llama3-8b-instruct --timeout=300s
kubectl wait --for=condition=Available deployment/phi4-mini --timeout=300s

The demo manifest runs in the default namespace and exposes services named vllm-llama3-8b-instruct and phi4-mini.

Step 5: Deploy vLLM Semantic Router​

Deploy the vLLM Semantic Router using its official Helm chart. This component runs as an ExtProc server that the selected gateway calls for routing decisions.

helm upgrade -i semantic-router oci://ghcr.io/vllm-project/charts/semantic-router \
  --version v0.0.0-latest \
  --namespace vllm-semantic-router-system \
  --create-namespace \
  -f https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/kubernetes/istio/semantic-router-values/values.yaml

# Wait for the router to be ready
kubectl -n vllm-semantic-router-system wait \
  --for=condition=Available deploy/semantic-router \
  --timeout=10m

The values file configures vSR to select llama3-8b for general prompts and phi4-mini for math prompts.

Step 6: Deploy Gateway and Routing Logic​

Apply the gateway-specific resources that create the public-facing Gateway, attach vSR as an ExtProc service, and route selected models to GIE InferencePools.

Apply the existing Istio gateway, InferencePool, HTTPRoute, DestinationRule, and EnvoyFilter resources:

# Apply all routing and gateway resources
kubectl apply -f https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/kubernetes/istio/gateway.yaml
kubectl apply -f https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/kubernetes/llmd-base/inferencepool-llama.yaml
kubectl apply -f https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/kubernetes/llmd-base/inferencepool-phi4.yaml
kubectl apply -f https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/kubernetes/llmd-base/httproute-llama-pool.yaml
kubectl apply -f https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/kubernetes/llmd-base/httproute-phi4-pool.yaml
kubectl apply -f https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/kubernetes/istio/destinationrule.yaml
kubectl apply -f https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/kubernetes/istio/envoyfilter.yaml

# Verify the Gateway is programmed by Istio
kubectl wait --for=condition=Programmed gateway/inference-gateway --timeout=120s
kubectl wait --for=condition=Available deployment/llm-d-inference-scheduler-llama3-8b --timeout=300s
kubectl wait --for=condition=Available deployment/llm-d-inference-scheduler-phi4-mini --timeout=300s

The Istio-specific EnvoyFilter inserts the ExtProc filter that calls the semantic-router service.

Testing the Deployment​

Port Forwarding​

Set up port forwarding to access the selected gateway from your local machine.

# The Gateway service is named inference-gateway-istio
kubectl port-forward svc/inference-gateway-istio 8080:80

Send Test Requests​

Once port forwarding is active, send OpenAI-compatible requests to localhost:8080.

Test 1: Explicitly request a model

This request should be served by the Llama simulator. Add -i to the command if you want to inspect response headers such as x-inference-pod and x-vsr-selected-model.

curl -sS http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "llama3-8b",
    "messages": [{"role": "user", "content": "Summarize the Kubernetes Gateway API in three sentences."}]
  }'

Test 2: Let the Semantic Router choose the model

By setting "model": "auto", you ask vSR to classify the prompt. It will identify this as a math query and add the x-selected-model: phi4-mini header, which HTTPRoute uses to route the request to the Phi-4 InferencePool. In the agentgateway tab, the AgentgatewayPolicy runs in the PreRouting phase so the header is present before route matching.

curl -sS http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "What is 2+2 * (5-1)?"}],
    "max_tokens": 64
  }'

Troubleshooting​

Problem: CRDs are missing

If you see errors like no matches for kind "InferencePool", check that the CRDs are installed.

# Check for GIE CRDs
kubectl get crd | grep inference.networking.k8s.io

Problem: Gateway is not ready

If kubectl port-forward fails or requests time out, check the Gateway status.

# The Programmed condition should be True
kubectl get gateway inference-gateway -o yaml

Problem: vSR is not being called

If requests work but routing seems incorrect, check the Istio proxy logs for ext_proc errors.

# Get the Istio gateway pod name
export ISTIO_GW_POD=$(kubectl get pod -l istio=ingressgateway -o jsonpath='{.items[0].metadata.name}')

# Check its logs
kubectl logs $ISTIO_GW_POD -c istio-proxy | grep ext_proc

Problem: Requests are failing

Check the logs for vSR and the backend models.

# Check vSR logs
kubectl logs deploy/semantic-router -n vllm-semantic-router-system

# Check backend logs
kubectl logs deployment/vllm-llama3-8b-instruct
kubectl logs deployment/phi4-mini

Cleanup​

To remove all resources created in this guide, run the cleanup commands for the gateway tab you used.

# Delete routing and gateway resources
kubectl delete -f https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/kubernetes/istio/envoyfilter.yaml
kubectl delete -f https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/kubernetes/istio/destinationrule.yaml
kubectl delete -f https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/kubernetes/llmd-base/httproute-phi4-pool.yaml
kubectl delete -f https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/kubernetes/llmd-base/httproute-llama-pool.yaml
kubectl delete -f https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/kubernetes/llmd-base/inferencepool-phi4.yaml
kubectl delete -f https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/kubernetes/llmd-base/inferencepool-llama.yaml
kubectl delete -f https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/kubernetes/istio/gateway.yaml

# Delete demo models
kubectl delete -f https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/kubernetes/llmd-base/inference-sim.yaml

# Uninstall Helm releases and Istio
helm uninstall semantic-router -n vllm-semantic-router-system
istioctl uninstall -y --purge

# Delete the kind cluster, if you created it
kind delete cluster --name vsr-gie

Next Steps​

  • Customize Routing: Modify the values.yaml file for the semantic-router Helm chart to define your own routing categories and rules.
  • Add Your Own Models: Replace the demo Llama3 and Phi-4 deployments with your own OpenAI-compatible model servers.
  • Explore Advanced GIE Features: Look into using InferenceObjective for more advanced autoscaling and scheduling policies.
  • Monitor Performance: Integrate your Gateway and vSR with Prometheus and Grafana to build monitoring dashboards.