System-level intelligence

Signal
before scale

Encoder-native system intelligence for mixture-of-model serving, built on Shannon signals, entropy folding, and neural-symbolic routing.

Public Beta Open white paper

Signals13

13 signal families spanning intent, safety, modality, context, and preference.

Selection12

12 selectors across symbolic policy, latency heuristics, reinforcement learning, and ML routing.

Surfaces03

One architecture across cpu-local, amd-local, and ci-k8s.

System thesis

Encoder priors. Shannon structure. Entropy folded into action.

A router should feel like a system brain: encoder-guided, entropy-aware, and ruthlessly clear.

Install locally See configuration

Core logic

Neural-symbolic routing, kept legible.

Encoder priors, Shannon mapping, entropy folding, and model selection stay visible from research prototypes to production paths.

Signal extraction

Encoder signals turn raw requests into legible semantic state.

Decision engine

Neural signals meet symbolic rules in auditable routing logic.

Plugin chain

Cache, safety, rewrite, and tracing attach as composable behaviors.

Intent-to-policy compile

Natural language intent compiles into neural-symbolic policy before execution begins.

Research-grade model selection

Selection stays measurable enough for papers, benchmarks, and production tuning.

System docs

Docs, papers, and product routes read as one system, not scattered collateral.

Routing Blueprint

How System Works

An interactive walkthrough of signal extraction, decision logic, and model routing behavior.

Shannon Mapping

Structural mapping from communication theory to the routing pipeline.

The user request is the raw source message before encoding.

Built on Encoder Models

Encoder-Based Intelligence

Purpose-built encoders read intent, rank relevance, and classify modality before generation begins.

Signal surfaces

Sequence classification, token labeling, embeddings, and reranking collapse into one system-intelligence layer.

SEQ_CLSSequence classification for domain, jailbreak, fact-check, and feedback routing.

TOKENToken labeling for PII and safety-sensitive spans that need localized intervention.

EMBEDEmbedding and rerank paths for semantic cache, similarity search, and candidate scoring.

Hugging Face Models

MOD

Multi-Modality

Detect and route text, image and audio inputs to the right modality-capable model.

Input

"Is machine learning related to AI?"

Tokenizer

[CLS]IsmachinelearningrelatedtoAI?[SEP]

Embedding

Token Emb

Segment Emb

Position Emb

h₀ = Σ

Encoder Block

×N

ATTNMulti-Head Attention

NORMAdd & Norm

FFNFeed-Forward

NORMAdd & Norm

Signals

CLS

Sentence-Level (CLS Token)[CLS] → Linear Head → "computer_science"TaskType: SEQ_CLS

DomainJailbreakFact-checkFeedbackModality

BIO

Token-Level (Per Token)Each token → BIO Label → O O B-LOC I-LOC OTaskType: TOKEN_CLS

PII Detection

EMB

Bi-Encodermean-pooling(h₁..hₙ) → [0.23, -0.41, 0.87, ...]TaskType: EMBEDDING

Semantic CacheSimilarityComplexity-CLJailbreak-CL

RER

Cross-Encoder[CLS] query [SEP] candidate [SEP] → scoreTaskType: CROSS_LEARNING

RerankMulti-Modal

BIE

Bi-Encoder Embeddings

Independently encode queries and candidates into dense vectors for similarity search and semantic caching.

XCE

Cross-Encoder Learning

Joint cross-attention scoring of query-candidate pairs for high-precision reranking.

CLS

Classification

Domain, jailbreak, PII and fact-check classification across 14 MMLU categories via ModernBERT with LoRA.

ATT

Full Attention

Bidirectional attention across tokens and sentences, with full context instead of causal masking.

2DM

2DMSE

Adjust embedding layers and dimensions at inference time to trade compute for accuracy on the fly.

Individual Contributor

Committer

aeft

Individual Contributor

Committer

Hao Wu

Individual Contributor

Committer

Qiping Pan

Individual Contributor

Maintainers, committers, and contributors across research, infrastructure, and open-source operations.

View All Team Members

Open-source ecosystem

Acknowledgements

vLLM Semantic Router is made possible by the open-source ecosystem.

HuggingFaceOpen project

EnvoyProxyOpen project

Dependency

KubernetesOpen project

Dependency

MilvusOpen project

Dependency

PrometheusOpen project

Dependency

GrafanaOpen project

Documentation

Architecture, written to be used.

Install, configure, train, and operate from one dense documentation graph.

Docs index

Community

Research and builders in one loop.

Papers, working groups, and contributors evolve the same system in public.

Community routes