Built on Encoder Models

Encoder-Based Intelligence

Purpose-built encoder models extract meaning from every request — understanding intent, ranking relevance, and classifying content across modalities in real time.

Input

"Is machine learning related to AI?"

Tokenizer

[CLS]IsmachinelearningrelatedtoAI?[SEP]

Embedding

Token Emb

Segment Emb

Position Emb

h₀ = Σ

Encoder Block

×N

🔗Multi-Head Attention

➕Add & Norm

⚙️Feed-Forward

➕Add & Norm

Signals

🎯

Sentence-Level (CLS Token)[CLS] → Linear Head → "computer_science"TaskType: SEQ_CLS

DomainJailbreakFact-checkFeedbackModality

🏷️

Token-Level (Per Token)Each token → BIO Label → O O B-LOC I-LOC OTaskType: TOKEN_CLS

PII Detection

🌊

Bi-Encodermean-pooling(h₁..hₙ) → [0.23, -0.41, 0.87, ...]TaskType: EMBEDDING

Semantic CacheSimilarityComplexity-CLJailbreak-CL

🔀

Cross-Encoder[CLS] query [SEP] candidate [SEP] → scoreTaskType: CROSS_LEARNING

RerankMulti-Modal

🎭

Multi-Modality

Detect and route text, image and audio inputs to the right modality-capable model.

🧬

Bi-Encoder Embeddings

Independently encode queries and candidates into dense vectors for similarity search and semantic caching.

⚡

Cross-Encoder Learning

Joint cross-attention scoring of query-candidate pairs for high-precision reranking.

🤔

Classification

Domain, jailbreak, PII and fact-check classification across 14 MMLU categories via ModernBERT with LoRA.

👁️

Full Attention

Bidirectional attention across tokens and sentences — full context in both directions, not causal masking.

🪆

2DMSE

Adjust embedding layers and dimensions at inference time to trade compute for accuracy on the fly.

📐

MRL

Truncate embedding vectors to any dimension without retraining — balance accuracy and speed per request.

🏗️ Architecture

🎯 Our Goals

Building the System Level Intelligence for Mixture-of-Models (MoM), bringing Collective Intelligence into LLM systems

How to capture the missing signals in request, response and context?

How to combine the signals to make better decisions?

How to collaborate more efficiently between different models?

How to secure the real world and LLM system from jailbreaks, pii leaks, hallucinations?

How to collect the valuable signals and build a self-learning system?

📍 Where it lives

It lives between the real world and models

👥 Meet Our Team

The amazing people behind vLLM Semantic Router

Maintainer

Huamin Chen

Distinguished Engineer @Red Hat

Maintainer

Chen Wang

Senior Staff Research Scientist @IBM

Maintainer

Yue Zhu

Staff Research Scientist @IBM

Maintainer

Xunzhuo Liu

Intelligent Routing @vLLM

Committer

Senan Zedan

R&D Manager @Red Hat

Committer

samzong

AI Infrastructure / Cloud-Native PM @DaoCloud

Committer

Liav Weiss

Software Engineer @Red Hat

Committer

Asaad Balum

Senior Software Engineer @Red Hat

Committer

Yehudit

Software Engineer @Red Hat

Committer

Noa Limoy

Software Engineer @Red Hat

Committer

JaredforReal

Software Engineer @Z.ai

Committer

Srinivas A

Software Engineer @Yokogawa

Committer

carlory

Open Source Engineer @DaoCloud

Committer

Yossi Ovadia

Senior Principal Engineer @Red Hat

Committer

Jintao Zhang

Senior Software Engineer @Kong

Committer

yuluo-yx

Individual Contributor

Committer

cryo-zd

Individual Contributor

Committer

OneZero-Y

Individual Contributor

Committer