VeloRoute v1.0•Sub-Millisecond Multi-Provider AI Gateway & Token Saviour

The Ultra-Fast AI Gateway with Zero Retention & Native Token Savings.

Eliminate LLM outages with automated sentinel circuit breakers, speculative hedging, and AST context compression. Route effortlessly between OpenAI, Anthropic, Gemini, Groq, and local Ollama with < 0.8ms overhead.

$docker run -d -p 8000:8000 -v veloroute_data:/data ghcr.io/mrgusux/veloroute:latest
Star on GitHub★★
veloroute-gateway :: live-proxy-stream

# Inbound request to unified gateway port 8000

POST /v1/chat/completions [200 OK] in 142ms

├─ Routing Engine: smart (classified as 'reasoning' query)

├─ Selected Provider: anthropic/claude-3-5-sonnet via healthy key pool #2

├─ Token-Saviour: AST Skeletonizer compressed context from 8,420 to 2,110 tokens (-74.9%)

├─ Guardrail: Shannon entropy scrubber redacted 2 API keys & unmasked on response

└─ Stream chunks delivered: 418 tokens at 94.2 tok/sec

Gateway Status: Operational•Overhead: 0.72ms
Memory: ~34MBActive Keys: 12Circuit: All Closed
< 0.8ms
Routing Overhead
Up to 75%
Context Cost Reduction
0 Outages
Sentinel Auto-Failover
219 / 219
Unit Tests Passing
Core Capabilities

Built for High-Throughput Production.

Engineered from the ground up for extreme concurrency, deterministic cost reduction, and mission-critical resilience.

< 0.8ms Overhead

Sub-Millisecond Wire Proxy

Deterministic routing layer adds less than 0.8ms latency. Pure async Python 3.12+ with zero heavy ML inference overhead.

Token Saviour

AST Context Skeletonizer

Strips implementation bodies from Python files, preserving typed function signatures and docstrings. Saves up to 75% context tokens.

Zero-Downtime

Mid-Stream Seamless Failover

If a provider halts mid-stream, VeloRoute reconnects to a healthy fallback provider and resumes without breaking client SSE connections.

Auto-Healing

Sentinel Circuit Breakers

3-state deterministic health machine (Closed, Open, Half-Open). Automatic exponential backoff with full jitter prevents cascading failures.

Zero-Retention

Shannon Entropy Secret Shield

Scans for high-entropy secrets and PII in real-time, redacting them before cloud transmission and unmasking them losslessly in responses.

MCP Native

Universal MCP SSE Bridge

Connects stdio and SSE Model Context Protocol tool servers directly to LLMs with automatic RTK token compression.

Interactive Simulator

The Token-Saviour Engine.

Stop paying frontier rates for function bodies, whitespace, and repetitive prompts. See exact dollar and latency savings calculated live.

Simulation ParametersDeterministic
50,000 reqs
5K250K500K
6,500 tokens
500 (Chat)16K (RAG)32K (Repo)
Monthly Savings
$761
$9,126 / year
Tokens Saved / Mo
253.5M
-78% payload reduction
Avg Latency Cut
43%
Fewer prompt bytes to ingest
Context Skeletonization Diff
Standard SDK Prompt (324 tokens)Uncompressed
def calculate_risk_index(
    user_portfolio: dict[str, float],
    volatility_matrix: list[list[float]],
    lookback_days: int = 90,
    stress_test_scenario: str | None = "2008_subprime",
    market_cap_weights: dict[str, float] | None = None,
) -> RiskAssessmentResult:
    """Computes multidimensional portfolio risk factoring covariance.
    
    Args:
        user_portfolio: Asset mapping to dollar balance
        volatility_matrix: Covariance array
    """
    logger.debug("Starting risk evaluation with %d assets", len(user_portfolio))
    if not user_portfolio:
        raise ValueError("Portfolio cannot be empty")
        
    normalized_weights = {}
    total_val = sum(user_portfolio.values())
    for asset, val in user_portfolio.items():
        # Strip out zero balances
        if val > 0:
            normalized_weights[asset] = val / total_val
            
    # Internal heavy matrix multiplication ...
    raw_var = _matrix_dot_product(normalized_weights, volatility_matrix)
    adj_var = _apply_monte_carlo_shocks(raw_var, iterations=10000)
    logger.info("Monte carlo run complete with status: NORMAL")
    
    return RiskAssessmentResult(score=adj_var.mean, p95=adj_var.p95)
VeloRoute AST Skeleton (76 tokens)-76.5% saved
def calculate_risk_index(
    user_portfolio: dict[str, float],
    volatility_matrix: list[list[float]],
    lookback_days: int = 90,
    stress_test_scenario: str | None = "2008_subprime",
    market_cap_weights: dict[str, float] | None = None,
) -> RiskAssessmentResult:
    """Computes multidimensional portfolio risk factoring covariance."""
    ...
⚡ Agent Performance: 100% of type signatures & docstring specs preserved. Zero hallucination increase.
High-Availability Routing & Resilience

Engineered for Pure Production Uptime.

No single point of failure. Explore how VeloRoute handles caching, failovers, parallel hedging, and enterprise zero-trust guardrails.

Hardware Cache Hits

Pin Prefixes to Warm GPU Instances to Slash 80% Input Latency & Cost

VeloRoute hashes incoming system prompts, large context windows, and codebase skeletons with SHA-256. Requests sharing the identical prefix are sticky-routed to the exact same provider cluster or vLLM node, unlocking maximum hardware KV-cache reuse.

Sub-millisecond SHA-256 deterministic prefix fingerprinting
Maximizes prompt caching on Anthropic, OpenAI & self-hosted vLLM/SGLang
Reduces TTFT (Time To First Token) from 850ms down to ~65ms
Zero coordination bottleneck across multi-tenant gateway instances
State Architecture // kv_affinityACTIVE ENGINE
Inbound System Prompt
42,000 token codebase AST
SHA-256 Hash
0x7c49...e2a
🎯 Affinity Worker Node [GPU-02]
KV Prefix Cache Status: WARM (HIT)
Cache Reuse: 94%Input Cost: -80%
Drop-In Compatibility

1 Line of Code to Switch.

No proprietary SDKs. Use the official OpenAI, Anthropic, or Ollama clients you already know. Simply repoint your base_url.

Seamless drop-in for OpenAI SDK. Simply repoint base_url to VeloRoute.python
import os
from openai import OpenAI

# Zero code modifications: just point base_url to VeloRoute!
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key=os.environ.get("VELOROUTE_KEY", "sk-velo-prod-key"),
)

# Route with intelligent fallbacks & AST compression
response = client.chat.completions.create(
    model="smart",  # Or 'gpt-4o', 'claude-3-5-sonnet', 'deepseek-r1'
    messages=[
        {"role": "system", "content": "You are a lead security architect."},
        {"role": "user", "content": "/terse Audit this auth flow for vulnerabilities."},
    ],
    temperature=0.2,
)

print(response.choices[0].message.content)
Architecture Benchmark

How VeloRoute Compares.

Built from first principles for ultra-low latency, zero mandatory database dependencies, and deterministic token optimization.

Capability⚡ VeloRouteLiteLLMPortkeyRaw Provider APIs
Gateway Routing OverheadPerformance
< 0.8ms (Microsecond)8 – 25ms15 – 45ms (Cloud RTT)0ms (Direct)
Memory FootprintPerformance
~34 MB RAM280+ MB RAMHosted Cloud0 MB
AST Python Context SkeletonizerToken Optimization
Caveman & Lossless ANSI TrimmingToken Optimization
KV-Cache Prefix Affinity RoutingLatency & Cost
Mid-Stream Automated FailoverResilience
Restart from chunk 0
Parallel Speculative Hedging (/hedge)Resilience
Sentinel 3-State Circuit BreakersResilience
Basic retry counterPartial (Cloud)
Shannon Entropy Zero-Retention ShieldSecurity
Regex onlyStored on Portkey Cloud
Universal MCP Bridge (stdio + SSE)Extensibility
Zero External Mandatory DatabasesSimplicity
Redis/Postgres requiredSaaS Database
100% Open-Source & Self-HostableLicensing
MIT LicenseEnterprise features lockedProprietary Cloud CoreN/A
Deployment Options

Run in 60 Seconds.

Deploy the gateway anywhere: single Docker container, production Kubernetes cluster, or host this Next.js site directly on Vercel for free.

Vercel Free Hosting

Deploy this landing page to Vercel

100% Free

Clone and deploy this ultra-fast Next.js landing page with automatic SSL, Edge CDN, and zero server maintenance.

Docker 1-Liner

Full Gateway & Dashboard

GHCR Ready

Run the multi-provider gateway and interactive telemetry web dashboard in a secure, unprivileged container.

docker run -d -p 8000:8000 -v veloroute_data:/data ghcr.io/mrgusux/veloroute:latest

Docker Compose

Persistent storage & key configs

Production

Clone the repository and run with pre-configured persistent volumes, key rotation, and health check probes.

git clone https://github.com/mrgusux/veloroute.git
cd veloroute
docker compose up -d

Enterprise Kubernetes

deploy/k8s/ manifests

Cloud Native

Deploy to AWS EKS, GCP GKE, or k8s via Helm chart or standard declarative manifests with zero-privilege securityContext and PVC persistence.

helm install veloroute ./deploy/helm/veloroute
kubectl apply -f deploy/k8s/

Local Python / CLI Quickstart

PyPI Ready

Run natively in Python 3.11+ virtual environments with built-in doctor diagnostics, recipe configurations, and zero heavy ML dependencies.

pip install -e . && veloroute ui