| Strategy | What it does |
|---|---|
| Manual | Pick a single model; only that model routes to this provider. |
| Balanced | Default — balances intelligence, speed, and reliability via Thompson sampling. |
| Smartest | Always pick the highest-ranked available model. |
| Iterative Refinement | Iteratively refine responses across multiple models. |
| Fastest | Prioritise lowest-latency completions. |
| Most reliable | Prioritise uptime and lowest error rates. |
| Custom | Provider-tailored combination of metrics. |
| Racing | Race multiple models and return the fastest winner. |
| Auto | Run all five dynamic strategies (balanced, smartest, fastest, reliable, racing (currently not included until racing is out of beta) as bandit arms and converge on the best one using existing telemetry as rewards. |
Tip
The Auto strategy is a Thompson-sampled meta-bandit with five arms. Each arms corresponds to a dynamic strategy under the hood. Rewards are computed from the project's existing request telemetry (latency, error rates, quality signals). Over time, Auto converges on whichever strategy gives the best outcome for the provider — without manual tuning.
Prerequisites: Node.js 20+, npm.
git clone https://github.com/animaios/animarouter.git
cd animarouter
npm install
cp .env.example .env
# Generate an encryption key for at-rest key storage
ENCRYPTION_KEY="$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')"
printf "ENCRYPTION_KEY=%s\nPORT=3001\n" "$ENCRYPTION_KEY" > .env
npm run devOpen http://localhost:5173 (the Vite dev UI), add your provider keys on the Keys page, reorder the Fallback Chain to taste, and grab your unified API key from the Keys page header. That unified key is what you point your OpenAI SDK at.
Reaching the dev UI from another device on your LAN? Use
npm run dev:lan— it passes--hostthrough to Vite. (Plainnpm run dev -- --hostdoes not work: the rootdevscript is aconcurrentlywrapper, so the flag never reaches Vite.)
npm run build
node server/dist/index.js # server + dashboard both served on :3001ENCRYPTION_KEY is required for startup. The server only falls back to a database-stored development key when DEV_MODE=true and NODE_ENV is not production; do not use that fallback with real provider keys.
Request analytics are retained for 90 days or 100000 request rows by default, whichever limit prunes first. Set REQUEST_ANALYTICS_RETENTION_DAYS=0 or REQUEST_ANALYTICS_MAX_ROWS=0 in .env to disable either retention limit.
Important
🚧 Under construction
(click to expand)
AnimaRouter ships a Cloudflare Workers proxy layer for IP rotation and header stripping. Deploy it to route requests through geographically-distributed exit IPs so upstream providers see consistent, non-identifying IP addresses instead of your real one.
Prerequisites: wrangler installed and logged in (npm i -g wrangler && wrangler login).
npm run proxy:upThat's it. The first run automatically:
- Checks wrangler auth
- Initializes the
freellmproxygit submodule - Installs proxy dependencies
- Generates secure secrets (
freellmproxy/.env) - Deploys proxy workers + router to Cloudflare
- Detects and prints your working endpoint URL
After deployment, register the proxy as a custom provider in the dashboard:
- Base64url-encode your target URL:
node -e "console.log(Buffer.from('https://api.example.com/v1').toString('base64url'))" - Construct:
https://{ROUTER_URL}/{AUTH_KEY}/{PROXY_NUM}/{BASE64_URL} - Add as a custom provider with that URL as the base URL
Custom domain (optional): Add ROUTER_DOMAIN=your.domain.com to freellmproxy/.env before deploying. This replaces the workers.dev subdomain with your own domain. The domain must be a Cloudflare-proxied zone.
| Command | Purpose |
|---|---|
npm run proxy:up |
Deploy everything to Cloudflare |
npm run proxy:dev |
Local dev server via wrangler |
npm run proxy:status |
Show deployment status |
npm run proxy:test |
Run proxy test suite |
Adjust PROXY_COUNT in freellmproxy/.env. See the proxy's README for the full architecture.
(click to expand)
When AnimaRouter falls over to a different model mid-conversation (quota, rate limit, cooldown), the new model has no idea it is picking up someone else's task. Context handoff adds a single compact system message to the outbound request that tells the new model exactly that:
AnimaRouter context handoff:
You are taking over an ongoing conversation from another model (groq:llama-3 → google:gemini-flash).
Continue the user's task using the conversation context already provided in this request.
Do not restart the task, re-ask already answered setup questions, or discard prior tool results.
Respect the user's latest message as the highest-priority instruction.
Recent session summary:
User: …
Assistant: …
Enable it in .env:
ANIMAROUTER_CONTEXT_HANDOFF=on_model_switchHow it works:
- Messages per session are stored in memory (TTL: 3 hours).
- Only injected when the selected model changes for a given session key.
- Not injected on the first request, on same-model continuations, or if a handoff message is already present.
- Session key:
X-Session-Idheader if present, otherwise SHA-1 of the first user message (same as sticky sessions). - Storage is in-memory only. Nothing is written to disk or logged.
Important: Context Handoff improves continuity for conversations routed through AnimaRouter. It cannot recover provider-internal hidden state or messages that were never sent to the proxy.
(click to expand)
Any OpenAI-compatible client works. Examples:
Python
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:3001/v1",
api_key="api-gateway-your-unified-key",
)
resp = client.chat.completions.create(
model="auto", # let the router pick; or specify e.g. "gemini-2.5-flash"
messages=[{"role": "user", "content": "Summarise the fall of Rome in one sentence."}],
)
print(resp.choices[0].message.content)
print("Routed via:", resp.headers.get("x-routed-via"))curl
curl http://localhost:3001/v1/chat/completions \
-H "Authorization: Bearer api-gateway-your-unified-key" \
-H "Content-Type: application/json" \
-d '{
"model": "auto",
"messages": [{"role": "user", "content": "hi"}]
}'Streaming
stream = client.chat.completions.create(
model="auto",
messages=[{"role": "user", "content": "Stream me a haiku about SQLite."}],
stream=True,
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="", flush=True)Tool calling
Pass OpenAI-style tools and tool_choice; the assistant response round-trips back through the proxy exactly like the OpenAI API. Multi-step flows (assistant tool_calls → tool role follow-up → final answer) work across every provider the router can reach.
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather for a city.",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}]
# 1. Model asks for a tool call
first = client.chat.completions.create(
model="auto",
messages=[{"role": "user", "content": "What's the weather in Karachi?"}],
tools=tools,
tool_choice="required",
)
call = first.choices[0].message.tool_calls[0]
# 2. You execute the tool, feed the result back
final = client.chat.completions.create(
model="auto",
messages=[
{"role": "user", "content": "What's the weather in Karachi?"},
first.choices[0].message,
{"role": "tool", "tool_call_id": call.id, "content": '{"temp_c": 32, "cond": "sunny"}'},
],
tools=tools,
)
print(final.choices[0].message.content)Vision / image input
Send images with the standard OpenAI image_url content blocks (base64 data: URLs or http(s) URLs). When a request contains an image, the router restricts itself to vision-capable models and ignores text-only ones. Vision models are tagged with a Vision badge on the Fallback Chain page.
resp = client.chat.completions.create(
model="auto", # auto-routes to a vision model
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "What's in this image?"},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,<...>"}},
],
}],
)
print(resp.choices[0].message.content)If no vision-capable model is enabled in your Fallback Chain, an image request returns 422 (code: "no_vision_model") rather than silently dropping the image.
Every response carries an X-Routed-Via: <platform>/<model> header so you can see which provider actually served each call. If a request fell over between providers, you'll also see X-Fallback-Attempts: N.
/v1/embeddings is OpenAI-compatible, with one deliberate difference from chat routing: failover never crosses models. Vectors from different models live in incompatible spaces — silently switching models would corrupt any vector store built on top of the proxy. So embeddings route by family (one model identity + dimension), and failover only walks the providers serving that same family.
resp = client.embeddings.create(
model="auto", # default family; or a family name like "bge-m3"
input=["the quick brown fox", "pack my box with five dozen liquor jugs"],
)
print(len(resp.data), "vectors of", len(resp.data[0].embedding), "dims")curl http://localhost:3001/v1/embeddings \
-H "Authorization: Bearer api-gateway-your-unified-key" \
-H "Content-Type: application/json" \
-d '{"model": "auto", "input": "hello world"}'model accepts auto (the configured default family), a family name, or a provider-specific model id (which resolves to its family). Available families:
Family (model) |
Dims | Providers (failover order) |
|---|---|---|
gemini-embedding-001 (default) |
3072 | |
text-embedding-3-large |
3072 | GitHub Models |
text-embedding-3-small |
1536 | GitHub Models |
embed-v4.0 |
1536 | Cohere |
bge-m3 |
1024 | Cloudflare → Hugging Face |
qwen3-embedding-0.6b |
1024 | Cloudflare |
nv-embedqa-e5-v5 |
1024 | NVIDIA |
llama-nemotron-embed-1b-v2 |
2048 | NVIDIA |
llama-nemotron-embed-vl-1b-v2 |
2048 | NVIDIA → OpenRouter |
embeddinggemma-300m |
768 | Cloudflare |
The default family, per-provider toggles, and priorities live on the dashboard's Models → Embeddings page.
Beyond the OpenAI-compatible endpoints above, the server exposes configuration APIs under /api. The new per-provider routing strategy endpoints:
| Method | Path | Purpose |
|---|---|---|
GET |
/api/fallback/routing/provider[?platform=] |
Read the current strategy for a provider. |
PUT |
/api/fallback/routing/provider |
Update the strategy for a provider ({ platform, strategy }). |
Configure these from the /models admin page or call them directly.
(click to expand)
Don't see yours? Add it. Any OpenAI-compatible endpoint — cloud service, local server, homelab GPU — becomes a provider in under a minute. It gets the same fallback chain, the same intelligent routing, the same rate-limit protection as every built-in. See Custom Providers →
The built-in provider list is a starting point, not a boundary. From the Keys page, the Platforms grid is the unified catalog — every built-in platform you've added a key for, alongside every custom platform you've registered. The grid ends with an Add New Platform tile that opens a modal for:
- Slug — a short identifier like
my-ollama(lowercase letters, digits, dashes; 2-32 chars; cannot collide with a built-in). - Display name — shown in the dashboard.
- Base URL — the OpenAI-compatible endpoint, e.g.
http://192.168.1.10:11434/v1. - Rate limits (optional) — RPM, RPD, TPM, TPD caps enforced per-provider.
- Max parallel requests (optional) — concurrency ceiling so this provider never hogs all connection slots.
Once a platform exists, its models are automatically discovered from the endpoint's /v1/models during creation by default. However, discovered models are disabled by default and must be manually enabled. You can also re-run discovery at any time via POST /api/custom-providers/:slug/sync-models, and there's an option to auto-enable discovered models during creation.
- Model ID and Display name — required.
- Context window, Supports tools, Supports vision — basic flags.
- Advanced toggle exposes intelligence rank, speed rank, size label, per-model rate limits, and max output tokens.
The model joins the fallback chain at the lowest priority and shows up everywhere built-in models do — /v1/models, the Fallback page, the Analytics page. You can edit any model (built-in or custom) later: adjust ranks, toggle tools/vision, cap output tokens, change rate limits — all from the dashboard.
Adding an API key for a custom platform works the same as for a built-in: pick the custom slug in the Add a provider key form, paste the bearer (or leave blank for local servers that don't need one), and the key routes to your endpoint.
Removing a custom platform archives it — models, keys, and fallback entries are preserved and restorable. Hard-delete is available but off by default.
(click to expand)
A self-hosted, single-user, personal-use setup was re-reviewed against each provider's ToS (May 2026). Summary:
| Provider | Verdict | Notes |
|---|---|---|
| Google Gemini | March 2026 ToS narrows scope to "professional or business purposes, not for consumer use" — a self-hosted developer proxy is still defensible, but the clause is new. | |
| Groq | ✅ Likely OK | GroqCloud Services Agreement permits Customer Application integration. |
| Cerebras | ✅ Likely OK | Permitted; explicitly forbids selling/transferring API keys. |
| Mistral | ✅ Likely OK | APIs allowed for personal/internal business use. |
| OpenRouter | ✅ Likely OK | April 2026 ToS sharpens the no-resale / no-competing-service clause; private single-user proxy still fine. |
| Cloudflare Workers AI | No anti-proxy clause; covered by general Self-Serve Subscription Agreement. | |
| NVIDIA NIM | Trial ToS §1.2 / §1.4: "evaluation only, not production." Free access is a recurring 40 RPM rate limit (the 2025 credit system was discontinued), but the evaluation-only scope stands. | |
| GitHub Models | Free tier explicitly scoped to "experimentation" and "prototyping." | |
| Cohere | ❌ Avoid | Terms §14 still forbids "personal, family or household purposes." |
| Zhipu (open.bigmodel.cn) | ✅ Likely OK | Personal/non-commercial research carve-out still in the platform docs. |
| Z.ai (api.z.ai) | Singapore entity (distinct from Zhipu CN). §III.3(l) anti-traffic-redirect clause could plausibly be read against a proxy; no explicit personal-use carve-out. | |
| Ollama Cloud | ✅ Likely OK | Free plan permits cloud-model access (1 concurrent, 5-hour session caps). No anti-proxy / anti-resale clauses found. |
| OVH AI Endpoints | ✅ Likely OK | Anonymous access is officially documented (2 req/min per IP per model). OVH reserves the right to introduce token/consumption caps. |
Rules of thumb: one account per provider, no reselling, no sharing your endpoint with other humans, don't hammer a free tier as a paid production backend. This is informational, not legal advice — read each provider's ToS and make your own call.
This project is for personal experimentation and learning, not production. Free tiers exist so developers can prototype against them; they aren't a stable, supported inference substrate and shouldn't be treated as one. If you build something real on top of AnimaRouter, swap in a paid API before you ship. Your relationship with each upstream provider is governed by the terms you accepted when you created your account — those terms still apply when the traffic is proxied through this project, and you're responsible for complying with them.
Contributors welcome! Good first PRs:
- Add an endpoint — images, moderations, audio. The provider base class can grow new methods; adapters declare which they support.
Development loop:
npm install
npm run dev # server on :3001, dashboard on :5173, both with HMR
npm test # server vitest + client typecheck
npm run build # compile server and dashboardOr use the api CLI: api start, api stop, api status.
PRs should include a test, keep the existing test suite green, and match the .editorconfig / tsconfig defaults. Issues and discussions are open.
Built on MLuqmanBR/api-gateway (itself a fork of tashfeenahmed/freellmapi). Maintained by AnimAIOS project.
MIT — © upstream contributors + AnimAIOS project.