0xmmo/190proof

LLM router with aggressive normalization and stubborn error-handling. - 0xmmo/190proof

16 stars · 1 forks

README

190proof An opinionated unified interface for interacting with multiple AI providers including OpenAI, Anthropic, Google, Groq, OpenRouter, and AWS Bedrock. This package provides a consistent API for making requests to different LLM providers while handling retries, streaming, and multimodal inputs. Features Fully-local unified interface across multiple AI providers that includes:

🛠️ Consistent function/tool calling across all providers 💬 Consistent message alternation & system instructions 🖼️ Image format & size normalization 🔄 Automatic retries with configurable attempts 📡 Streaming by default ☁️ Cloud service providers supported (Azure, AWS Bedrock) 🔌 Provider prefix strings for any model without waiting for package updates

Installation npm install 190proof

Usage Basic Example Use any model from any provider with the provider:model-id format: import { callWithRetries, GenericPayload } from "190proof";

const payload: GenericPayload = { model: "openai:gpt-4o-mini", messages: [ { role: "user", content: "Tell me a joke.", }, ], };

const response = await callWithRetries("my-request-id", payload); console.log(response.content);

Using Different Providers import { callWithRetries, GenericPayload } from "190proof";

// OpenAI const openaiPayload: GenericPayload = { model: "openai:gpt-5", messages: [{ role: "user", content: "Hello!" }], };

// Anthropic const claudePayload: GenericPayload = { model: "anthropic:claude-sonnet-4-5", messages: [{ role: "user", content: "Hello!" }], };

// Google const geminiPayload: GenericPayload = { model: "google:gemini-2.0-flash", messages: [{ role: "user", content: "Hello!" }], };

// Groq const groqPayload: GenericPayload = { model: "groq:llama-3.3-70b-versatile", messages: [{ role: "user", content: "Hello!" }], };

// OpenRouter const openRouterPayload: GenericPayload = { model: "openrouter:google/gemma-4-31b-it:free", messages: [{ role: "user", content: "Hello!" }], };

const response = await callWithRetries("request-id", claudePayload);

With Function Calling const payload: GenericPayload = { model: "openai:gpt-4o", messages: [ { role: "user", content: "What is the capital of France?", }, ], functions: [ { name: "get_country_capital", description: "Get the capital of a given country", parameters: { type: "object", properties: { country_name: { type: "string", description: "The name of the country", }, }, required: ["country_name"], }, }, ], };

const response = await callWithRetries("function-call-example", payload); // response.function_call contains { name: string, arguments: Record<string, any> }

With Images const payload: GenericPayload = { model: "anthropic:claude-sonnet-4-5", messages: [ { role: "user", content: "What's in this image?", files: [ { mimeType: "image/jpeg", url: "https://example.com/image.jpg", }, ], }, ], };

const response = await callWithRetries("image-example", payload);

How images reach the model depends on the provider. OpenAI, Anthropic, and Google get native image blocks. Groq is text-only: images degrade to an inline Image (url) text reference. OpenRouter sends OpenAI-style image_url content parts (the remote URL when present, else a data: URI) — but if the model has no vision-capable endpoints, OpenRouter rejects the request with a routing-layer 404, so the retry loop resends the payload with images degraded to the same inline text references Groq gets, and remembers the model (in-process, until restart) so later calls degrade up front. Messages without image attachments serialize identically either way. With System Messages const payload: GenericPayload = { model: "google:gemini-2.0-flash", messages: [ { role: "system", content: "You are a helpful assistant that speaks in a friendly tone.", }, { role: "user", content: "Tell me about yourself.", }, ], };

const response = await callWithRetries("system-message-example", payload);

Inspecting Model Routing Use parseModelString to see how a model string will be routed: import { parseModelString } from "190proof";

parseModelString("openai:gpt-7"); // → { provider: "openai", modelId: "gpt-7" }

parseModelString("openrouter:org/model-name:free"); // → { provider: "openrouter", modelId: "org/model-name:free" }

Provider Prefix Format The model string format is provider:model-id, where the provider prefix is one of:

Prefix Provider

openai OpenAI

anthropic Anthropic

google Google (Gemini)

groq Groq

openrouter OpenRouter

The prefix is stripped before sending to the API, so the model ID should be exactly what the provider expects (e.g. "openai:gpt-4o" sends "gpt-4o" to OpenAI). Supported Models These models are tested. You can use any model with the provider:model-id format. OpenAI

openai:gpt-5 openai:gpt-5-mini openai:gpt-4.1 openai:gpt-4.1-mini openai:gpt-4.1-nano openai:gpt-4o openai:gpt-4o-mini openai:o3-mini openai:o1-preview openai:o1-mini

Anthropic

anthropic:claude-opus-4-5 anthropic:claude-sonnet-4-5 anthropic:claude-haiku-4-5 anthropic:claude-opus-4-1 anthropic:claude-opus-4-20250514 anthropic:claude-sonnet-4-20250514 anthropic:claude-3-5-sonnet-20241022 anthropic:claude-3-5-haiku-20241022

Google

google:gemini-3.1-flash-lite-preview google:gemini-3-flash-preview google:gemini-2.5-flash-preview-04-17 google:gemini-2.0-flash google:gemini-2.0-flash-exp-image-generation google:gemini-1.5-pro-latest

Groq

groq:llama-3.3-70b-versatile groq:llama3-70b-8192 groq:qwen/qwen3-32b groq:deepseek-r1-distill-llama-70b

OpenRouter

openrouter:google/gemma-4-31b-it:free openrouter:google/gemma-4-31b-it

Environment Variables Set the following environment variables for the providers you want to use:

OpenAI

Anthropic

Google

Groq

OpenRouter

AWS Bedrock (for Anthropic via Bedrock)

API Reference callWithRetries(identifier, payload, config?, retries?, chunkTimeoutMs?) Main function to make requests to any supported AI provider. Parameters

identifier: string | string[] - Unique identifier for the request (used for logging) payload: GenericPayload - Request payload containing model, messages, and optional functions config: OpenAIConfig | AnthropicAIConfig - Optional configuration for the specific provider retries: number - Number of retry attempts (default: 5) chunkTimeoutMs: number - Timeout for streaming chunks in ms (default: 15000)

Optional per-request knobs live on payload (GenericPayload):

payload.requestTimeoutMs: number - Per-attempt HTTP timeout in ms (default: 120000), honored by every adapter — except streaming OpenRouter attempts, which it deliberately does NOT bound (see below). For OpenRouter's non-streaming transport the default is 180000. payload.streaming: boolean - OpenRouter-only (default: true). Streams the completion over SSE. A streaming attempt is bounded by two independent timers instead of requestTimeoutMs: streamTimeoutMs (total wall clock, default 600000) and the per-useful-chunk stall timeout (chunkTimeoutMs argument, default 15000). A chunk is "useful" only if it advances content, reasoning, tool-call fragments, finish_reason, or usage — SSE comment keep-alives (: OPENROUTER PROCESSING) and role-only deltas don't reset the stall timer, so a hung provider dies within one stall window while a healthy long generation can run to the total budget. Set streaming: false for the old single-JSON-body transport. payload.streamTimeoutMs: number - OpenRouter-only: total wall-clock budget per streaming attempt (default: 600000). payload.streamDeadlineAt: number - OpenRouter-only: absolute deadline (epoch ms) for the whole call including retries — the caller's turn budget. Each attempt gets min(streamTimeoutMs, deadline - now), and once under 10s remain the call fails fast instead of starting a generation that cannot be delivered. Use it whenever the caller has its own timeout: a per-attempt budget alone is re-granted on every retry and can outlive that timeout. payload.thinkingConfig: Record<string, unknown> - Google-only: forwarded verbatim as generationConfig.thinkingConfig on the Gemini request — e.g. { thinkingBudget: 0 } to disable thinking, { thinkingLevel: "HIGH" } on models that take a level. Ignored by all other adapters; shapes are model-specific and validated by Google, not the SDK. payload.reasoningEffort: string - OpenAI and OpenRouter reasoning effort. Valid values are model-dependent (none/minimal/low/medium/high/xhigh/max). OpenRouter: sent as the nested reasoning: { effort } object — the canonical form, and the only one that accepts max (the flat reasoning_effort field caps at xhigh). Direct OpenAI: sent flat as reasoning_effort; max is rejected there, and reasoning-by-default models (the gpt-5.6 family) reject function tools on /chat/completions with a 400 unless this is explicitly "none" — their implicit default is medium. Via OpenRouter the same models accept tools at any effort (OpenRouter fronts /v1/responses), so omitting this runs them at their native default. Ignored by all other adapters.

When a streaming attempt is cut at its total deadline and prose has already arrived, the partial answer is returned with truncated: true on the response rather than discarded — those tokens were generated and billed, so throwing them away costs money and gives the user nothing. Surface such a reply as incomplete. Salvage never applies to tool-call turns (half-streamed arguments are unparseable JSON), to stalls (the provider died mid-thought), or to caller aborts. When nothing is salvageable, the discard is logged with an approximate token count — aborted attempts never receive OpenRouter's usage chunk, so that log line is the only record of the wasted spend. OpenRouter retries also perform moderation eviction: a provider content-moderation rejection (e.g. "Upstream error from Alibaba: Output data may contain inappropriate content.") is deterministic for a given payload, so on the first one the refusing provider is removed from the request's provider preferences (ignore += slug, order -= slug) and every remaining attempt reroutes to the next provider. Non-moderation errors retry with unchanged preferences, and fallbackModel still applies if the whole pool refuses.

payload.signal: AbortSignal - Caller-supplied cancellation. When it aborts, the in-flight provider request is cancelled and callWithRetries rejects immediately — it does not retry or fall back (both the retry loop and the fallback branch bail on signal.aborted). Threaded to the underlying fetch/axios/SDK call of each provider.

Returns Promise<ParsedResponseMessage>: interface ParsedResponseMessage { role: "assistant"; content: string | null; function_call: FunctionCall | null; function_calls: FunctionCall[]; files: File[]; // For models that return files (e.g., image generation) // Who actually served the response: OpenRouter's upstream provider from the // response body (e.g. "Baidu"), or the SDK provider name ("anthropic", ...) // for direct providers. On fallback, reflects the model that answered. provider?: string; usage: { prompt_tokens: number; completion_tokens: number; total_tokens: number; // Reasoning/thinking tokens spent before the visible answer; currently // populated from Google's usageMetadata.thoughtsTokenCount. thoughts_tokens?: number; } | null; // null when streaming }

parseModelString(model) Parses a model string into its provider and model ID components. Parameters

model: string - A model string in "provider:model-id" format

Returns { provider: Provider, modelId: string } Configuration Options OpenAI Config interface OpenAIConfig { service: "azure" | "openai"; apiKey: string; /** * Optional base URL. Defaults to `https://api.openai.com/v1`. Set to point * at any OpenAI-compatible endpoint (e.g. a self-hosted proxy). The path * `/chat/completions` is appended automatically. Ignored for Azure. */ baseUrl?: string; orgId?: string; modelConfigMap?: Record< string, { resource: string; deployment: string; apiVersion: string; apiKey: string; endpoint?: string; } >; }

To talk to an OpenAI-compatible server instead of OpenAI itself: await callWithRetries( "my-identifier", { model: "openai:gpt-4o-mini", messages: [{ role: "user", content: "hi" }], }, { service: "openai", apiKey: process.env.SOME_SERVER_API_KEY, baseUrl: "https://your-proxy.example.com/v1", }, );

Anthropic Config interface AnthropicAIConfig { service: "anthropic" | "bedrock"; }

License ISC

source