NandhaKishorM/laya
Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per re...
29.5k stars · 2.6k forks
README
Multilingual, non-autoregressive System 1 decision engine. Typed decisions over 100+ languages in a single forward pass — 33 ms — trained with reinforcement learning against strictly proper scoring rules (RLCD), with a router that picks the right checkpoint per request.
Installation python -m pip install laya With uv, run uv add laya in a uv project or uv pip install laya in a virtual environment. Python 3.10 or newer. Optional extras: laya[serve] (HTTP server), laya[mcp] (MCP server), laya[langchain] (LangChain and LangGraph), laya[llamaindex] (LlamaIndex selectors), laya[crewai] (CrewAI routing), laya[onnx] (ONNX Runtime), laya[fast] (TileLang GPU fast path). Step-by-step setup for each platform, CPU-only or GPU PyTorch builds, and troubleshooting are in Installation details. For TypeScript / Node.js / browser, see laya-ts/. npm releases (npm install laya-ts) are published from this repository's laya-ts-v* release tags. Long documents. laya-multilingual reads up to 8,192 tokens with max_len=8192. Measured accuracy and time by document length, reproducible with research/scripts/bench_long_context.py:
Quickstart
Long documents: laya-multilingual reads up to 8,192 tokens. It ships with a 1,024-token limit that cuts long documents off, so pass max_len=8192 for them: result = router.predict(long_document, questions, model="multilingual", max_len=8192) In the table above, 16 to 18 of 20 requests were answered correctly with up to about 4,000 tokens of text before them; beyond that results vary (8 to 17 of 20), so check long-document accuracy on your own data. Short inputs give identical answers with max_len=8192, and speed follows the input's real length, not the limit: short inputs are unchanged, and a 4,000-token input takes about 1.7 s on an Apple GPU. Name the checkpoint with model="multilingual", since long mostly-English text would otherwise route to the English checkpoint.
from laya import Router
router = Router() # downloads a checkpoint on first use; Router(preload=True) loads all three up front
state = "Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan." questions = { "department": {"type": "choice", "instructions": "Which department should handle this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, system errors", "other": "everything else"}}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["not urgent", "soon", "blocking"]}, "churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"}, }
result = router.predict(state, questions) print(result["answers"]["department"]["choice"]) # billing print(result["answers"]["churn_risk"]["noul"]) # probability the answer is yes print(result["routing"]["model"]) # english The same call works in any of 100+ languages. The Router detects the script and language and sends non-English text to laya-multilingual: for text in ["मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।", "La aplicación se cierra cada vez que abro la configuración."]: r = router.predict(text, {"department": questions["department"]}) print(r["routing"]["model"], r["answers"]["department"]["choice"])
multilingual billing
multilingual technical
From the command line, laya "My payment failed twice" --preset triage answers a ready-made question set. More in the full quickstart and the docs. Fine-tune for better accuracy The shipped checkpoints work zero-shot, but fine-tuning on decisions from your own domain is where accuracy jumps. On the typed-decisions benchmark (2,000 decisions across four workflows), the fine-tuned laya-typed-decisions checkpoint scores 0.766 accuracy, against 0.362 for the base English checkpoint on the same decisions. Fine-tuning notebook: runs the whole loop on Kaggle's free 2x T4 GPUs (build the dataset, train, fit calibration temperatures, evaluate, and push the result to the Hub). Details in Fine-Tuning. Documentation nandhakishorm.github.io/laya: guides for prediction hooks, schema-driven decisions, Docker and LangChain and LangGraph, plus a full API reference. What's new in 0.3.21ONNX catches up with PyTorch. ONNXAgent gains predict_batch (with sort_by_length), predict_long and decide_batch, scripts/export_onnx.py --quantize writes a per-channel INT8 copy for CPU, and laya-evals run --onnx scores an export with the same gates as the torch path. Opt-in abstention. min_confidence= on predict, predict_batch, decide and decide_batch flags answers below a threshold on answer_confidence with low_confidence: True, and decide returns None for them. Batch everywhere. decide_batch, Router.predict_long, laya --batch FILE, the MCP laya_predict_batch / laya_route_batch / laya_decide tools, and LangChain batch() / abatch() all run on shared forward passes. New LayaDecision (LangChain), LlamaIndex selectors (laya[llamaindex]) and CrewAI routing (laya[crewai]). Per-request token budget. max_len / head_max_len now reach every surface: laya-serve (capped by LAYA_MAX_TOKEN_BUDGET), Router.predict_batch requests, the CLI (--questions, --max-len, --head-max-len), MCP tools and LangChain nodes. Operations. LAYA_MAX_LOADED, LAYA_REVISION and per-checkpoint SHA-256 maps; /health reports the device a checkpoint really runs on and its CPU-fallback count; the 503 busy answer carries Retry-After; compile=True no longer recompiles for every request shape. Stricter inputs. A null or duplicate choice label, a short temperature list, a None state and non-dict questions are refused with a message that names them, and usage["options"] says when the head budget left two options with the same tokens.
Laya evaluates typed questions (choice, score, noul) over any state (text, email, ticket or JSON document) in a single forward pass — 33 ms for one question, 7.2 ms/question batched, measured on a T4. No text generation, so nothing to parse and nothing to hallucinate. Three checkpoints, and a Router that picks between them per request:
encoder params context use it for
laya ModernBERT-large 421M 512 English
laya-multilingual mmBERT-base 322M 1024 (up to 8,192) 100+ languages, 2x faster
laya-typed-decisions ModernBERT-large 421M 1024 the typed-decisions workflows
Installation details Python 3.10 or newer. The dependencies set that floor: huggingface_hub 1.x, transformers 5.x and torch 2.14 all require 3.10. Optional PyTorch build selection: If you need a CPU-only or GPU-specific PyTorch build, follow PyTorch's installation guide after creating your virtual environment and before installing Laya. Replace pip or pip3 in the selected command with the environment's Python executable followed by -m pip. If you already use a virtual environment, install the PyPI release with: python -m pip install laya For a new environment, choose the commands for your platform below. Run them from your project directory; the explicit Python paths keep installation and verification in the same environment. macOS / Linux (with Python 3.10 or newer): On Debian/Ubuntu, the system Python may require sudo apt install python3-venv before creating a virtual environment. If venv reports that ensurepip is unavailable, install that package and retry. python3 -m venv .venv .venv/bin/python -m pip install laya .venv/bin/python -I -c "import laya; print(laya._version_)" Windows PowerShell (this example uses an installed Python 3.11): py -3.11 -m venv .venv .\.venv\Scripts\python.exe -m pip install laya .\.venv\Scripts\python.exe -I -c "import laya; print(laya._version_)" Both checks print the installed Laya version without loading a checkpoint. -I excludes the current directory from the import search path, so a local source copy cannot mask a missing installation. Keep using the same virtual environment's Python when running your application. Intel GPU (XPU) Install a supported Intel GPU driver first. For an XPU-enabled PyTorch build, install its wheel before Laya; the default PyPI wheel may be CPU-only. PyTorch's validated hardware and OS list is in the Intel GPU guide. .\.venv\Scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/xpu .\.venv\Scripts\python.exe -m pip install laya .\.venv\Scripts\python.exe -c "import torch; print(torch.xpu.is_available())" For a source checkout, replace pip install laya with pip install -e .. Laya automatically selects an available XPU when no device is specified; you can also request one explicitly with device="xpu" in laya.load() or Router(device="xpu"). Install from GitHub To use the development version instead of the PyPI release, create the virtual environment above and replace its installation command with the appropriate command below. Git must be installed.
macOS / Linux
.venv/bin/python -m pip install "git+https://github.com/NandhaKishorM/laya.git"Windows PowerShell
.\.venv\Scripts\python.exe -m pip install "git+https://github.com/NandhaKishorM/laya.git" Run the same version check afterward. The GitHub version follows the repository's default branch and may differ from the published release. Install with uv uv creates the virtual environment, downloads a matching Python if none is installed, and installs into it. The commands are the same on macOS, Linux and Windows PowerShell: uv venv --python 3.12 uv pip install laya uv pip install targets the .venv in the current directory without activating it, so run the version check for your platform above afterward. Extras and the GitHub version install the same way: uv pip install "laya[serve]", uv pip install "git+https://github.com/NandhaKishorM/laya.git". For a CPU-only or GPU-specific PyTorch build, add --torch-backend=auto to pick the build that matches the machine's GPU driver, or name one such as --torch-backend=cpu; uv's PyTorch guide has the list. If your application is a uv project, add Laya as a dependency instead: uv add laya uv run python -I -c "import laya; print(laya._version_)" --torch-backend applies to uv pip only; in a uv project, uv's PyTorch guide shows how to set the PyTorch index in pyproject.toml. Model setup and troubleshooting Continue with the Router quickstart to run inference. Loading a Hub checkpoint requires access to Hugging Face on its first download; the quickstart's Router(preload=True) loads all three configured checkpoints at construction.ModuleNotFoundError: No module named 'laya': run both installation and your script with the same virtual environment's Python executable shown above. In an editor, select that interpreter as well. Missing rl_agent_config.json: this file ships with a Laya checkpoint alongside model.safetensors; it is not a configuration file you need to create in the source repository. For a local model, pass the directory containing those checkpoint files.
Command line Installing the package also installs a laya command for quick local testing, no script needed: laya "I was charged twice, please refund" # routing decision only; works offline, no download laya "Refactor this service" --predict # full answers (downloads the checkpoint on first use) laya "Mein Konto wurde zweimal belastet" --lang de # force a language instead of detecting it laya "My payment failed twice" --model ml # pin a checkpoint: names, aliases and casing all resolve as the SDK resolves them laya "My payment failed twice" --preset triage # answer a ready-made preset (triage, email, guard, moderation, router) laya --batch tickets.txt --predict # score a file of requests, one per line, in one batch cat tickets.txt | laya --batch - --predict --json # stdin; one JSON line of answers per request laya "Where is my card" --questions intents.json # answer your own questions, written in a JSON file laya # interactive mode Routing alone never downloads a checkpoint, so it returns in milliseconds. --predict loads the routed checkpoint, which needs network access to the Hugging Face hub the first time; if a checkpoint cannot be downloaded, the CLI says so instead of crashing. --batch (with or without --predict) sends the whole file through Router.predict_batch in one process, so the requests share checkpoint loads and forward passes — measured 2.6x on 20 tickets vs looping predict one by one, with --batch-size N to bound the forward pass, --sort-by-length to group similarly sized requests inside it, and --json for JSONL output. Batch routing (laya --batch FILE, no --predict) likewise answers with route_batch in one pass, still without loading anything. --sort-by-length reaches the length grouping Agent.predict_batch has done since #294, which until now only the library call in Batch Mode could ask for. A forward pass pads every state in it to the longest one, so a two-line request sharing a pass with a two-paragraph one spends most of its compute on padding; grouping by length first puts comparable sizes together. Measured on 128 real Yelp reviews of 69-2,293 characters, run as laya --batch FILE --predict --model laya --batch-size 8 with and without the flag, four interleaved rounds on an Apple-silicon Mac at the CLI's own device pick: 41.25 s median unsorted against 29.23 s sorted, a paired median of 1.42x (1.39-1.44x across the four pairs), with all 128 decisions identical and still in input order. research/ measures the same knob at 2.15x over 10,000 synthetic multilingual tickets. It needs --batch-size N with 1 < N < the number of requests — a single pass over the whole file has no group to reorder — and the CLI says so on stderr rather than printing a same-speed result and leaving you to notice. --questions takes the same question dict the SDK takes, as JSON: either the mapping itself, or {"state_key": "body", "questions": {...}} when the question's instructions name a field other than request. It implies --predict, and a question set with many labels usually wants --head-max-len with it: on 58 MASSIVE-INTENT labels written as one choice question, the English checkpoint goes from 24/58 correct at its default 192-token option budget to 34/58 at --head-max-len 384, for about 1.4x the per-request time on CPU. Widening it further costs the accuracy back, because max_len then leaves fewer tokens for the request itself. Honest limits describes the same budget ceiling for a 77-option question.
Try it locally: web GUI + JSON API examples/server.py is a self-contained FastAPI app for testing Laya without writing any code: a two-pane playground (edit the request as a form or as JSON, run it with Ctrl+Enter, read each answer's full distribution and calibrated confidence, copy it as curl or Python), plus a plain JSON API (/predict, /predict/batch) for scripting against. pip install "laya[serve]" python examples/server.py # http://127.0.0.1:8000 Open http://127.0.0.1:8000 in a browser for the playground, or hit it directly: curl -s localhost:8000/predict -H 'content-type: application/json' -d '{ "state": {"body": "We were billed twice for March. Please refund it today."}, "questions": { "department": {"type": "choice", "instructions": "Which department should handle this?", "criteria": {"billing": "invoices, payments, refunds", "other": "everything else"}}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["not urgent", "soon", "critical"]} } }' | python -m json.tool --no-preload loads checkpoints lazily instead of all three up front; --device cuda|cpu|mps pins the device. See python examples/server.py --help for the rest. A request body may carry model to pin a checkpoint instead of letting the router choose, and it takes exactly the spellings laya --model takes: names, aliases and casing all resolve through laya.router. GET /models lists the checkpoints and the aliases alongside them. /predict/batch accepts two optional body fields that control the shape of the forward passes without changing any answer: batch_size (states per pass; omit it and the whole batch is one pass) and sort_by_length (group similarly sized states so each pass pads to a shorter maximum — it needs a batch_size below the number of states, since one pass has nothing to reorder). Measured on the running server: 64 real reviews of 91–2,293 characters, one choice question, batch_size: 8, three interleaved rounds on MPS — the default one-pass shape 5.35s median against 3.04s sorted, a paired median of 1.77x (min 1.76x), and 64/64 labels identical in input order. batch_size: 8 on its own was 4.95s, so the length grouping is what earns the second factor: 1.66x over the sized batch. JavaScript / TypeScript laya-client supports JavaScript and TypeScript applications over HTTP to a self-hosted laya-serve /v1/systemone server. It includes inferred answer types, all five question presets, ESM/CommonJS exports, and cancellation.
From this checkout, install and start the Python inference service:
pip install -e '.[serve]' LAYA_HOST=127.0.0.1 LAYA_MODELS=english laya-serveIn another terminal:
cd sdk/typescript npm ci npm run build node examples/triage.mjs The npm package is named laya-client. Use it when a JavaScript or TypeScript application talks over HTTP to self-hosted Python laya-serve. Use laya-ts when inference must run directly inside JavaScript through its local ONNX runtime, without a Python server. See installation, examples, and publishing and the repository analysis for architecture and scope.Quickstart: Route Mode (Recommended) To try the Python SDK in a CPU container, see the Docker Compose quickstart. It runs a sample request and keeps downloaded models between runs. Laya ships three checkpoints. The built-in Router is the recommended entry point: it evaluates any state in any language, automatically detects scripts and languages in sub-milliseconds, and dispatches to the optimal checkpoint in a single forward pass. from laya import Router