DaqulaLin/MemStitch
Zero-Copy Context Bridging Gateway for Multi-Agent GPU Inference. Bypasses the expensive prefill phase by dynamically stitching KV Caches at the memory level using PagedAttention. Cuts TTFT latency...
3 stars · 0 forks · HTML
README
Context-Stitcher 🪡 Zero-Copy Context Bridging Gateway for Multi-Agent GPU Inference.
💡 Core Value Proposition In multi-agent collaborative workflows, separate agents often process the same long text context sequentially. For example:
Agent A (Legal Auditor): Reads a 200-page contract and runs compliance analyses (populating the GPU KV Cache). Agent B (Financial Compliance): Reads the same 200-page contract and audits financial liabilities.
Under standard inference engines, Agent B is forced to repeat the expensive prefill phase, duplicate GPU activations, and suffer from high Time-to-First-Token (TTFT) latency. Context-Stitcher solves this by bridging caches at the memory level:
Context Topological Hashing: Segmenting prompts into physical block-sizes and mapping them to cryptographic fingerprints (Merkle-chains). Zero-Copy Block Stitching: Bypassing prefill for matched prefixes by mapping the logical attention table of Agent B directly to the physical GPU memory address of Agent A's cache blocks. Zero-Trust Secure Gate: Enforcing boundary control lists so unauthorized agent sessions cannot access shared physical blocks.
📊 Performance Profiles Below is the benchmark analysis of Context-Stitcher compared to standard vLLM cold-prefills when executing consecutive agents over a shared 200-page document: ⚡ TTFT Prefill Latency (Agent B Response Time) — Lower is better Baseline (vLLM Cold): ██████████████████████████████ 1200 ms Context-Stitcher: █ 48 ms ( 25.0x Prefill Speedup! 🚀 )
💾 GPU Physical Cache Blocks Allocated (Total VRAM) — Lower is better Baseline (vLLM Cold): ██████████████████████████████ 53 blocks (No sharing) Context-Stitcher: ████████████████ 30 blocks ( 43.4% Memory Saved! 📉 )
⚙️ Installation & Quick Start 1. Install Dependencies pip install -r requirements.txt 2. Launch the Gateway & Dashboard python run.py Once booted, the gateway routes are active on http://localhost:8000:
API Proxy Gateway: http://localhost:8000/v1/chat/completions Real-time Developer Console: Open http://localhost:8000 in your web browser.
🖥️ Visual Developer Portal Context-Stitcher includes a responsive developer portal to monitor physical cache block states (idle, private allocations, shared/stitched pages, security alarms) in real time.
🛠️ Client Integration & Usage Guide Context-Stitcher supports Python SDK Decorators and OpenAI-Compatible REST APIs for cross-application integrations: Pattern A: Python SDK Decorator (For Python-based Agent Orchestration) If your agent pipelines are written in Python, you can utilize the StitcherMesh and @stitch_agent decorators to link context memory: from context_stitcher import StitcherMesh, stitch_agent
1. Initialize the pointer-sharing inference control mesh (ref: vLLM)
mesh = StitcherMesh(backend="vllm", model="meta-llama/Llama-3.1-8B-Instruct")Configure secure gate policy: allow Agent B to read Agent A's KV Cache blocks
mesh.sg.add_policy("agent_a", "agent_b")2. Decorate your agent workflows
@stitch_agent(mesh) def agent_a(): prompt = "[Long Document Context...]\nAnalyze the intellectual property clauses." # First execution: Prefills the cache and registers topological hashes res = mesh.generate(prompt=prompt, fingerprint="legal_doc_v1") return "legal_doc_v1"@stitch_agent(mesh) def agent_b(context_fingerprint): prompt = "[Long Document Context...]\nEvaluate the financial compliance risk." # Subsequent execution: Automatically stitches blocks, bypassing prefill phase res = mesh.generate(prompt=prompt, fingerprint=context_fingerprint) print(f"Agent B Response: {res['generated_text']}") print(f"Time Saved: {res['prefill_time_saved_ms']}ms")
Pattern B: OpenAI-Compatible REST API (For multi-language clients, Dify, or Flowise) The gateway exposes standard OpenAI endpoints. Point your LLM client base URL to Context-Stitcher to activate sharing. 1. Python OpenAI SDK Client: from openai import OpenAI
Direct client to the Context-Stitcher proxy gateway
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")Send standard completions request with agent_id metadata inside extra_body
response = client.chat.completions.create( model="context-stitcher-sim", messages=[ {"role": "user", "content": "[Long Document Context...]\nEvaluate compliance risk."} ], extra_body={ "agent_id": "AgentB", # Identifies the requesting Agent "session_id": "session_legal_audit" # Identifies the shared session cache } )print("Generated Output:", response.choices[0].message.content) 2. Raw HTTP cURL Request: curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "context-stitcher-sim", "messages": [ {"role": "user", "content": "[Long Document Context...]\nEvaluate compliance risk."} ], "agent_id": "AgentB", "session_id": "session_legal_audit" }'
Pattern C: Controlling Security Rules via API (Zero-Trust Gate) You can inspect, add, or revoke access authorization rules dynamically between agents. 1. Retrieve all active policy rules: curl -X GET http://localhost:8000/policies 2. Authorize AgentB to stitch AgentA's KV Cache: curl -X POST http://localhost:8000/policy \ -H "Content-Type: application/json" \ -d '{"owner_agent": "AgentA", "allowed_reader": "AgentB"}' 3. Revoke AgentB's permission to read AgentA's Cache: curl -X DELETE http://localhost:8000/policy \ -H "Content-Type: application/json" \ -d '{"owner_agent": "AgentA", "allowed_reader": "AgentB"}'