ChrystianSchutz/ThreadShelf

Local-first archive, semantic search, and continuation for AI conversations across ChatGPT, Claude, Gemini, Grok, OpenRouter, LM Studio and Google/Gemini Ai Studio - ChrystianSchutz/ThreadShelf

12 stars · 4 forks

README

ThreadShelf Local-first archive, semantic search, and continuation for your AI conversations.

One private workspace across ChatGPT, Claude, Google AI Studio, OpenRouter, LM Studio, and Grok. Search old conversations by meaning, reopen the complete thread, and continue it with a local GGUF model through llama.cpp or the explicitly external OpenRouter provider. The archive pipeline—parsing, embeddings, LanceDB storage, search, HTTP API, and MCP—runs locally. Conversation generation is optional: llama.cpp stays loopback-only; switching to the clearly marked OpenRouter · external provider sends the selected user/assistant context and new prompt off-device.

Capability What ThreadShelf provides

Archive & retrieval Multi-provider normalization, local multilingual embeddings, semantic and exact search, complete thread reconstruction

Continue & create Local GGUF inference through managed llama.cpp, plus optional OpenRouter streaming

Use it anywhere React UI, command-line ingest/search, HTTP API, and an MCP stdio server over the same index

Keep control Loopback defaults, isolated local storage, explicit off-device labeling, synthetic test data

What it does (in 30 seconds)

You export/copy your chats as JSON (see Get your data). ThreadShelf parses them into a common format, embeds them locally, and stores them in a local vector database (LanceDB). You search by meaning in the web UI, open the full original thread, and export any conversation to Markdown — or query the same index from an MCP client. Optionally start a new chat or continue an archived thread through local llama.cpp or explicitly external OpenRouter.

Architecture at a glance

flowchart LR exports["AI chat exports"] --> parser["Provider parsers"] parser --> turns["Normalized turns"] turns --> chunks["Chunks"] chunks --> embed["Local embeddings"] embed --> db[("LanceDB + thread store")] db --> ui["React UI"] db --> api["HTTP API"] db --> mcp["MCP stdio"] db --> generation["Generation registry"] generation --> llama["llama.cpp<br/>local GGUF"] generation -. "explicit off-device provider" .-> openrouter["OpenRouter"]

Loading

How is this different from other history viewers?

One normalized archive across six providers. It includes sources whose history is otherwise split between exports, Drive files, browser pages, and local application data. Semantic, not just keyword. Find a conversation by topic even when you don't remember the exact words — plus an exact-match mode for when you do (error strings, identifiers, code). A local archive with an optional generation layer. Search remains useful without configuring any LLM. When generation is wanted, the primary engine is a loopback-only llama.cpp server; OpenRouter is a separately marked external choice. One index, three front-ends. The same archive is searchable from the web UI, the HTTP API, and any MCP-capable tool.

Supported sources

Source How export works Format stability

Google AI Studio Download your Drive "Google AI Studio" folder ⚠️ undocumented, tested July 2026

OpenRouter Browser-console export script (this repo) ⚠️ undocumented

LM Studio Copy local conversation files ⚠️ undocumented, tested on 0.4.x

Grok / xAI Account data export (prod-grok-backend.json) ⚠️ undocumented

ChatGPT / OpenAI Official data export (conversations.json) semi-stable

Claude / Anthropic Official data export semi-stable

⚠️ Format note. Google AI Studio, OpenRouter, LM Studio, and Grok have no documented, stable export schema — their vendors can change it without warning. ThreadShelf is tested against snapshots of specific versions (e.g. LM Studio 0.4.x). If a newer app version changes the shape, parsing may break; please open an issue with an anonymized sample.

Get your data You only need the providers you actually use. Put exported files anywhere, then point ThreadShelf at that folder. Google AI Studio Your prompts are saved to Google Drive in a folder named Google AI Studio.

Open Google Drive, find the Google AI Studio folder. Right-click → Download (Drive zips it). Unzip somewhere local. Index that folder. (Files have no .json extension — that's fine, they're detected by content.)

Text chats and Imagen prompt histories from the July 2026 Drive export shape are supported. The format is undocumented and may change. OpenRouter OpenRouter has no bulk export, so this repo ships two browser-console scripts:

All chats: scripts/openrouter-export-all.js — walks every chat in your sidebar and downloads one JSON per chat. Single chat: scripts/openrouter-export-browser.js — exports just the chat currently open.

To export everything:

Open openrouter.ai signed in, with your chat list (sidebar) visible. Open DevTools → Console (F12 → Console tab). Copy the entire contents of scripts/openrouter-export-all.js, paste into the console, press Enter. The script clicks each chat, scrolls to load full history, and downloads one JSON per chat. Allow "multiple downloads" if the browser asks. Move the downloaded files into a folder and index that folder.

The selectors are pinned to OpenRouter's current chat UI and covered by test/playwright/openrouter-export.spec.js. If OpenRouter changes its markup and the script finds no chats, that test is where to update the contract.

You'll get a file shaped like this (this is what the parser reads): { "platform": "openrouter", "exportedAt": "2026-05-24T12:00:00.000Z", "pageTitle": "OpenRouter chat title", "sourceUrl": "https://openrouter.ai/chat/...", "turns": [ { "role": "user", "content": "User message text" }, { "role": "assistant", "content": "Assistant reply", "model": "optional/model-name" } ] } Verify a download before indexing: npm run parse -- path/to/openrouter-export.json See docs/OPENROUTER.md for details and limitations. LM Studio LM Studio stores one JSON file per conversation locally — no export step needed, just point ThreadShelf at the folder (paths below):

OS Path

Windows %USERPROFILE%\.lmstudio\conversations\

macOS / Linux ~/.lmstudio/conversations/

Older builds ~/.cache/lm-studio/conversations/ (check here too)

Paste that path straight into the manual folder path box and index it — or copy it somewhere first:

macOS / Linux

Windows (PowerShell)

Note — LM Studio's local server ("router"). LM Studio can also run an OpenAI-compatible local server (Developer tab, default http://localhost:1234) that routes requests across loaded models. That's an inference endpoint, not an export — ThreadShelf indexes the on-disk conversations/ files above, so you don't need the server running to import or search your history.

ChatGPT / OpenAI ChatGPT → Settings → Data controls → Export data. You'll get an email with a zip; index the conversations.json inside it. Claude / Anthropic Claude → Settings → export your data. Index the exported conversation JSON. Grok / xAI Grok → request your account data export (xAI account/privacy settings). You'll get a download containing a prod-grok-backend.json somewhere under an export_data/.../ folder. Index the folder that contains it — ThreadShelf detects the file by content, so the surrounding directory names don't matter. The file holds every conversation in one document, shaped roughly like this (the parser reads conversations[].responses[].response, splitting the model's agent_thinking_traces reasoning from its message answer): { "conversations": [ { "conversation": { "id": "…", "title": "My chat", "create_time": { "$date": { "$numberLong": "1771000000000" } } }, "responses": [ { "response": { "sender": "human", "message": "Hi", "create_time": { "$date": { "$numberLong": "1771000000000" } } } }, { "response": { "sender": "assistant", "message": "Hello!", "model": "grok-3", "agent_thinking_traces": [{ "thinking_trace": "reasoning…" }] } } ] } ] } Timestamps are MongoDB extended JSON ({ "$date": { "$numberLong": … } }). The format is undocumented and may change.

Quick Start Requires Node.js 20.19+ and npm. npx threadshelf # downloads, starts, opens on http://localhost:3000 That is the whole install. The package ships the prebuilt web UI, so there is nothing to compile and no repository to clone. Pick another port with npx threadshelf 3001, and see npx threadshelf --help for the rest. The terminal tools come with it: npx threadshelf search "what did I decide about caching?" npx threadshelf ingest ./my-exports my-collection npx threadshelf parse ./chatgpt-export.json npx threadshelf-mcp # stdio MCP server Your archive is never stored inside the npm package: an npx install directory is disposable and npm may wipe it at any time. Persistent data lives in a per-user directory instead, so upgrading or clearing the npm cache leaves your index untouched.

Platform Default data directory

Windows %LOCALAPPDATA%\ThreadShelf

macOS / Linux ~/.threadshelf

Override it with --data-dir <path> or THREADSHELF_DATA_DIR, and print the resolved locations with npx threadshelf --where. The narrower overrides (LANCEDB_PATH, UPLOADS_DIR, …) still win over the defaults. Run from a clone instead Developing on ThreadShelf, or just prefer the source? npm install # installs server + client (npm workspaces) npm run build:client # builds the React UI into public/ npm start # serves on http://localhost:3000 A repository checkout keeps its data in the repo (.lancedb/, .threadshelf/, .collections.json) exactly as before, so a clone and an npx install never fight over the same files. Use ThreadShelf from an MCP client ThreadShelf includes a local stdio MCP server over the same archive used by the web UI and HTTP API. npm run mcp It exposes five tools for listing indexed data, inspecting statistics, semantic or exact search, and retrieving complete conversations. See MCP setup for clean-clone installation, client configuration, and security notes. Then in the browser:

Create or select a collection (think of it as a folder/project, e.g. ai_studio, chatgpt, work_2026). Start with a throwaway one. Select a folder of exports, or paste an absolute folder path. Click Index folder and watch live progress. Search in natural language; use role filters (user / thinking / response). Switch Semantic → Exact for case-insensitive substring matches — best for identifiers, error messages, and code fragments the embedding model blurs. Click a result to open the full source thread; copy or export to Markdown or JSON.

Use Stop indexing to cancel an active run. The current embedding batch is allowed to finish safely; completed files stay indexed and an uncommitted file keeps its previous index rows.

First indexing is slow while the local embedding model downloads once, then it's cached. Working on the UI? npm run dev (backend) + npm run dev:client (Vite hot reload on :5173, API proxied to :3000).

For scheduled local backups or headless indexing: npm run ingest -- path/to/exports work_2026 Omit the collection to use chunks. Add --clear only when you want to wipe the collection before indexing. To keep a folder indexed as it changes (LM Studio rewrites its conversation files as you chat; AI Studio folders grow), add --watch: npm run ingest -- ~/.lmstudio/conversations lmstudio -- --watch After the initial pass the process stays running, watches the folder recursively, and re-indexes just the files that changed (debounced; tune with --debounce <ms>). Deleting a source file never removes it from the index — ThreadShelf is an archive, and indexed conversations outlive their files. You can also search straight from the terminal: npm run search -- "that regex for polish postal codes" npm run search -- "ECONNREFUSED 127.0.0.1" -- --mode keyword --n 5 npm run search -- "prompt injection" -- --collection work_2026 --json The second -- before CLI options is intentional: current npm versions can otherwise consume option names that appear after positional arguments. Full walkthrough: docs/GETTING_STARTED.md. Browse complete conversations

Click any result to open the full source thread, jump to the matched turn, filter user/reasoning/response roles, copy text, or export the conversation to Markdown or JSON. With no query, the search page lists all indexed conversations with provider badges — sort them by recency, length, or title, and narrow the list with the filter box.

More like this — one click on any search result or thread turn runs a semantic search seeded with that passage, for "I know I discussed this somewhere else" moments. Saved searches & pins — star a query (with its filters and mode) to rerun it later, and pin conversations to keep them at the top of the browse list. Both are stored in your browser's localStorage; nothing leaves the machine.

Conversation generation: llama.cpp + OpenRouter

Generation is optional: archive indexing and search never depend on it, and original export files are never modified.

Use New chat to start a locally saved ThreadShelf conversation, or open an imported thread and choose Continue this conversation:

llama.cpp · local — the primary engine. ThreadShelf discovers GGUF files, launches a loopback-only llama-server, streams tokens, reports CPU/GPU/hybrid placement, and can eject the model from memory without deleting it. OpenRouter · external — an optional provider with a live model catalog. Selecting this tab is the off-device choice: the model button carries an off-device chip and the composer says that the request is sent through OpenRouter. Archived thinking turns are excluded. Optional ZDR-only and data-collection-denial routing can reduce eligible providers.

Completed chats and archive continuations are stored locally, embedded, and searchable. A ghost-icon Private conversation is tab-scoped and is never written to the thread store or semantic index. Failed or stopped streams remain in a clearly marked, unsaved recovery card for the current tab. The model menu discovers common LM Studio, llama.cpp, and Hugging Face model directories plus custom roots. It separates model selection from context/output settings, supports favorites, and exposes detailed runtime logs only on demand. Active streams hold a model lease so concurrent eject or configuration changes cannot unload a model mid-response. Use your local models from other apps ThreadShelf also serves its GGUF models over the OpenAI- and Anthropic-compatible API that LM Studio and Ollama use. Point any SDK or tool at http://localhost:3000/v1, use any API key, and pass a model id from GET /v1/models: curl http://localhost:3000/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "Bielik-11B-v3.0-Instruct.Q8_0", "messages": [{"role": "user", "content": "Cześć!"}]}' The endpoints are /v1/chat/completions, /v1/completions, /v1/responses and /v1/messages, with streaming. Requests are not saved to the archive.

Models: the model named in a request loads on first use and replaces the one in memory, whose VRAM is freed first. GET /v1/models marks the loaded one with "loaded": true. Memory: a model stays loaded until another is requested. Free it with POST /v1/models/unload or Eject, or let ThreadShelf unload it after an idle time (Unload model from memory, default: never). Access: like LM Studio and Ollama, the API is open to programs on this computer with no key. Optional settings add an API key (required on every request once set), a network port that serves only /v1 to other devices, and an off switch.

Settings → Conversation generation → Local model API shows the URL, your model ids, copyable examples and those settings. Its Guide explains what is on right now, every setting in plain words, and how to connect Claude Code, Codex CLI or DeepSeek Harness. Details: Local model API; speeds and tuning: Performance. Guided setup: runtime and model in one confirmation Settings → Conversation generation → Set up local generation resolves one plan for the whole first run: the official llama.cpp build for this machine, plus a GGUF model sized for the detected accelerator. The plan is shown before anything is fetched — every URL, SHA-256 digest, size, and destination — and one confirmation runs it. Nothing downloads until you confirm, and Cancel stops an in-flight transfer immediately. The plan is re-resolved on the server when you confirm; a client can never hand the server a URL to fetch and execute. If the resolved plan no longer matches the one you approved — a nightly build moved, the catalog changed, free VRAM shifted the recommendation — the run stops and shows the new plan for a fresh confirmation instead of downloading something you never agreed to. Browse and download models from Hugging Face Settings → Conversation generation → Download a model… — or the same action in the chat's model menu — opens a read-only browser over the public Hugging Face API. It needs no account or token: search or sort by downloads, likes, trending, or recency, inspect a repository's quantizations, and see a per-quantization verdict for your hardware — fits inside the accelerator budget, tight (partly on CPU), or too large. Builds from known publishers (unsloth, lmstudio-community, bartowski, ggml-org, Qwen, google, mistralai) are marked. Downloads land in .threadshelf/models (override with THREADSHELF_MODELS_PATH), which is gitignored and always searched for models, so a downloaded model appears in the model menu without further configuration and is selected automatically when the download finishes. Every file is verified against the repository's LFS oid (its SHA-256) before it is moved into place. A cancelled transfer keeps its .part file and the next attempt resumes from there; any other failure deletes it. Gated repositories are detected up front and marked in the UI. To use one, accept its licence on Hugging Face and set HF_TOKEN in the gitignored root .env. A configured token is sent with every Hugging Face request; without one, public repositories still work. No conversation content is ever sent to Hugging Face — only catalog metadata requests and file downloads. Find or install llama.cpp safely The guided setup above covers the common case. The CLI installer remains the scripted, offline, and custom-build path, and stays the only way to install an archive ThreadShelf did not resolve itself. The setup command is cross-platform (Windows x64/ARM64, macOS x64/Apple Silicon, Linux x64/ARM64 where official release assets exist). With no arguments it only looks for llama-server; it does not make a network request, download, in

source