datopian/portaljs
🌀 AI-native framework for building data portals. Scaffold a full portal from a brief and load datasets in minutes with agentic skills — any backend (CKAN, GitHub, Frictionless). - datopian/portaljs
2.3k stars · 333 forks · TypeScript Topics: ai, skills, mcp, nextjs, open-data, ckan, dkan, socrata, dataverse, datahub, data-portal, ai-agents, open-data-portal, opendatasoft, data-mesh, openmetadata, agent-skills, claude-code
README
PortalJS The AI-native framework for building data portals. Describe the portal you want — your agent helps you choose an architecture, scaffolds it, and loads your data. Docs · Discussions · Report a bug
Quickstart Create a portal — one command, nothing to install beyond Node 22+: npm create portaljs@latest my-portal cd my-portal npm run dev # → http://localhost:3000 You get the three surfaces — Home, a Catalog (/search), and a dataset Showcase (/@<namespace>/<slug>) — over sample data. Plain, editable Next.js, no lock-in. Add your own CSV/JSON to datasets.json and it renders automatically. Build it with your AI assistant — PortalJS ships Claude Code skills that do the assembly. Install them once (into ~/.claude/commands): curl -fsSL https://raw.githubusercontent.com/datopian/portaljs/main/scripts/install-portaljs-skills.sh | bash Then, in a Claude Code session from any directory: /portaljs-architect not sure what stack you need? start here /portaljs-new-portal "Auckland Council open data portal" /portaljs-add-dataset ./data/air-quality.csv
/portaljs-new-portal scaffolds the three surfaces; /portaljs-add-dataset (or /portaljs-add-resource) loads data; /portaljs-connect-ckan points it at a CKAN backend; /portaljs-deploy ships it. (All skills + install →) Prefer the bare template — plain Next.js, no AI, no lock-in: npx tiged datopian/portaljs/examples/portaljs-catalog my-portal cd my-portal && npm install && npm run dev # → http://localhost:3000 You get Home, a Catalog (/search), and a dataset Showcase (/@<namespace>/<slug>) over sample data. Add your own CSV/JSON to datasets.json and it renders automatically. ⭐ If it's useful, a star helps others find it. Why PortalJS Building a data portal has always meant more than a website. You have to decide where the data lives, how it's versioned, how people search it, how it's served, and how it's governed — and then wire a frontend on top. Teams either over-build on a heavy data warehouse they don't need, or under-build on a pile of scripts that doesn't scale. PortalJS is an open-source, agentic skills framework that helps data teams build, develop, and ship data portals — and the data infrastructure underneath them. It isn't only a frontend. The skills do two jobs:
Advise — given what you're building, what your data is, and what it's for, they recommend an architecture: storage, compute, catalog, access, hosting, metadata. Build — they scaffold that stack as plain, editable Next.js code with no lock-in.
It is opinionated but open: the recommended modern path is git + object storage (Cloudflare R2) + Parquet, queried with DuckDB — an open lakehouse instead of a classic warehouse. For living, incremental tables you can layer on DuckLake, and a traditional datastore (CKAN, a warehouse) stays a first-class option when you need it. You always own plain code. Built and maintained in the open by Datopian and the PortalJS community. Architecture at a glance 🧑 you describe what you want to build │ ▼ ╭─ 🤖 AGENTIC SKILLS ────────────────────────────────── decide + build │ /portaljs-architect · /portaljs-new-portal · /portaljs-add-dataset · /portaljs-add-chart · /portaljs-add-map … ╰─ generates plain, editable Next.js code — no lock-in │ ▼ ╭─ 🖥️ SURFACES ──────────────────────────────────────── what users see │ 🏠 Home / 🔎 Catalog /search 📊 Showcase /@ns/slug ╰─ read data through one DataProvider contract │ ▼ ╭─ 🔌 PROVIDERS ─────────────────────────────────────── pluggable backends │ 📁 static·git 🐘 CKAN 🔭 OpenMetadata 🗂️ git-LFS + R2 ╰─ swap the source without touching a page │ ▼ 📦 STORAGE + COMPUTE — choose your point on the spectrum:
flat files ─▶ Git-LFS + R2 ─▶ Parquet on R2 + 🦆 DuckDB ─▶ warehouse / CKAN simplest ⭐ open lakehouse (default) heaviest (+ DuckLake for living tables)
☁️ Substrate — Cloudflare R2 (storage) · Workers (runtime) · D1 (catalog) · Pages (static) object storage stays S3-compatible — R2 is the default, never a lock-in
Three surfaces. Every data portal is built from three: a Home page that explains it and offers search, a Catalog (/search) to discover datasets, and a Showcase (/@<namespace>/<slug>) to explore one dataset — metadata, preview, download/API, and charts/maps. (Core concepts →) One seam. The surfaces read data only through a DataProvider, so the source — static files today, a CKAN or lakehouse backend tomorrow — can change without touching a page. See ROADMAP.md for the full model and the architecture decision framework for how /portaljs-architect turns your needs into a stack. Build a portal with your AI assistant PortalJS ships Claude Code skills that turn a brief into a working portal. Setup Install the skills once into your personal scope so they're available from any directory: curl -fsSL https://raw.githubusercontent.com/datopian/portaljs/main/scripts/install-portaljs-skills.sh | bash Restart Claude Code (or open a new session) and type / to see them. See .claude/INSTALL.md for other install options (versioned plugin, or running straight from a clone of this repo). Use If you're not sure how to set up your portal, start with the advisor, then build: /portaljs-architect we have ~200 public CSVs, updated quarterly, and must publish DCAT-AP /portaljs-new-portal "Auckland Council open data portal" /portaljs-add-dataset ./data/air-quality.csv /portaljs-add-dataset https://example.com/parks.geojson
The skills are interactive — if your brief is thin, they interview you in short rounds rather than erroring. /portaljs-architect recommends a stack and hands off; /portaljs-new-portal scaffolds the three surfaces; /portaljs-add-dataset appends to the datasets.json manifest and the showcase renders automatically at /@<namespace>/<slug>. Run npm run dev and you have a portal. Prefer to build by hand? The skills are a convenience, not a requirement — scaffold the template directly with the CLI: npm create portaljs@latest my-portal (Or grab the bare template with no prompts: npx tiged datopian/portaljs/examples/portaljs-catalog my-portal.) Available skills
Skill What it does
/portaljs-architect Advisory — turns your needs (data, scale, governance) into a recommended architecture before you build. Start here if you're unsure of the stack.
/portaljs-new-portal Scaffold a new portal (Home + Catalog + Showcase) from a brief — copies the template, substitutes your project name and description, installs deps, verifies the build.
/portaljs-add-dataset Add a CSV, TSV, JSON, or GeoJSON dataset — registers it in the catalog and renders its showcase automatically; large local files are pushed to Cloudflare R2 via Git LFS for you.
/portaljs-add-resource Attach another file (data dictionary, methodology, extra data) to an existing dataset — it becomes multi-resource and the showcase renders a section per file.
/portaljs-add-chart Add a line, bar, area, pie, or scatter chart to a dataset's showcase.
/portaljs-add-map Render a GeoJSON dataset on an interactive map and register it on the home page.
/portaljs-add-geo Auto-ingest a geospatial file (GeoJSON, Shapefile, GeoPackage, KML/KMZ, FlatGeobuf, CSV-with-geometry) on your own machine — no server: normalizes CRS to EPSG:4326, derives a PMTiles render tier and a GeoParquet query tier, pushes both plus the original to R2, and registers one dual-tier dataset the showcase maps and queries in place.
/portaljs-define-schema Infer a Frictionless Table Schema from a dataset's data, add license/source/keyword metadata, and surface a typed field table on its showcase.
/portaljs-add-dcat Make the portal harvestable — emit standards-compliant DCAT feeds (DCAT 2/3, DCAT-AP, DCAT-US, national profiles) in JSON-LD, Turtle, and RDF/XML so national/EU/US open-data portals can harvest its datasets.
/portaljs-connect-ckan Wire the portal to a CKAN backend over its API instead of static files.
/portaljs-check-data-quality Validate a dataset against its schema and flag quality issues (type mismatches, missing values, constraint violations).
/portaljs-migrate Harvest or migrate a whole catalog into the portal from CKAN, Socrata, OpenDataSoft, ArcGIS, or DCAT-US, over a canonical Frictionless/DCAT model.
/arcgis-to-portaljs Migrate a whole ArcGIS Hub site into the portal end-to-end — harvest its /data.json, export every FeatureService layer via the ArcGIS REST API, convert to the serverless dual tier (PMTiles + GeoParquet), push to R2, and write a source-vs-derived parity report.
/portaljs-deploy Build a static export and publish it to PortalJS Arc — Datopian-managed hosting on Cloudflare — with a live <slug>.arc.portaljs.com URL.
Large-data scaling — big files pushed to Cloudflare R2 via Git LFS — already ships in /portaljs-add-dataset. More skill families — metadata schemas (Frictionless/DCAT), more backends (OpenMetadata), a browser DuckDB query layer, and access control — are on the roadmap. Write your own — see .claude/AUTHORING.md. What's in this repo .claude/commands/ the agentic skills (slash commands) examples/ reference portals — portaljs-catalog is the canonical template packages/ core/ layout/UI components (@portaljs/core) ckan/ CKAN catalog UI + React (@portaljs/ckan) ckan-api-client-js/ pure CKAN API client (@portaljs/ckan-api-client-js) site/ portaljs.com — the marketing site + docs ROADMAP.md direction, the four contracts, sequencing
The canonical template, examples/portaljs-catalog, is where the three surfaces and the DataProvider seam live — read it before building. What makes it different
🌱 Open source, MIT, no lock-in — every skill emits plain Next.js you can fork and own. 🧭 Advisory, not just generative — /portaljs-architect helps you decide the infrastructure, not only scaffold a UI. 🦆 Open lakehouse by default — git + R2 + Parquet queried with DuckDB, over a heavy warehouse; add DuckLake for living/incremental tables. A datastore/warehouse stays a supported choice. ☁️ Cloudflare-first, portable — R2 / Workers / D1 / Pages as the default substrate, but object storage stays S3-compatible. 🧩 Decoupled, any backend — one DataProvider contract in front of CKAN, DKAN, OpenMetadata, DataHub, GitHub, Frictionless, plain files — or your own. 🎨 Bring your own stack — adopt the template or lift the skills and the three-surface model into an app you already have.
Examples Reference implementations live in examples/:
Example Backend
portaljs-catalog Canonical template — Home + Catalog + Showcase over a static manifest
portaljs-template Minimal single-page starter
ckan · ckan-ssg CKAN
github-backed-catalog GitHub
dataset-frictionless Frictionless Data Package
fivethirtyeight · openspending · turing Real-world portals
Community & support
💬 Discord — live chat and help: join the server 🗣️ Discussions — questions, ideas, show-and-tell: github.com/datopian/portaljs/discussions 🐛 Issues — bugs and feature requests: open an issue 📖 Docs — portaljs.com/docs
Contributing PortalJS is built in the open and we welcome contributions of all sizes — new skills, examples, docs, and fixes. See CONTRIBUTING.md to get started, and read ROADMAP.md and VISION.md for where the project is headed. License MIT © Datopian