gatewai-dev/gitframes

Code-first video, rendered natively on WebGPU. Contribute to gatewai-dev/gitframes development by creating an account on GitHub.

3 stars · 1 forks

README

Photoshop-, After Effects-, and Blender-class video tools as one npm package that AI agents drive with code.

⚠️ Beta: gitframes is under active development. APIs may change between releases and some features may be incomplete or unstable.

Gitframes is built for coding agents. It packs the work people usually split across three desktop apps (Photoshop-grade compositing and VFX, After Effects-style motion, typography and keyframing, and Blender-style 3D scenes, cameras and models) into one lightweight npm package. Your agent writes a TypeScript composition, checks frames, and renders an MP4, and nobody has to install or license a multi-gigabyte creative suite. Code-first video as pure software engineering — no headless browser, no DOM reflow, no screenshot pipeline. Renders directly on GPU hardware via Dawn / WebGPU / Metal / Vulkan in Node.js and modern WebGPU browsers.

Made with gitframes Every frame of these films is rendered by gitframes from TypeScript in examples/. Click a still to watch it on YouTube.

Gitframes launch 22_gitframes_launch Full Circle 21_full_circle Dancer showcase 19_gitframes_film

NoteUsing an AI coding agent? Install the gitframes skills in one line. Claude Code /plugin install gitframes

Any other agent (Codex, Cursor, Hermes, Gemini CLI, Copilot, and more) npx skills add gatewai-dev/gitframes See Agent Skills & Plugins for details.

Table of Contents

Why Gitframes Architectural Comparison Key Features & Capabilities

1. Slug GPU Vector Typography & AE Parity 2. Photoshop-Grade WebGPU 2D VFX 3. Unified 3D Scene Graph, Camera & Mesh Shading 4. Audio Layers, Procedural SFX & Reactive Signals 5. Animated Charts 6. On-Device Vision & Tracking 7. Headless Conformance & FrameGrid Testing 8. Live Preview in the Browser

Monorepo Architecture Quickstart Guide

1. Basic Composition & Kinetic Auto-Layout 2. Unified 3D Scene with Camera & 3D Model 3. Audio Soundtrack, Procedural SFX & Reactive Signals 4. Chained WebGPU Post-Processing VFX 5. Vision: Pin, Matte & Reframe 6. Headless Video & FrameGrid Rendering

Engineering Doctrines & Best Practices Agent Skills & Plugins Reference Showcase Examples Development & Building Community License

Why Gitframes Modern automated video generation is usually constrained by the architectures of general-purpose web browsers: process overhead, non-deterministic DOM layout reflows, and slow screenshot capture. Gitframes treats video composition as software engineering:

Pillar What it means

🚀 Zero Headless-Browser Overhead No Puppeteer, no Chromium IPC, no page.screenshot(). Gitframes talks straight to native GPU devices via Dawn/WebGPU and hardware-encodes with @napi-rs/webcodecs.

🎯 Deterministic Frame-Accurate Clock Absolute frame clocks, discrete sample points, and frame-accurate audio BeatGrids. No floating timers, no drift, no dropped frames.

🔠 Analytic, Resolution-Independent Type The Slug algorithm evaluates glyph contours per-pixel in WGSL — no texture atlases, no scaling artifacts, razor-sharp from 10 px to 10,000 px.

🎨 Photoshop-Grade Tonal & Spatial VFX 50+ modular GPU shaders: Curves, Levels, Selective Color, 3D LUTs, Halftone, Film Grain, Unsharp Mask, Mesh Warp, and Screen-Space Relighting.

🧊 Unified 3D & 2D Depth Compositing Nest 2D flex/box trees inside 3D homography planes, multiplane rigs, and meshes (OBJ, FBX, glTF/GLB, STL, PLY, VOX, 3DS, OFF), with PBR glass and SSAO.

🔊 Built-in Procedural Audio DSP Multi-track soundtracks, deterministic procedural transition SFX (whoosh, impact, riser), and reactive signals that drive visuals from audio.

👁️ On-Device Neural Vision Object tracking, instance segmentation, multi-person pose, and person mattes from Apache-2.0 ONNX models — feeding reactive signals without a round trip to disk.

☁️ Cloud-Native & CI/CD Ready ~200–400 MB RAM per worker (vs. 2–4 GB for Chromium), ideal for serverless GPU render clusters (AWS G4/G5, Modal, RunPod, Kubernetes).

Architectural Comparison: Gitframes vs. Remotion vs. Hyperframes Developers generating video programmatically commonly weigh Remotion (React/Chromium) or Hyperframes (Canvas2D/SVG web animation). The matrix below compares the fundamental engineering dimensions. Detailed Comparison Matrix

Capability / Dimension Gitframes Remotion Hyperframes

Underlying Engine Native WebGPU (WGSL compute & render pipelines via Dawn / Metal / Vulkan) Chromium / Puppeteer (React DOM, HTML/CSS layout) Canvas2D / WebGL / SVG (browser or Node Skia)

Rendering Architecture Direct hardware framebuffer rendering & hardware video encoding (@napi-rs/webcodecs) Spawns headless Chrome; captures frames via CDP / page.screenshot() Software or hardware 2D canvas context

Throughput 60–120+ FPS (real-time to faster-than-real-time GPU execution) 5–20 FPS (DOM reflow, IPC, rasterization) 20–40 FPS (CPU draw commands / JS)

Memory Footprint ~200–400 MB per render (zero browser) 1.5–4.0 GB+ per worker (Chromium + V8 DOM heap) ~500 MB–1 GB (Skia/Canvas bindings)

Typography Engine Slug GPU — analytic Bézier evaluation in WGSL, infinite zoom, After Effects selectors Browser DOM text (CSS fonts, rasterized, blurry under 3D transforms) Canvas2D / path text (CPU-rasterized glyphs)

2D VFX & Post-Processing 50+ WebGPU shaders (Curves, Levels, Selective Color, 3D LUT, Film Grain, Halftone, Liquify, PBR Glass, Relight) CSS Filters or custom WebGL canvas wrappers Basic Canvas2D composites and 2D filters

3D Graphics & Depth Native 3D scene graph — LookAt/Turntable camera, multiplane, skinning (OBJ/FBX/glTF), SSAO, PCSS, DoF None built-in (embed Three.js/Fiber inside React DOM) Minimal 2.5D layers; no unified mesh pipeline

Motion Blur & Physics Physical 180° shutter velocity buffers in MRT + closed-form spring kinematics CSS transitions / JS interpolation; synthetic blur hacks Frame interpolation or manual multipass

Audio Engine & DSP Native audio DSP & procedural SFX (multi-track mixing, beat grids, reactive signals) <Audio> playback; basic volume curves Basic static audio playback

Charts & Data Viz Layer.chart — line, area, bar, scatter, candlestick, pie and donut charts built from native vector nodes, with staggered reveal animations DOM chart libraries (Recharts, Chart.js) Custom canvas draw operations

AI & Computer Vision On-device ONNX vision — COCO-80 detection + instance masks (RTMDet-Ins), COCO-17 pose (RTMO), person mattes (Selfie Segmenter); WebGPU tensor conditioning (Canny, depth-to-normals, optical flow, deflicker) External pre-rendered assets; no native GPU tensor conditioning External pre-rendered assets

Headless Verification FrameGrid contact sheets, single-frame snapshots, Skia MSE pixel-invariant assertions Playwright/Puppeteer visual snapshots Manual frame inspection / canvas diffing

Docker / Cloud Portability Compact (~500 MB slim image with native GPU/Vulkan drivers) Heavy (~2–3 GB with Chromium, fonts, X11/Mesa) Moderate container size

Key Features & Capabilities 1. Slug GPU Vector Typography & AE Parity Traditional text relies on CPU rasterization or low-res SDF atlases that soften under 3D camera sweeps. Gitframes integrates the Slug algorithm (SlugPipeline):

Analytic GPU evaluation — WGSL fragment shaders solve exact cubic/quadratic Béziers per-pixel. Glyphs stay sharp at 10 px or 10,000 px with zero CPU re-rasterization. After Effects–parity animators — range selectors (square, ramp_up, ramp_down, triangle, smooth), easeHigh/easeLow curves, and seeded PRNG character shuffling (TextAnimator). Human typing cadence — weighted punctuation delays (commas 3×, sentence ends 5.5×, newlines 7×) and trailing scramble resolution (TypewriterAnimator). 3D volumetric formations — map text onto cylindrical drums, logarithmic vortex spirals, and double-helix ribbons with surface-normal banking (evaluateVolumetricFormation). Dynamic leading & skew — area-preserving unimodular shear and accordion line-leading anchored to baseline, center, or top.

2. Photoshop-Grade WebGPU 2D VFX (50+ Shaders) A comprehensive suite of professional image/video shader nodes in nodes/ and packages/webgpu-renderers:

Tonal grading — Curves (RGB/R/G/B spline), Levels (black/white point, gamma, output), Shadows/Highlights, Selective Color (CMYK gamut isolation), 3D Cube LUT (ApplyLUT). Stylization & grain — Film Grain (Gaussian emulsion with spatial seed variation), Halftone (mono/RGB/CMYK, adjustable dot shape & angle), Gradient Map, High Pass. Optics & lens — Bilateral Gaussian Blur, Unsharp Mask, Vignette, Refraction Caustics, PBR Glassmorphism with chromatic dispersion (PBRGlass). Distortion & warping — Displacement Maps, Liquify, Mesh Warp, Corner Pin homography.

3. Unified 3D Scene Graph, Camera & Mesh Shading

Calibrated camera rig — LookAt and Turntable cameras (Camera3D) calibrated so z = 0 matches 2D canvas pixel coordinates 1:1. 3D layout primitives — Layer3D.cube, carousel, prism, plane, grid with unified depth-buffer testing. Zero-dependency model parsers — OBJ, FBX, glTF/GLB, STL, PLY, VOX, 3DS, OFF. Skeletal animation & shading — 128-bone Linear Blend Skinning, Blinn-Phong & PBR multi-light shading, PCSS/Poisson contact shadows, SSAO, and optical DoF. Physical motion blur — 180° shutter motion blur with per-vertex velocity vectors packed into rg16float MRT buffers.

4. Audio Layers, Procedural SFX & Reactive Signals

Soundtrack layers — .audio media nodes with frame-exact lifecycle control. Procedural SFX — deterministic CPU-synthesized whooshes, impacts, risers, downshifters, and glitches placed on the bar/beat grid (renderSfx, mixSfxInto, softLimit). Multi-track mixing — master tracks headlessly with mixAudioTracks and encodeStereoWav. Reactive signals — drive transforms, scale, borders, or shader uniforms from tempo signals (Signal.builder) or audio analysis.

5. Animated Charts Layer.chart builds line, area, bar (grouped or stacked), scatter, candlestick, pie and donut charts. d3 computes the scales, ticks and geometry; every bar, line, slice and label is an ordinary box, path or text node:

Labels use the composition's registered fonts and the same GPU text renderer as the rest of the film. A built-in reveal draws lines on, grows bars from the baseline and staggers points and slices (animate: { start, duration, stagger, ease }, or animate: false). The chart is one box, so it positions, animates, grades and tilts into 3D like any other layer.

Layer.chart( { type: "bar", width: 900, height: 480, categories: ["Q1", "Q2", "Q3", "Q4"], series: [ { name: "Revenue", data: [12, 19, 24, 31] }, { name: "Costs", data: [8, 11, 13, 15] }, ], yAxis: { format: "$,.0f" }, animate: { start: 10, duration: 30 }, }, { position: "absolute", x: 120, y: 200 }, ); 6. On-Device Vision & Tracking @gitframes/vision runs ONNX models via onnxruntime-node (CPU) or onnxruntime-web (WebGPU) and wires every result into the same reactive signal surface the rest of Gitframes consumes. TipLazy by construction. VisionRunner.create(), comp.withVision(...) and VisionNode.attach(...) perform zero I/O — no downloads, no sessions, no file probes. A model is fetched the first time a task actually runs. To warm up ahead of time, call await runner.preload(["detect", "pose"]) (or await vision.ready() on an attached node).

Tasks and models Every model is Apache-2.0, pinned to an immutable Hugging Face revision, and verified by SHA-256 after download.

Task Option Model Output

Detect enableDetection RTMDet-Ins t/s/m (OpenMMLab) COCO-80 boxes + scores, tracked over time

Segment enableSegmentation RTMDet-Ins (same forward pass as detect) Soft per-instance masks, frame-aligned

Pose enablePose RTMO t/s/m (OpenMMLab) 17 COCO keypoints + visibility per person

Matte enableMatte MediaPipe Selfie Segmenter (Google) Fast person-vs-background alpha for portrait / webcam framing

Variants — variant: "t" | "s" | "m" (default "s"; ~24 / 43 / 116 MB for RTMDet-Ins). Tune confidence and a COCO classes filter per composition. On CPU, a 2K frame takes roughly 200–340 ms to detect + segment, ~120 ms for pose and ~20 ms for the matte with "s". One pass, two tasks — detection and segmentation share a single RTMDet-Ins inference per frame. Picking a matte — the Selfie Segmenter is tuned for a person filling much of the frame: it misses distant figures and can report "person" on close-ups with nobody in them. For anything else, cut out with instance masks (matteSource: "instance", the default). Whole-subject cutouts — mask / matte / crop modes merge every comparably sized instance that overlaps the main subject, so a flowing dress or a held instrument stays attached to the person, while a tunnel or window framing them does not. One-frame delay — vision reads each layer's previous rendered frame, so results trail the plate by one frame and frame 0 has none. Verify vision layers with the exported video or consecutive frames, not frame grids. Model cache & mirrors — models are cached atomically (temp + rename) in $GITFRAMES_MODELS_DIR (default ~/.cache/gitframes/models). Point baseUrl or GITFRAMES_MODELS_BASE_URL at your own mirror for air-gapped or CI renders.

GPU helpers

OpenPose-style skeleton textures — rasterize COCO-17 keypoints into a VRAM conditioning texture (PoseSkeletonRenderer). GPU segmentation texture pool — reusable silhouette textures (SegmentationTexturePool).

Temporal tracking & analysis

Multi-object tracker (TemporalObjectTracker) assigns stable trackIds via IoU association, with configurable minHits, positionSmoothing, and velocity-based coasting for up to maxMissedFrames (default 15) so a transient miss holds the track instead of flashing. Pose↔track matching (pose-track-matcher) binds keypoints to the right track by id, then by spatial IoU fallback. One-shot sequence analysis — comp.analyzeVisionSequence(src, { tasks, categories }) decodes frames through the mediabunny pipeline, tracks them, and returns a zod-serializable report (per-track frame ranges, mean speed, sampled center paths, per-class presence/confidence, mean mask coverage, model download bytes/timing) (analyzeSequence).

Reactive vision signals Every tracked entity is exposed as reactive ProgrammaticSignals that animate layers and shader uniforms:

Group Highlights

objects get(trackId), byCategory(cat, rank), primary, count, hasCategory, detectedCategories

objects.*.bounds x/y/width/height, screenX/screenY/screenWidth/screenHeight, aspectRatio, area

objects.*.anchors 9 anchors (corners, edges, center) ready for pinning

objects.*.kinematics vx, vy, speed, acceleration, headingRad/Deg

objects.*.pose All 17 COCO keypoints, plus hasPose, wristSpeed, handRaised, bodyTiltAngle

masks get(trackId), subject, count; per-mask area, coverage, solidity, bboxFill

segmentation subject, humanSilhouette, instanceMasks, matte.coverage, GPU stencilTexture

classes Per-class count, maxConfidence, present, primary, plus a detection histogram

Tensors poseLandmarksTensor [17,3], objectsTensor [16,8], masksTensor [16,2], histogramTensor [80]

Spatial pinning Project normalized landmarks to screen space with a configurable camera FOV, then bind any node to a track or landmark (SpatialLandmarkTransformer, spatial-pin):

pinToObject(track, { anchor, offsetX/Y/Z, matchWidth, matchHeight, smoothFrames, hideWhenLost }) pinToLandmark(coord, { offsetX/Y/Z })

High-level composition helpers

Subject Sandwich — comp.addSubjectSandwich({ source, behind, feather, fit }) cuts the foreground subject out and places typography/graphics behind them. Smart Reframing — comp.addSmartFraming({ source, target, targetAspect, damping, leadHeadroom }) auto-crops 16:9 → 9:16 while tracking target. Subject Outline — comp.addSubjectOutline(vision.segmentation.subject, { source, color, width, blur }) strokes the segmented boundary as an audio-reactive contour glow. Tracked Region Blur — layer.blurRegion(track, { strength }) blurs faces, plates, or any detected class. Node modes — passthrough, mask, matte, crop, skeleton, boxes, tracking; pick the cutout alpha with matteSource: "instance" | "selfie", and optionally keyBackground to grow the subject into connected foreground.

Agent-first DX

Runtime config is zod-validated and available from a zod-only entry (@gitframes/vision/schemas) so the hot path stays zod-free. Unknown or removed options are rejected, not silently ignored. vision.summary(frame) returns a deterministic, serializable snapshot (objects, classes, masks) safe to call inside a frame hook. Clear failures — a model that is the wrong size, fails its checksum, or lacks an expected output raises an error naming the model and its source. Browser entry — @gitframes/vision/web re-exports the engine plus createWebGPUProvider() / hasWebGPU(); onnxruntime-web is an optional lazy peer.

7. Headless Conformance & FrameGrid Testing

Pixel-sampling invariant assertions — test compositions in Vitest with skia-canvas to verify shader math, font coverage, and Mean Squared Error (MSE) temporal deltas. FrameGrid contact sheets — comp.renderFrameGrid(...) outputs sequential-frame contact sheets for instant review of easing, kinetic type, and transitions.

8. Live Preview in the Browser

Runs your composition, not a video — startPreview({ entry, export }) serves a localhost WebGPU player that loads the composition's own module and renders every frame live in the browser. Nothing is streamed: the server only hands over the bundle, the project's assets, and the soundtrack mixed by the export engine. Timeline, waveform & frame stepping — play/pause, scrub, step frame by frame, and read resolution, FPS, duration, and audio status at a glance. One stable URL per project — the port is derived from the working directory, so re-running the preview replaces the running server and any open tab reloads into the new version by itself. Close the tab and the server shuts down about five seconds later. Shown where you are — startPreview serves the page and returns its URL instead of opening a browser, so an agent can show it in its own pane (Claude Code, Codex); pass open: true to open the system browser.

import { startPreview } from "gitframes";

Monorepo Architecture Managed with pnpm workspaces and turbo: gitframes/ ├── packages/ │ ├── gitframes/ # Unified SDK (Composition, Layer, LayerAnimation, Signal, effects) │ ├── core/ # Core AST, Effect base class, VirtualMediaData, vision types │ ├── compositions/ # Layout engine, Flex/Box AST compiler, timeline evaluator │ ├── webgpu-renderers/ # WGSL shaders, Slug text engine, 3D renderer, camera, lights, materials │ ├── tensor-webgpu/ # WebGPU compute pipelines (Canny, depth-to-normals, flow, deflicker, landmarks) │ ├── vision/ # ONNX vision engine: detect, segment, pose, matte, tracking, signals │ ├── renderer/ # Headless Node.js WebGPU renderer via Dawn, WebCodecs, skia-canvas │ ├── renderers/ # Higher-level render orchestration │ ├── media/ # Media decoding / encoding adapters │ ├── node-sdk/ # Node renderer contracts and result schemas │ ├── server-utils/ # Server infrastructure, storage, asset caches │ └── client-utils/ # Shared browser utilities ├── nodes/ # 58+ specialized domain nodes (VFX, audio, layout, node-vision) ├── apps/ │ └── renderer-service/ # Production HTTP / gRPC rendering microservice container ├── examples/ # Reference compositions and films ├── plu

source

gatewai-dev/gitframes: Code-first video, rendered natively on WebGPU. Contribute to gatewai-dev/gitframes development by cr

gatewai-dev/gitframes

README

Comments (0)