tgo-app-dev/vpipe

Real-time multimodal AI pipelines on Apple Silicon - tgo-app-dev/vpipe

45 stars · 4 forks

README

VPIPE Lightweight local multimodal AI pipelines and custom Metal inference for Apple Silicon Macs.

Multimodal graph: video, audio, images, text, and tool actions in one pipeline. Native Metal inference: custom kernels and no third-party tensor runtime, for optimal speed. Base-model Mac friendly: weight streaming and 4-bit model preparation for image/video generation on 16 GB machines. Out-of-the-box workflows: image editing, MiniMax H3 video, Qwen chat, realtime VQA, ASR/TTS, and local tool calling. Reproducible Composer: save layouts, prompts, params, and model config with each pipeline spec. Easy to use: macOS app, web UI control, and remote access from a phone browser.

Download the macOS app · Run the first example · Try image editing · Try MiniMax H3 video For the easiest install, choose the largest .dmg in the latest release.

  Prompt-driven image editing in Composer: live pipeline graph, preview panels, profiler, and reproducible model configuration. Why VPIPE

Runs MiniMax H3 FL2VA and REF2VA on an Apple Silicon Mac — a 33B model generating video and its soundtrack together, on as little as 16 GB. Now with Turbo LoRA support! See docs/MINIMAX-H3.md

5s @ 0.5 MP 24p, 6 steps takes ~13 minutes on a fanless 15-inch base-model M5 MacBook Air, 16 GB 1

Runs LTX-2.5 on an Apple Silicon Mac — a 22B model generating video and its soundtrack together, again on as little as 16 GB, through the vpipe-ltx-2.5 plugin.

Image and video generation on base-model Macs with weight streaming — walk through a reference image edit in docs/KLEIN-KV.md

Realtime multimodal pipelines for VQA, ASR, chat, TTS, image editing, video generation, and tool use. Now with Qwen 3.8 27B support!

Full-modality runtime under 30 MB 2, with a mobile-friendly web UI

Extra acceleration from the NAX matmul2d and convolution2d units on M5-generation hardware

Flagship workloads / performance MiniMax H3 Turbo LoRA — matched runs Matched MiniMax H3 runs using the same 960 × 544, 124-frame, 6-DiT-step workload and settings for both runtimes:

Hardware VPIPE h3.c

M4 Pro Mac mini, 64 GB RAM 21 min 50 sec 27 min 40 sec

Base M5 MacBook Air 15", 16 GB RAM 12 min 15 sec 16 min 22 sec

These numbers are workload-specific measurements, not a universal runtime speed claim. 3 Watch the VPIPE vs h3.c speed & quality comparison on YouTube →

Workload family Verified status

Video + audio generation MiniMax H3 FL2VA/REF2VA, Turbo LoRA, and LTX-2.5 plugin workflows.

Image generation / editing FLUX.2-klein-9b-kv: 4-bit reference image editing with 4-step default pipeline and compare-image UI on Apple Silicon.

Multimodal / LLM / VLM inference Qwen chat and VQA: local chat with image input, sampler control, stateful turns, and documented per-run token logs.

See docs/MINIMAX-H3.md for the H3 workload, settings, caveats, and Turbo LoRA notes; see docs/KLEIN-KV.md and docs/QWEN35-CHAT.md for image editing and chat workflows. Local AI use cases

Use case What VPIPE provides

Local image editing Prompt tuning, image comparison, mask/reference pipelines, and reproducible Composer layouts.

Text-to-video and image-to-video MiniMax H3 pipelines with video and soundtrack generation on Apple Silicon.

Multimodal chat Stateful local chat that can read images without re-prefilling the whole thread.

Realtime VQA and video monitoring Video capture, detection, tracking, audio tagging, overlays, preview, and profiling in one graph.

Local AI agents Sandboxed file, shell, Python, and web tools exposed through local MCP-style stages.

Developer integration A compact embeddable C++ runtime plus ready-to-run pipeline specs and reference workflows.

FAQ What is VPIPE? VPIPE is a local multimodal AI runtime for Apple Silicon Macs. It turns models, media streams, user input, and tool actions into inspectable C++ pipeline graphs. Does VPIPE use MPS, MLX, or Python for inference? No. VPIPE's generative model forward pass runs through Metal kernels directly via its own metal-compute backend. Python support exists only outside that forward path. Can VPIPE run local image and video generation on a 16 GB Mac? Yes, for supported workflows. VPIPE uses weight streaming and 4-bit model preparation to make image editing and video generation practical on base-model Apple Silicon machines. Is VPIPE a ComfyUI alternative for Mac? For some local image and video workflows, yes. VPIPE focuses on ready-to-run, reproducible pipelines with saved UI layouts and a native Apple Silicon compute backend, rather than a large general-purpose node ecosystem.

Install · Quickstart · First example · Overview · Examples For developers: Requirements · Build from source · Run · Tests · Structure · Acknowledgements · License

Install back to top The macOS app Download the latest release ▸ Open the .dmg, drag Vpipe Manager to Applications, and launch it. Requires an Apple Silicon Mac running macOS 26 or later. Two builds are published. They are the same app; they differ only in whether FFmpeg travels with it:

Download Size Pick this if

VpipeManager-<version>-with-ffmpeg.dmg ~28 MB You want it to work immediately. Nothing else to install.

VpipeManager-<version>-slim.dmg ~16 MB You already have FFmpeg installed — Homebrew's, say — and would rather use it.

The bundled copy is a minimal LGPL build: it covers the common formats and keeps hardware H.264/HEVC/ProRes through VideoToolbox, but it has no libx264/x265. If you take the slim build, point the app at your own FFmpeg — Quickstart covers where. The app is signed and notarized, so it opens without a Gatekeeper warning, and it updates itself: Vpipe Manager ▸ Check for Updates…

What you get. The app is a launcher around the same two binaries the command line uses — it does not reimplement anything. Its real advantage is permissions: Camera, Microphone and Local Network access are granted to the app under its own name, instead of to whichever terminal you happened to run from.

Build it yourself Everything also builds from source. The pipeline core and the non-Apple stages are portable C++20 and build on Linux and Intel macOS; the on-device model stack needs an Apple Silicon Mac. git clone --recursive https://github.com/tgo-app-dev/vpipe.git cd vpipe && cmake -S . -B build && cmake --build build -j That is the whole story when the dependencies are already in place. For prerequisites, build options, the Metal toolchain and the rest, see Requirements and Build from source below.

Quickstart back to top Getting from a fresh install to a working browser UI. Everything here is in the app; the command-line equivalents are under Run. 1. Choose a work directory. Settings ▸ Work Directory ▸ Path ▸ Choose… — it defaults to ~/vpipe. This is where everything lives: models/ for anything downloaded or quantized, sandbox/ for files pipelines are allowed to write, and the LMDB registry and logs. Pick a volume with room. Prepared models run to tens of gigabytes — MiniMax H3 downloads ~115 GB and peaks near 155 GB while it quantizes — and the app shows the free space on the volume you select. Moving it later means moving all of that. 2. Point at FFmpeg — only if you took the slim build. Settings ▸ FFmpeg ▸ Library Path ▸ Choose… Homebrew's is normally /opt/homebrew/lib. The row above the button says whether a usable FFmpeg was found, so you are not guessing. Use Default returns to the bundled copy. Either way the change takes effect the next time you start a pipeline or the server — it is not picked up by something already running.

3. Decide who can reach the web UI. Settings ▸ Web UI ▸ Bind To

This Mac only (127.0.0.1) — nothing else on the network can connect, and no access key is needed. Start here. Automatic (this Mac's LAN address), or a specific interface — so a phone or another computer can connect.

Anything other than This Mac only makes the web UI reachable from your network, and anyone who reaches it can start and stop pipelines, browse the sandbox and drive models on this Mac. An 8-character access key is the only thing in front of that, so use a LAN binding only when you actually need another device, and not on networks you do not trust. The app warns you in place when you select one. Port defaults to 9876; change it if something else is using it. HTTPS (Self-Signed) is only needed for the low-latency Preview view on another device — browsers restrict WebCodecs to secure contexts — and costs a one-time certificate warning.

4. Start it. Open the Web UI pane and press Start Server, then Open in Browser. If you chose a LAN address in step 3, the pane also shows a QR code: point a phone camera at it and the UI opens already authenticated, with no key to retype. If you kept This Mac only, there is no QR code — a phone could not reach 127.0.0.1 anyway. The access key is still shown, but this Mac connects without it. Open Work Folder and Open Sandbox Folder reveal those directories in Finder.

What the QR code contains. Only a URL: http://<this Mac's LAN address>:<port>/<token>. The token is 14 random characters, generated fresh at every start and never written to disk. Nothing about you, your files, your models or your machine is encoded in it — the single identifying detail is the LAN address, and that is a private one like 192.168.x.x, which means nothing outside your own network. Treat it as a password anyway, because the token is not merely information. Scanning it redirects to /?key=<access key> and hands the access key over, so anyone who can both see the code and reach that address gets exactly the control you have: starting and stopping pipelines, browsing the sandbox, driving models on this Mac. A photo, a screenshot or a screen share is enough for someone on the same network. Restarting the server invalidates it.

The status row along the bottom shows the machine's thermal state. Sustained image or video generation heat-soaks a Mac, a fanless MacBook Air especially. When it reads Throttling, steps are taking longer because of the hardware, not because something has stalled. Next: the first example, below.

First example — chat with a 9B model back to top The shortest path from a working install to a model answering you: two pipeline files, ~7.7 GB on disk, and no conversion step to sit through — this checkpoint arrives already quantized.

Get both pipelines — prepare-qwen35-9b-optiq-4bit.vpipeline and qwen35-9b-chat.vpipeline (Raw ▸ Save as, or straight from docs/pipelines/ in a clone). In the web UI from the Quickstart, open the Pipeline Manager, Load the prepare- one and press Start. It downloads the model into your work directory and registers it there. Once, and never again. Load the chat pipeline, Start it, open the User I/O panel — and type at the you> prompt.

you> In one sentence, what is the Pacific Ocean? The Pacific Ocean is the largest and deepest ocean on Earth, covering more than 30% of the planet's surface area and separating the continents of Asia and Australia to the east from North and South America to the west.

Five stages: a text input, the chat stage, a sampler stage carrying the values this checkpoint recommends for itself, and a feedback pair that makes it turn-by-turn. You can attach a picture to a turn — this model reads images too.

  The same pipeline from a phone — scan the QR code the server prints and the browser opens already authenticated. Here a photographed receipt is transcribed, then questioned: the follow-up answers from the same context, and the Mac decodes at ~19.7 tok/s throughout. From a source build the same two files run on the command line, which is where a chat prompt is most at home: cd ~/vpipe # your work directory, with both files in it /path/to/vpipe --launch prepare-qwen35-9b-optiq-4bit.vpipeline /path/to/vpipe --launch qwen35-9b-chat.vpipeline /path/to/vpipe is the binary from your build — build/apps/vpipe/vpipe inside the source tree. ▸ docs/QWEN35-CHAT.md — the walkthrough: what each stage is for, why sampling is a stage rather than a config key, and the knobs worth knowing. Then: EXAMPLES.md builds the same chat by hand in the web UI, and adds speech transcription. For image editing from a reference photo, docs/KLEIN-KV.md; for text-to-video with sound, docs/MINIMAX-H3.md. Each ships the pipelines it describes.

Overview back to top VPIPE has three main surfaces:

Pipeline core — coroutine-based Job stages connected by buffered ports, driven by a runtime that launches and drains them concurrently. Stages are composed into a pipeline from a JSON spec; each stage registers under a type name (e.g. rtsp-capture, video-to-rgb, yolo-detection, onvif-discovery, rest-client). This layer is portable C++20.

On-device generative-model stack (Apple Silicon) — a from-scratch LLM/VLM/ASR/diffusion/video inference stack running on metal-compute, with custom kernels, model loading, quantization support, weight streaming, and resource planning for memory-constrained Macs. It powers stages such as text-chat, visual-qa, realtime-vqa, audio-transcribe, generate-image, diffusion-conditioner, vae-encode, vae-decode, and generate-video.

Web UI and Composer — a self-contained browser UI for launching, inspecting, profiling, and editing pipelines. The Composer can arrange pipeline editors, previews, image comparison views, text I/O, profiler views, files, logs, and stage-provided panels, then save that layout with the pipeline spec so a workflow can be reopened and reproduced.

Models are loaded from local directories (sharded safetensors / GGUF) and are not bundled with the source.

Platform note. The pipeline core and the non-Apple stages build on Linux and Intel macOS, but the generative-model stack and the CoreML/Metal stages require an Apple Silicon Mac. On arm64 macOS these features are detected and enabled automatically.

For developers and power users back to top Everything above describes the app. The rest of this file is the source tree: what it needs to build, how to drive the same session from the command line, from Python or from a test binary, and where things live.

Requirements back to top

CMake ≥ 3.25 and a C++20 compiler (Apple Clang or a recent Clang/GCC).

Git (the build pulls a few dependencies as submodules).

FFmpeg development headers — libavformat, libavcodec, libavutil, libswresample. VPIPE compiles against the headers and dlopens the libraries at runtime, so FFmpeg must also be installed at runtime to decode media.

macOS: brew install ffmpeg Debian/Ubuntu: apt install libavformat-dev libavcodec-dev libavutil-dev libswresample-dev

libcurl — used by the rest-client stage. Provided by the SDK on macOS; on Linux install e.g. apt install libcurl4-openssl-dev.

Python 3 + development headers — only for the optional Python extension (built by default). Disable with -DVPIPE_BUILD_PYTHON=OFF if you don't need it.

Apple Silicon Mac — for the on-device inference stack and CoreML/Metal stages.

Metal shader toolchain (Apple Silicon builds only; recommended, not required) — by default the build compiles .metal kernel sources into embedded metallibs using xcrun -sdk macosx metal and xcrun -sdk macosx metallib. These compilers are part of Xcode's Metal Toolchain; the standalone Command Line Tools do not include them, even though the rest of the build never opens Xcode. No toolchain? The build falls back automatically. If metal/metallib aren't found at configure time, the build switches to runtime-compile mode: it embeds the Metal shader source and compiles each kernel on first use via the OS's built-in runtime compiler (newLibraryWithSource:), which needs no toolchain on the build or run machine. The Metal Toolchain is therefore optional; the tradeoff is a one-time per-kernel compile on first use instead of at build time. Force either mode with -DVPIPE_METAL_RUNTIME_COMPILE=ON|OFF. To get the faster build-time (AOT) path, install the toolchain. Two independent things can leave metal/metallib unavailable — both surface the same way, as error: cannot execute tool 'metal' or xcrun: error: unable to find utility "metal": 1. The Metal Toolchain isn't installed. On Xcode 26 and later (macOS 26) the Metal Toolchain is no longer bundled with Xcode by default — it's an optional component you download once. Install it from the command line (or via Xcode ▸ Settings ▸ Components ▸ Metal Toolchain ▸ Get): xcodebuild -downloadComponent metalToolchain # download + install On air-gapped or CI machines, export once and import where needed: xcodebuild -downloadComponent metalToolchain -exportPath ~/Downloads xcodebuild -importComponent metalToolchain ~/Downloads/metalToolchain.dmg 2. xcrun points at the Command Line Tools, not Xcode. If you installed the CLT and then Xcode, xcrun often still resolves to the standalone CLT, which lacks metal/metallib. Point the toolchain at Xcode:

Direct xcrun at the Xcode app (run once; needs admin)

Build from source back to top 1. Fetch dependencies. LMDB and pugixml are always required; nanobind is needed for the Python bindings, and metal-cpp for the Apple Silicon features: git submodule update --init extern/lmdb extern/pugixml extern/nanobind extern/metal-cpp 2. Configure (out-of-source build directory): cmake -S . -B build This defaults to an optimized Release build, so the binaries you get are performant out of the box. Override the build type explicitly if you want a debug build: cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug 3. Build: cmake --build build -j cmake --build is generator-agnostic; use it rather than calling make directly so the build works regardless of which generator CMake selected. Useful options (pass at configure time with -D):

Option Default Effect

VPIPE_BUILD_PYTHON ON Build the vpipe Python extension (needs Python dev headers).

VPIPE_BUILD_APPLE_SILICON auto (on for arm64 macOS) Build the CoreML/metal-compute wrappers and the inference stages.

CMAKE_BUILD_TYPE Release (when unset) Set Debug for an unoptimized debug build.

Install (optional): cmake --install build --prefix /path/to/install

Run back to top Web UI vpipe-web-ui serves a browser-based Pipeline Manager bound to one VPIPE session. The web assets are embedded in the binary, so no extra files are needed: ./build/apps/web-ui/vpipe-web-ui # listens on the LAN address, port 9876 ./build/apps/web-ui/vpipe-web-ui --bind 127.0.0.1 # this machine only Then open the printed URL (e.g. http://localhost:9876). By default it binds to the machine's LAN address so other devices can connect; remote connections must supply the 8-character access key printed at startup, while localhost connects without one. Options: --bind ADDR, --port N (0 = any free port), --config CFG (inline JSON, a file path, or empty for defaults), --help. Connecting a phone (--show-qr) The UI has a phone layout, and typing an 8-character key into a phone is exactly the friction that stops anyone from using it. --show-qr prints a QR code to the console next to the usual startup lines: ./build/apps/web-ui/vpipe-web-ui --show-qr Point a phone camera at it and the UI opens already authenticated — no key to read off the screen and retype. The phone layout is selected automatically from the device; ?ui=desktop (or the drawer's Desktop layout) overrides it, and ?ui=phone is how that layout is developed on a desktop. How it works, and what it costs:

source