ashvardanian/ForkUnion
Lower-latency OpenMP-style NUMA-aware minimalistic scoped thread-pool designed for 'Fork-Join' parallelism in C, C++, Rust, & Zig, avoiding memory allocations, mutexes, CAS-primitives, ...
338 stars · 22 forks · C++ Topics: arm, concurrency, openmp, mpi, parallel-computing, multithreading, parallelism, thread-pool, numa, memory-model, threadpool, rayon, atomics, compare-and-swap, parallel-stl
README
ForkUnion
ForkUnion is a NUMA-aware fork-join thread-pool for C++, C, Rust, and Zig — built for the tightest #pragma omp parallel for-style loops, not task queues. 🍴 It already powers NUMA-sharded vector search with USearch, LLM KV-cache and attention kernels with NumKong, and unbalanced bioinformatics workloads — thousands of combinatorial tasks per core — with StringZilla. On the hot path it makes zero heap allocations, zero system calls, zero CAS operations, and suffers no false-sharing of cache-lines. So dispatch latency stays flat into the hundreds of cores, precisely where task-queue runtimes like Rayon collapse and even OpenMP begins to slip. It is also unique in parking idle workers on a hardware address monitor — x86 UMONITOR/UMWAIT, Arm WFET, RISC-V Zawrs — so a worker light-sleeps on the exact cache-line it is waiting for and the silicon wakes it the instant another core writes there, with no hot spinning and no kernel futex. That bet compounds as sockets multiply: Intel Xeon Platinum and NVIDIA Vera pack up to 8 sockets per node, where shared CAS thrashes the interconnect and private fetch_add cursors keep synchronization off it.
One parallel-for dispatch: ForkUnion is often ~3–6× faster than OpenMP and ~12–16× faster than Rayon and Taskflow — the fork-join tax, paid once per loop, cut by an order of magnitude. Full tables ↓
It is exhaustively tested for boundary-condition scheduling with miniaturized uint8_t indices, even runs on your big-endian 32-bit IBM mainframe, and ships with no_std and Miri coverage. The core is a C++ 17 library; the C 99, Rust, and Zig APIs bind it, and all four can pin threads to NUMA nodes or individual cores and allocate node-local memory. Despite being far more deeply tied to hardware and the OS than most alternatives, ForkUnion runs on six operating systems — Linux, FreeBSD, Windows, macOS, Android, and iOS — including asymmetric compute and memory topologies. Topology harvesting and thread placement work on all six; NUMA-local memory placement is implemented on Linux, FreeBSD, and Windows, and elsewhere allocation falls back to a single memory domain. Basic Usage ForkUnion is dead-simple to use! There is no nested parallelism, exception handling, or "future promises"; they are banned. The thread pool itself has a few core operations:
try_spawn to initialize worker threads, and for_threads to launch a blocking callback on all threads.
Higher-level APIs for index-addressable tasks are also available:
for_n - for individual evenly-sized tasks, for_n_dynamic - for individual unevenly-sized tasks, for_slices - for slices of evenly-sized tasks.
For additional flow control and tuning, following helpers are available:
sleep(microseconds) - for longer naps, terminate - to kill the threads before the destructor is called, unsafe_for_threads - to broadcast a callback without blocking, returning a generation token, unsafe_join - to block until the completion of a broadcasted generation, is_complete - to poll a generation token for completion without blocking.
Every dispatch is identified by an always-odd generation token. On caller_exclusive_k pools you can dispatch, overlap your own work, poll is_complete, and join - the classic poll-then-join pattern. On caller_inclusive_k pools the calling thread owes one slice of the work, which only runs inside unsafe_join, so completion can't be reached by polling alone. The same rule shapes the RAII guard returned by for_threads and the for_n family: on exclusive pools the work starts at the guard's construction, while on inclusive pools it runs at join or destruction. On Linux, in C++, given the maturity and flexibility of the HPC ecosystem, it provides NUMA extensions. That includes the colocated_pool analog of the flat_pool and the linux_numa_allocator for allocating memory in a specific memory domain. Those are out-of-the-box compatible with the higher-level APIs. Most interestingly, for Big Data applications, a higher-level distributed_pool class will address and balance the work across all compute domains. Intro in Rust To integrate into your Rust project, add the following lines to Cargo.toml: [dependencies] forkunion = "3.0.0" # detect what the platform offers forkunion = { version = "3.0.0", features = ["portable"] } # STL thread pool only forkunion = { version = "2.3.1", features = ["place-memory-on-domain"] } # require NUMA-aware allocations Or for the preview development version: [dependencies] forkunion = { git = "https://github.com/ashvardanian/ForkUnion.git", branch = "main-dev" } A minimal example may look like this: use forkunion as fu; let topology = fu::Topology::new().expect("Failed to detect hardware topology"); let mut pool = fu::spawn(&topology, 2); pool.for_threads(&|thread_index, compute_domain_index| { println!("Hello from thread # {} on compute domain # {}", thread_index + 1, compute_domain_index + 1); }); Higher-level APIs distribute index-addressable tasks across the threads in the pool: pool.for_n(100, |prong| { println!("Running task {} on thread # {}", prong.task_index + 1, prong.thread_index + 1); }); pool.for_slices(100, |prong, count| { println!("Running slice {}, {}) on thread # {}", prong.task_index, prong.task_index + count, prong.thread_index + 1); }); pool.for_n_dynamic(100, |prong| { println!("Running task {} on thread # {}", prong.task_index + 1, prong.thread_index + 1); }); A more realistic example with named threads and error handling may look like this: use std::error::Error; use forkunion as fu;
fn heavy_math(_: usize) {}
fn main() -> Result<(), Box<dyn Error>> { let topology = fu::Topology::new()?; let mut pool = fu::ThreadPool::try_named_spawn(&topology, "heavy-math", 4)?; pool.for_n_dynamic(400, |prong| { heavy_math(prong.task_index); }); Ok(()) } The Topology handle reports the hardware topology, to size and place work: let topology = fu::Topology::new()?; for domain in 0..topology.compute_domains_count() { let domain = fu::ComputeDomain(domain); println!("domain {}: {} cores, level {}, allocate on memory domain {}", domain.get(), topology.logical_cores_count_in(domain), topology.compute_level_in(domain), topology.local_memory_of(domain).get()); } For advanced usage, refer to the NUMA section below. For convenience Rayon-style parallel iterators pull the prelude module and check out related examples. Intro in C++ To integrate into your C++ project, either copy the include/ directory into your project, add a Git submodule, or CMake. The forkunion.hpp umbrella is a list of #includes; the implementation lives beside it in include/forkunion/, one header per concern, so a platform backend can be read or replaced on its own. For a Git submodule, run: git submodule add https://github.com/ashvardanian/ForkUnion.git extern/forkunion Alternatively, using CMake: FetchContent_Declare( forkunion GIT_REPOSITORY https://github.com/ashvardanian/ForkUnion GIT_TAG v3.0.0 ) FetchContent_MakeAvailable(forkunion) target_link_libraries(your_target PRIVATE forkunion::forkunion) Then, include the header in your C++ code: #include <forkunion.hpp> // `flat_pool_t` #include <cstdio> // `stderr` #include <cstdlib> // `EXIT_SUCCESS`
namespace fu = ashvardanian::forkunion;
int main() { alignas(fu::default_alignment_k) fu::flat_pool_t pool; if (!pool.try_spawn(fu::allowed_cores_count())) { std::fprintf(stderr, "Failed to fork the threads\n"); return EXIT_FAILURE; }
// Dispatch a callback to each thread in the pool pool.for_threads([& noexcept { std::printf("Hello from thread # %zu (of %zu)\n", thread_index + 1, pool.threads_count()); });
// Execute 1000 tasks in parallel, expecting them to have comparable runtimes // and mostly co-locating subsequent tasks on the same thread. Analogous to: // // #pragma omp parallel for schedule(static) // for (int i = 0; i < 1000; ++i) { ... } // // You can also think about it as a shortcut for the `for_slices` + `for`. pool.for_n(1000, [](std::size_t task_index) noexcept { std::printf("Running task %zu of 1000\n", task_index + 1); }); pool.for_slices(1000, [](std::size_t first_index, std::size_t count) noexcept { std::printf("Running slice %zu, %zu)\n", first_index, first_index + count); });
// Like `for_n`, but each thread greedily steals tasks, without waiting for // the others or expecting individual tasks to have same runtimes. Analogous to: // // #pragma omp parallel for schedule(dynamic, 1) // for (int i = 0; i < 3; ++i) { ... } pool.for_n_dynamic(3, [ noexcept { std::printf("Running dynamic task %zu of 3\n", task_index + 1); }); return EXIT_SUCCESS; } For advanced usage, refer to the NUMA section below. Every kernel and ISA facility the library uses is detected by default. CMake pins each with an AUTO/ON/OFF tri-state like -D FORKUNION_WITH_PLACE_MEMORY_ON_DOMAIN=ON or -D FORKUNION_WITH_PLACE_THREADS_BY_AFFINITY=OFF. Finer preprocessor gates - like the cache-line hints FU_WITH_DEMOTE_CACHE_LINES and FU_WITH_PROMOTE_CACHE_LINES, which push a just-written line toward the shared last-level cache via x86 CLDEMOTE, Arm DC CVAC, or RISC-V Zicbom so a consumer core finds it faster - accept the same overrides as compile definitions. Call fu_comptime_capabilities() to see what survived the build, fu_runtime_capabilities() to see what the machine underneath actually offers, and fu_name_capabilities() to render either mask as text - every binding exposes the same trio, Rust spelling it runtime_capabilities() and Zig runtimeCapabilities, alongside a version accessor: fu_version_major/minor/_patch in C, version() in Rust and Zig, and the FORKUNION_VERSION* macros in C++. Intro in Zig To integrate into your Zig project, let zig fetch pin the dependency and its content hash into build.zig.zon: zig fetch --save=forkunion https://github.com/ashvardanian/ForkUnion/archive/refs/tags/v3.0.0.tar.gz Then import and use in your code: const std = @import("std"); const fu = @import("forkunion");
const Context = struct { results: []i32 };
pub fn main() !void { const topology = try fu.Topology.init(); defer topology.deinit(); var pool = try fu.Pool.init(topology, 4, .inclusive); defer pool.deinit();
// Execute work on each thread (OpenMP-style parallel) pool.forThreads(struct { fn work(thread_idx: usize, compute_domain_idx: usize) void { _ = compute_domain_idx; std.debug.print("Thread {}\n", .{thread_idx}); } }.work, {});
// Distribute 1000 tasks across threads (OpenMP-style parallel for) var results = [_]i32{0} ** 1000; pool.forN(1000, struct { fn process(prong: fu.Prong, ctx: Context) void { ctx.results[prong.task_index] = @intCast(prong.task_index * 2); } }.process, Context{ .results = results[0..] }); } The Topology handle reports the hardware topology, to size and place work: const topology = try fu.Topology.init(); defer topology.deinit(); var domain: usize = 0; while (domain < topology.countComputeDomains()) : (domain += 1) { std.debug.print("domain {d}: {d} cores, level {d}, allocate on memory domain {d}\n", .{ domain, topology.countLogicalCoresIn(domain), topology.computeLevelIn(domain), topology.localMemoryOf(domain), }); } Unlike std.Thread.Pool task queue for async work, ForkUnion is designed for data parallelism and tight parallel loops — think OpenMP's #pragma omp parallel for with zero allocations on the hot path. Intro in C ForkUnion provides a pure C99 API via forkunion.h, wrapping the C++ implementation in pre-compiled libraries: forkunion_static.a or forkunion_dynamic.so. The C API uses opaque fu_pool_t handles and function pointers for callbacks, making it compatible with any C99+ compiler. To integrate using CMake: FetchContent_Declare( forkunion GIT_REPOSITORY https://github.com/ashvardanian/ForkUnion GIT_TAG v2.3.1 ) FetchContent_MakeAvailable(forkunion) target_link_libraries(your_target PRIVATE forkunion::forkunion_static) A minimal C example: #include <stdio.h> // printf #include <forkunion.h> // fu_pool_t, fu_pool_new, fu_pool_spawn
void hello_callback(void *context, size_t thread, size_t compute_domain) { (void)context; printf("Hello from thread %zu (compute_domain %zu)\n", thread, compute_domain); }
int main(void) { fu_topology_t topology = fu_topology_new(); fu_pool_t pool = fu_pool_new("my_pool", fu_capabilities_all_k); if (!topology || !pool || !fu_pool_spawn(topology, pool, fu_logical_cores_count(topology), fu_caller_inclusive_k)) return 1;
fu_pool_for_threads(pool, hello_callback, NULL); fu_pool_delete(pool); fu_topology_delete(topology); return 0; } The fu_-prefixed functions report the hardware topology, to size and place work: fu_topology_t topology = fu_topology_new(); for (size_t domain = 0; domain < fu_compute_domains_count(topology); ++domain) { printf("domain %zu: %zu cores, level %zu, allocate on memory domain %zu\n", domain, fu_logical_cores_count_in(topology, domain), fu_compute_level_in(topology, domain), fu_local_memory_of(topology, domain)); } fu_topology_delete(topology); For parallel tasks with context: struct task_context { int *data; size_t size; };
void process_task(void *ctx, size_t task, size_t thread, size_t compute_domain) { (void)thread; (void)compute_domain; struct task_context context = (struct task_context )ctx; context->data[task] = task * 2; }
int main(void) { fu_topology_t topology = fu_topology_new(); fu_pool_t pool = fu_pool_new("tasks", fu_capabilities_all_k); fu_pool_spawn(topology, pool, 4, fu_caller_inclusive_k);
int data[100] = {0}; struct task_context ctx = { .data = data, .size = 100 }; fu_pool_for_n(pool, 100, process_task, &ctx); // static scheduling fu_pool_for_n_dynamic(pool, 100, process_task, &ctx); // dynamic scheduling
fu_pool_delete(pool); fu_topology_delete(topology); return 0; } GCC Nested Functions Extension GCC supports nested functions that can capture variables from the enclosing scope: #include <stdio.h> #include <stdatomic.h> #include <forkunion.h>
int main(void) { fu_topology_t topology = fu_topology_new(); fu_pool_t pool = fu_pool_new("gcc_nested", fu_capabilities_all_k); fu_pool_spawn(topology, pool, 4, fu_caller_inclusive_k);
atomic_size_t counter = 0;
// GCC nested function - captures 'counter' from enclosing scope void nested_callback(void *ctx, size_t task, size_t thread, size_t compute_domain) { (void)ctx; (void)thread; (void)compute_domain; atomic_fetch_add(&counter, 1); }
fu_pool_for_n(pool, 100, nested_callback, NULL); printf("Completed %zu tasks\n", (size_t)atomic_load(&counter));
fu_pool_delete(pool); fu_topology_delete(topology); return 0; } Compile: gcc -std=c11 test.c -lforkunion_static -lpthread -lnuma Clang Blocks Extension Clang provides blocks with ^{} syntax: #include <stdio.h> #include <stdatomic.h> #include <Block.h> #include <forkunion.h>
typedef void (^task_block_t)(void *, size_t, size_t, size_t);
struct block_wrapper { task_block_t block; };
void block_wrapper_fn(void *ctx, size_t task, size_t thread, size_t compute_domain) { ((struct block_wrapper *)ctx)->block(NULL, task, thread, compute_domain); }
int main(void) { fu_topology_t topology = fu_topology_new(); fu_pool_t pool = fu_pool_new("clang_blocks", fu_capabilities_all_k); fu_pool_spawn(topology, pool, 4, fu_caller_inclusive_k);
__block atomic_size_t counter = 0;
task_block_t my_block = ^(void *c, size_t task, size_t t, size_t col) { (void)c; (void)t; (void)col; atomic_fetch_add(&counter, 1); };
task_block_t heap_block = Block_copy(my_block); struct block_wrapper wrapper = { .block = heap_block };
fu_pool_for_n(pool, 100, block_wrapper_fn, &wrapper);
Block_release(heap_block); printf("Completed %zu tasks\n", (size_t)atomic_load(&counter));
fu_pool_delete(pool); fu_topology_delete(topology); return 0; } Compile: clang -std=c11 -fblocks test.c -lforkunion_static -lpthread -lnuma -lBlocksRuntime Alternatives & Differences Most "thread-pools" are not thread-pools at all, but task-queues - think a std::queue<std::function<void()>> behind a std::mutex, executing each heap-allocated task on whatever core the OS scheduler picks. That shape is what most alternatives are built around, and it is the wrong shape for a tight parallel for:
Modern C++: taskflow/taskflow, progschj/ThreadPool, bshoshany/thread-pool Traditional C++: vit-vit/CTPL, mtrebi/thread-pool Rust: tokio-rs/tokio, rayon-rs/rayon, smol-rs/smol Zig: std.Thread.Pool
Reach for ForkUnion when you have data-parallel loops, bulk-synchronous phases, or NUMA-sharded scans, and you want them to dispatch in nanoseconds with neither an allocator nor the kernel on the hot path. Reach for something else when you need task graphs, async I/O, futures and promises, work that outlives its scope, or nested parallelism — Taskflow, Tokio, and oneTBB are built for exactly that, and ForkUnion deliberately bans it. The trade-offs line up like this, verified against each project's own source and documentation:
Property ForkUnion OpenMP Rayon Taskflow oneTBB
Hot-path heap allocations None ≈ None ¹ Per spawned task ² Per task node Per task, pooled
Syscalls to dispatch None Spin, then futex Spin/yield, then futex Spin, then futex Spin, then futex
Dynamic-schedule primitive fetch_add cursors CAS retry loop ³ Chase-Lev CAS deque ⁴ Chase-Lev CAS deque ⁴ Chase-Lev CAS deque ⁴
NUMA pinning & local allocators Yes Yes ⁵ No No ⁶ Partial ⁷
Hardware timed-wait Yes x86 only, opt-in ⁸ No No No
Languages C++ · C · Rust · Zig C · C++ · Fortran Rust C++ C++
Design center Fork-join loops Loops + tasks Stealing tasks + iterators Task-graph DAG Stealing tasks
¹ OpenMP reuses a persistent thread team, so a repeated parallel for over a fixed-size team does not re-create it; the per-region worksharing descriptor is reused but not provably allocation-free on every path. ² Rayon keeps join and parallel-iterator jobs on the stack, but scope/spawn heap-allocate one job per task, and the Chase-Lev deque grows on the heap under deep nesting. ³ libgomp and libomp hand out schedule(dynamic) iterations with an atomic fetch-add plus a CAS retry loop. ⁴ Rayon, Taskflow, and oneTBB all steal through a Chase-Lev deque whose steal path is a CAS. ⁵ OMP_PROC_BIND / OMP_PLACES pin threads, and the OpenMP 5 allocator and memory-space API places memory. ⁶ Taskflow exposes only a manual WorkerInterface hook for affinity, nothing automatic. ⁷ oneTBB pins threads to a NUMA node via task_arena constraints, but does not place memory locally for you. ⁸ Only LLVM/Intel libomp issues x86 UMONITOR / UMWAIT, gated behind KMP_USER_LEVEL_MWAIT and off by default; there is no Arm WFET or RISC-V Zawrs path in any OpenMP runtime.
Locks and Mutexes Unlike the std::atomic, the std::mutex is a system call, and it can be expensive to acquire and release. Its implementations generally have 2 executable paths:
the fast path, where the mutex is not contended, where it first tries to grab the mutex via a compare-and-swap operation, and if it succeeds, it returns immediately. the slow path, where the mutex is contended, and it has to go through the kernel to block the thread until the mutex is available.
On Linux, the latter translates to "futex" "system calls", which is expensive. Memory Allocations C++ has rich functionality for concurrent applications, like std::future, std::packaged_task,