AdmiralGloom / North-Mini-Code-1.0-GGUF-Q6 like 1

North-Mini-Code-1.0 - GGUF

GGUF quantization of CohereLabs/North-Mini-Code-1.0 \- a 30B-A3B (\~3B active) sparse Mixture-of-Experts coding model from Cohere Labs.

Requires cohere2\_moe llama.cpp support

This model uses the cohere2\_moe architecture (leading dense FFN layer, explicit head\_dim, hybrid full + 4096 sliding-window attention with a dense-first SWA phase). Mainline llama.cpp does not support it yet. Use the upstream pull request that adds full support - chat parser, tokenizer/EOG fixes, and native tool calls:

• llama.cpp PR #24260 (cohere2-moe). Build that branch until it merges.

Build:

git clone https://github.com/ggml-org/llama.cpp.git && cd llama.cpp
git fetch origin pull/24260/head:cohere2-moe && git checkout cohere2-moe
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build --config Release -j

On stock llama.cpp you will hit "wrong shape ... blk.0.attn\_q" or "missing tensor 'blk.0.ffn\_gate\_inp.weight'".

Files

| File | Quant | Size | Notes | | ------------------------------ | ----- | ------- | ----------------------------------------------------- | | North-Mini-Code-1.0-Q6\_K.gguf | Q6\_K | \~25 GB | Recommended for coding - minimal quality loss vs bf16 |

Run

./llama-server -m North-Mini-Code-1.0-Q6_K.gguf \
  --host 0.0.0.0 --port 8080 -c 40960 -ngl 99 --split-mode layer \
  --jinja --reasoning-format auto

Fits comfortably on 2x24 GB (e.g. dual RTX 3090). A single 24 GB card needs Q4/Q5.

Reasoning and tool use

This model is trained for interleaved thinking. With PR #24260's chat parser, `--jinja` exposes a separate `reasoning_content` field (think blocks) and native tool calls. For best agentic behaviour, feed both the thinking content AND tool calls back into the chat history on subsequent turns.

Sampling

Cohere's default is `temperature=1.0, top_p=0.95` (tuned for chat). For long code generation that can fall into repetition loops, so for coding prefer:

--temp 0.3 --top-p 0.9 --repeat-penalty 1.1 --repeat-last-n 320
plus DRY if your build has it:
--dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 2

Notes

• This is an early / pre-release model (shared by Cohere for community testing). Expect rough edges in instruction following, tool calling, and long-session stability; quality will improve toward the official release.
• Architecture: decoder-only MoE - 128 experts, 8 active/token, 49 layers (layer 0 dense), hybrid attention.
• Parameters: 30B total, \~3B active. Context: base supports 256K; these quants validated at 40K.

License and credit

Apache-2.0, inherited from the base model. Full credit to Cohere and Cohere Labs for North-Mini-Code-1.0, and to the llama.cpp contributors behind PR #24260. Quantized with llama.cpp.

Downloads last month 301

GGUF[](https://huggingface.co/docs/hub/gguf)

Model size

30B params

Architecture

cohere2moe

Chat template

Hardware compatibility

Log In to add your hardware

6-bit

Q6\_K 25.1 GB

Inference Providers NEW

Text Generation

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for AdmiralGloom/North-Mini-Code-1.0-GGUF-Q6 [](https://huggingface.co/docs/hub/model-cards#specifying-a-base-model)

Base model

CohereLabs/North-Mini-Code-1.0

Quantized

(33)

this model