Exporting to GGUF

axolotl export converts a trained checkpoint into GGUF, the format used by llama.cpp, Ollama, LM Studio, and llamafile.

Note

GGUF is for the llama.cpp family of runtimes. For vLLM, SGLang, or TGI, serve the merged safetensors checkpoint directly (see merge-lora), optionally quantized with torchao or llm-compressor — vLLM’s GGUF loader is experimental and slower than its native path.

Prerequisites

Export shells out to a built llama.cpp checkout, which is not a Python dependency of Axolotl:

git clone https://github.com/ggml-org/llama.cpp
export LLAMA_CPP_DIR=$PWD/llama.cpp
cmake -S $LLAMA_CPP_DIR -B $LLAMA_CPP_DIR/build && cmake --build $LLAMA_CPP_DIR/build --config Release -j

Point at it with $LLAMA_CPP_DIR or the export.llama_cpp_dir config key. Building is only required if you request quantized outputs; llama-quantize is looked up in $LLAMA_CPP_DIR/build/bin/, $LLAMA_CPP_DIR/, then $PATH.

Warning

Do not pip install -r llama.cpp/requirements/... into your training environment. Those files pin an older transformers and a CPU-only torch, which would replace the versions Axolotl depends on. The conversion script runs under Axolotl’s interpreter, which already has everything it needs, and it loads gguf-py from the checkout itself.

Usage

axolotl export config.yml --quantize Q4_K_M,Q8_0

LoRA/QLoRA runs must be merged first — export refuses an adapter-only directory:

axolotl merge-lora config.yml
axolotl export config.yml

Configuration

Options are named after their convert_hf_to_gguf.py counterparts:

export:
  format: gguf         # Optional[str] = "gguf". Deployment format.
  outtype: f16         # Optional[str] = "f16". Weight type of the conversion: f32, f16, bf16, q8_0, tq1_0, tq2_0, auto.
  quantize: [Q4_K_M]   # Optional[list[str]] = []. Additional llama.cpp quant types to emit.
  outfile:             # Optional[str]. Defaults to {output_dir}/gguf/{run}-{ftype}.gguf.
  llama_cpp_dir:       # Optional[str]. Defaults to $LLAMA_CPP_DIR.

Every key can be overridden from the CLI (--outtype, --quantize, --outfile, --llama-cpp-dir), and --model-dir exports an arbitrary checkpoint instead of the one in output_dir.

As in llama.cpp, {ftype} in outfile is replaced by the weight type, so each requested quant type is written as its own file alongside the unquantized conversion:

outputs/my-run/gguf/
├── my-run-f16.gguf
├── my-run-Q4_K_M.gguf
└── my-run-Q8_0.gguf

Preflight checks

Conversion is slow, so a few failure modes are caught up front:

  • Pre-quantized checkpoints (torchao fp8/nvfp4, bitsandbytes) cannot be converted — re-export in bf16 with axolotl merge-lora --dequant.
  • Vocab mismatch between tokenizer.json and config.json (a common outcome of adding tokens without resizing embeddings).
  • Multi-token-prediction layers (num_nextn_predict_layers), which llama.cpp cannot load. See GLM-4.5.
  • Missing chat template, which is only a warning but usually means the runtime will fall back to a default template and undo your fine-tune.

Known gaps

Not yet supported: importance-matrix (imatrix) calibration, gguf-split for large models, multimodal mmproj files, exporting a LoRA adapter as a standalone GGUF, and pushing to the Hub.