North Micro Vision Instruct

North Micro Vision Instruct is a 2.4B-parameter open-weight vision-language model comprising a 2B language model and a custom-trained 400M native-resolution vision encoder. Its compact scale and broad image-understanding capabilities make it a practical foundation for task-specific customization.

This guide shows how to fine-tune the model using Axolotl’s multimodal supervised fine-tuning (SFT) support. For architecture, training, and benchmark details, read the North Micro Vision technical blog post.

Thanks to the Cohere team for providing early access ahead of the release.

Getting Started

  1. Install Axolotl from main following the installation guide.

  2. Install Transformers. This model lands in v5.16, which is not released yet, so install the commit that adds it:

    uv pip install "transformers @ git+https://github.com/huggingface/transformers.git@f93eb6dbec8c2f5563f6d69bd998771929e8263f"

  3. Install Cut Cross Entropy to reduce training VRAM usage.

  4. Run one of the finetuning examples:

    # 5.1 GiB VRAM, adapters on the language decoder
axolotl train examples/cohere-north-micro-vision-instruct/qlora.yaml

# 5.2 GiB VRAM, adapters on the decoder, vision tower and projector
axolotl train examples/cohere-north-micro-vision-instruct/qlora-vision.yaml

# 21.2 GiB VRAM
axolotl train examples/cohere-north-micro-vision-instruct/fft.yaml

Let us know how it goes. Happy finetuning! 🚀

TIPS

  • Liger Kernels RMSNorm, RoPE and FLCE are not supported.

  • For Transformers inference, use the model’s generation defaults: temperature=0.7, top_p=0.8, and top_k=20.

  • Public vLLM support is coming soon. For vLLM inference, use temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, presence_penalty=1.5, and repetition_penalty=1.0.

  • Images are processed at native resolution, so token count scales with pixel count and sequence_len does not bound it. Cap it with the image processor’s max_pixels:

    processor_kwargs:
  max_pixels: 1003520  # total pixels

  • Read more on how to load your own dataset at docs.

  • The dataset format follows the OpenAI Messages format as seen here.

Optimization Guides

Please check the Optimizations doc.