Kimi Linear

Kimi Linear is a MoE model (48B total, 3B active) by MoonshotAI using a hybrid linear attention architecture to achieve a 1M token context length. It uses Kimi Delta Attention (KDA), a refined version of Gated DeltaNet that reduces KV cache size by up to 75% and boosts decoding throughput by up to 6x for long contexts.

This guide shows how to fine-tune it with Axolotl with multi-turn conversations and proper masking.

Note: Axolotl uses experimental training code for Kimi Linear as their original modeling code is inference-only.

Getting started

  1. Install Axolotl following the installation guide.

  2. Install CCE via docs

  3. Check that FLA 0.4.1 is installed — the KDA layers import their kernels from it:

    pip install fla-core==0.4.1 flash-linear-attention==0.4.1

    Axolotl already pins both on x86_64, so a standard install covers this. The version matters: fused_kda_gate changed signature in 0.4.1, so 0.4.0 raises a TypeError at the first KDA layer. FLA is not published for aarch64, which leaves Kimi Linear unsupported there.

  4. Run the finetuning example:

    axolotl train examples/kimi-linear/kimi-48b-lora.yaml

This config uses about 98.7GiB VRAM.

Let us know how it goes. Happy finetuning!

TIPS

  • Kimi Linear requires trust_remote_code: true.
  • You can run a full finetuning by removing the adapter: lora and load_in_8bit: true.
  • Read more on how to load your own dataset at docs
  • The dataset format follows the OpenAI Messages format as seen here

Optimization Guides

See 👉 docs.

Limitations

This is not yet compatible with MoE kernels from transformers v5.