integrations.cut_cross_entropy
integrations.cut_cross_entropy
Module for the Plugin for Cut Cross Entropy integration with Axolotl.
Cut Cross Entropy is an optimized implementation of cross entropy loss from Apple’s ML team.
Classes
| Name | Description |
|---|---|
| CutCrossEntropyPlugin | Plugin for Cut Cross Entropy integration with Axolotl. |
CutCrossEntropyPlugin
integrations.cut_cross_entropy.CutCrossEntropyPlugin()Plugin for Cut Cross Entropy integration with Axolotl.
Methods
| Name | Description |
|---|---|
| patch_llama_like | Generic patch for model architectures with causal lm similar to llama |
| pre_model_load | Apply cut cross entropy before model loading if enabled. |
patch_llama_like
integrations.cut_cross_entropy.CutCrossEntropyPlugin.patch_llama_like(
model_type_to_patch,
)Generic patch for model architectures with causal lm similar to llama
pre_model_load
integrations.cut_cross_entropy.CutCrossEntropyPlugin.pre_model_load(cfg)Apply cut cross entropy before model loading if enabled.