monkeypatch.models.llama4.modeling
monkeypatch.models.llama4.modeling
Modified Llama-4 text experts modeling for linearized experts for improved LoRA support
Classes
| Name | Description |
|---|---|
| Llama4TextExperts | Modified Llama-4 text experts modeling for linearized experts |
Llama4TextExperts
monkeypatch.models.llama4.modeling.Llama4TextExperts(config)Modified Llama-4 text experts modeling for linearized experts
Methods
| Name | Description |
|---|---|
| forward | Forward method using separate Linear layers for each expert. |
forward
monkeypatch.models.llama4.modeling.Llama4TextExperts.forward(hidden_states)Forward method using separate Linear layers for each expert.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| hidden_states | torch.Tensor | (num_experts * batch_size, hidden_size) The input should be organized by expert | required |
Returns
| Name | Type | Description |
|---|---|---|
| torch.Tensor | torch.Tensor: (num_experts * batch_size, hidden_size) |
Functions
| Name | Description |
|---|---|
| patch_llama4_linearized_modeling | Patch Llama4TextExperts to use separate Linear layers for each expert. |
patch_llama4_linearized_modeling
monkeypatch.models.llama4.modeling.patch_llama4_linearized_modeling()Patch Llama4TextExperts to use separate Linear layers for each expert.