loaders.model

loaders.model

Model loader class implementation for loading, configuring, and patching various models.

Classes

Name	Description
ModelLoader	Manages model configuration, initialization and application of patches during

ModelLoader

loaders.model.ModelLoader(
    cfg,
    tokenizer,
    *,
    processor=None,
    inference=False,
    reference_model=False,
    **kwargs,
)

Manages model configuration, initialization and application of patches during model loading.

This class orchestrates the entire process of loading a model from configuration to final preparation. It handles device mapping, quantization, attention mechanisms, adapter integration, and various optimizations.

The loading process includes

• Loading and validating model configuration
• Applying monkey patches for optimizations / fixes
• Setting up device mapping (including multi-GPU configurations)
• Configuring quantization
• Setting attention mechanisms (Flash Attention, SDPA, etc.)
• Loading and initializing the model
• Applying adapters (LoRA, QLoRA, etc.)

Attributes

Name	Type	Description
model	PreTrainedModel | PeftModel | PeftMixedModel	The loaded model instance (available after load() is called).
model_kwargs	dict[str, Any]	Dictionary of keyword arguments passed to model initialization.
base_model		Name or path of the base model to load.
model_type		Type of model to load (e.g., AutoModelForCausalLM).
model_config		Configuration object for the model.
auto_model_loader		class used for loading the model (default: AutoModelForCausalLM).

Methods

Name	Description
load	Load and prepare the model with all configurations and patches.

load

loaders.model.ModelLoader.load()

Load and prepare the model with all configurations and patches.

Returns

Name	Type	Description
	tuple[PreTrainedModel | PeftModelForCausalLM, PeftConfig | None]	A tuple with the loaded model and its LoRA configuration (if applicable).

Functions

Name	Description
check_tensor_parallel_adapter_support	Adapters under TP exist only for native NVFP4 bases (torchao_tp_lora); on a bf16
check_tensor_parallel_rl_adapter_support	DPO/IPO/KTO hand TRL the peft config and TRL wraps the already TP-sharded model, after
no_split_module_classes	Module class names a device map must keep on one device (transformers stores them as a set).
should_convert_embedding_dtypes	Whether the post-quantization pass converts modules back to torch_dtype.
should_skip_peft_embedding_upcast	Whether PEFT’s kbit preparation should leave the embeddings alone.

check_tensor_parallel_adapter_support

loaders.model.check_tensor_parallel_adapter_support(native_nvfp4_prepared)

Adapters under TP exist only for native NVFP4 bases (torchao_tp_lora); on a bf16 base PEFT sizes the LoRA factors from the already-sharded weight, so every rank trains a different shard-local adapter and the save writes one shard.

Takes the result of prepare_native_nvfp4_tp_lora (or a bool for the all-or-nothing case). A partially quantized checkpoint whose unquantized modules are still in the tp_plan is rejected too: those LoRA modules would get plain PEFT on a TP-sharded base.

check_tensor_parallel_rl_adapter_support

loaders.model.check_tensor_parallel_rl_adapter_support(cfg)

DPO/IPO/KTO hand TRL the peft config and TRL wraps the already TP-sharded model, after the loader could install torchao_tp_lora, so even an NVFP4 base ends up shard-local.

no_split_module_classes

loaders.model.no_split_module_classes(model)

Module class names a device map must keep on one device (transformers stores them as a set).

should_convert_embedding_dtypes

loaders.model.should_convert_embedding_dtypes(cfg)

Whether the post-quantization pass converts modules back to torch_dtype.

should_skip_peft_embedding_upcast

loaders.model.should_skip_peft_embedding_upcast(cfg)

Whether PEFT’s kbit preparation should leave the embeddings alone.

Upcasting them there is pure peak memory whenever the later conversion back to torch_dtype reverts it, so skip it then unless the user asked otherwise.