evaluate

evaluate

Module for evaluating models.

Functions

Name	Description
enable_callbacks_for_evaluation	Give trainer callbacks that gate a forward path on training state their chance to
evaluate	Evaluate a model on training and validation datasets.
evaluate_dataset	Helper function to evaluate a single dataset.
prepare_fsdp2_model_for_eval	Prepare the model and optimizer together under FSDP2 before a train-less evaluate.

enable_callbacks_for_evaluation

evaluate.enable_callbacks_for_evaluation(trainer)

Give trainer callbacks that gate a forward path on training state their chance to switch it on, since no on_train_begin fires in a standalone evaluation.

evaluate

evaluate.evaluate(cfg, dataset_meta)

Evaluate a model on training and validation datasets.

Parameters

Name	Type	Description	Default
cfg	DictDefault	Dictionary mapping axolotl config keys to values.	required
dataset_meta	TrainDatasetMeta	Dataset metadata containing training and evaluation datasets.	required

Returns

Name	Type	Description
	Dict[str, float]	Dictionary mapping metric names to their values.

evaluate_dataset

evaluate.evaluate_dataset(trainer, dataset, dataset_type, flash_optimum=False)

Helper function to evaluate a single dataset.

Parameters

Name	Type	Description	Default
trainer	Trainer	The trainer instance.	required
dataset	Dataset	Dataset to evaluate.	required
dataset_type	str	Type of dataset (‘train’ or ‘eval’).	required
flash_optimum	bool	Whether to use flash optimum.	False

Returns

Name	Type	Description
	Optional[Dict[str, float]]	Dictionary of metrics or None if dataset is None.

prepare_fsdp2_model_for_eval

evaluate.prepare_fsdp2_model_for_eval(trainer)

Prepare the model and optimizer together under FSDP2 before a train-less evaluate.

Trainer.evaluation_loop prepares a not-yet-prepared model on its own, which accelerate rejects for FSDP2 since the optimizer’s parameters must be swapped for the sharded ones in the same prepare() call.