core.trainers.mixins.checkpoints

core.trainers.mixins.checkpoints

Checkpoint save handling: unsavable FSDP optimizers and retries after disk errors.

Classes

Name Description
CheckpointSaveMixin Mixin to handle saving the optimizer and scheduler if they are not savable.
CheckpointWriteRetryMixin Retry checkpoint file writes that fail with a full disk, quota or I/O error.

CheckpointSaveMixin

core.trainers.mixins.checkpoints.CheckpointSaveMixin()

Mixin to handle saving the optimizer and scheduler if they are not savable.

CheckpointWriteRetryMixin

core.trainers.mixins.checkpoints.CheckpointWriteRetryMixin()

Retry checkpoint file writes that fail with a full disk, quota or I/O error.

Functions

Name Description
with_checkpoint_write_retry Run a _save_checkpoint override inside the trainer’s write-retry scope.

with_checkpoint_write_retry

core.trainers.mixins.checkpoints.with_checkpoint_write_retry(method)

Run a _save_checkpoint override inside the trainer’s write-retry scope.