core.trainers.mixins.distributed_parallel
core.trainers.mixins.distributed_parallel
Mixin for correctly saving fsdp
Classes
| Name | Description |
|---|---|
| DistributedParallelMixin | Mixin for correctly saving fsdp |
DistributedParallelMixin
core.trainers.mixins.distributed_parallel.DistributedParallelMixin()Mixin for correctly saving fsdp
Functions
| Name | Description |
|---|---|
| tp_save_joins_all_ranks | Whether every TP rank must enter save_pretrained for the gather + barrier. |
tp_save_joins_all_ranks
core.trainers.mixins.distributed_parallel.tp_save_joins_all_ranks(
accelerator,
is_fsdp_enabled,
)Whether every TP rank must enter save_pretrained for the gather + barrier.