monkeypatch.accelerate.parallelism_config

monkeypatch.accelerate.parallelism_config

ParallelismConfig monkeypatch.

Two extensions: - Allow pure CP standalone via ACCELERATE_ALLOW_CP_STANDALONE. - Add Expert Parallel (ep) as a first-class mesh axis inside the data-parallel group. Mesh order is (dp_replicate, dp_shard, cp, ep, sp, tp): ep sits innermost of the data axes so its all-to-all groups are runs of consecutive ranks and stay inside a node whenever ep_size * tp_size fits.

See expert_parallel/README.md for the full integration story.

Functions

Name	Description
data_parallel_index	Row-major index over the data axes (dp_replicate, dp_shard, ep) from a rank’s mesh
mesh_placement_report	Per axis: the size of its process groups and the fraction of those groups that span nodes,
patch_prepare_data_loader_for_ep	Apply the EP-aware data-loader patch.
patched_is_fsdp2	Patched version of is_fsdp2 that guards against a None fsdp_plugin.

data_parallel_index

monkeypatch.accelerate.parallelism_config.data_parallel_index(
    mesh_dim_names,
    coordinate,
    sizes,
)

Row-major index over the data axes (dp_replicate, dp_shard, ep) from a rank’s mesh coordinate, so ranks that only differ on cp/tp/sp share a data shard whatever the axis order.

mesh_placement_report

monkeypatch.accelerate.parallelism_config.mesh_placement_report(
    shape,
    names,
    gpus_per_node,
)

Per axis: the size of its process groups and the fraction of those groups that span nodes, assuming rank r lives on node r // gpus_per_node (the launcher default).

patch_prepare_data_loader_for_ep

monkeypatch.accelerate.parallelism_config.patch_prepare_data_loader_for_ep()

Apply the EP-aware data-loader patch.

Idempotent: replacing the bound function more than once is harmless because the wrapper closes over the current prepare_data_loader.

patched_is_fsdp2

monkeypatch.accelerate.parallelism_config.patched_is_fsdp2(self)

Patched version of is_fsdp2 that guards against a None fsdp_plugin.