integrations.expert_parallel.args
integrations.expert_parallel.args
Pydantic args for the Expert-Parallel plugin.
Classes
| Name | Description |
|---|---|
| ExpertParallelArgs | Input args for the expert_parallel plugin. See the integration README. |
ExpertParallelArgs
integrations.expert_parallel.args.ExpertParallelArgs()Input args for the expert_parallel plugin. See the integration README.
Attributes
| Name | Description |
|---|---|
| expert_parallel_backend | Token dispatch backend. deep_ep: DeepEP fused kernels (NVLink/RDMA). torch: plain all_to_all_single over the EP process group (any NCCL/gloo setup). auto: deep_ep when importable, else torch. |
| expert_parallel_dispatch_chunks | torch backend only: split each MoE forward’s tokens into this many chunks and pipeline them, so chunk i+1’s dispatch all-to-all overlaps chunk i’s expert GEMMs. 1 keeps the unchunked path. Overlaps the forward only. Each chunk adds collective and launch overhead, so this can be slower than 1; benchmark before raising it past 2. |
| expert_parallel_save_dispatch | torch backend under activation checkpointing (gradient_checkpointing or FSDP2 fsdp_config.activation_checkpointing): save the dispatch/combine all-to-all outputs so backward does not re-issue the collectives, at the cost of holding the received rows per layer (24 GiB on Qwen3-30B-A3B at 32k packed on 2 GPUs). Recomputing them cost 1.6 s/step on a PCIe-only pair and was a tie on an NVLink pair, so the default recomputes; enable this on bandwidth-starved links with memory to spare. The routing topk and host-side split counts are always saved. |
| expert_parallel_size | Number of EP ranks. 1 = disabled (default), > 1 = enabled. |
| expert_parallel_token_capacity | Max tokens routed to any single expert per forward; the lowest-weight excess (token,expert) assignments are dropped. DeepEP’s intranode combine deadlocks once one expert is overloaded, and GLM-style routers concentrate more with depth — set this (e.g. 1024) for them. None = no cap. |
Functions
| Name | Description |
|---|---|
| validate_expert_parallel_topology | Reject EP layouts the sharding, clipping and save paths do not handle. |
validate_expert_parallel_topology
integrations.expert_parallel.args.validate_expert_parallel_topology(cfg)Reject EP layouts the sharding, clipping and save paths do not handle.