Skip to main content

core.trainers.async_grpo.weight_sync

core.trainers.async_grpo.weight_sync

Async GRPO’s weight sync to vLLM.

Merged sync streams W + ΔW computed out of place, bitwise equal to the lora_merge_aware forward, instead of merging into the trained weights; adapter sync retires each adapter leaving the staleness window rather than deleting it; and AliasingVLLMClient serves the synced adapter under a fixed name.

Classes

Name Description
AliasingVLLMClient TRL’s vLLM client that also loads every synced adapter under alias.
WeightSyncMixin TRL’s weight sync, skipped after steps that trained nothing, merging out of

AliasingVLLMClient

core.trainers.async_grpo.weight_sync.AliasingVLLMClient(*args, alias, **kwargs)

TRL’s vLLM client that also loads every synced adapter under alias.

vLLM keys its prefix cache on the adapter name, which the alias keeps across versions, so a sync drains in-flight requests (pausing with mode=wait also clears the cache) rather than freezing them under the old adapter.

WeightSyncMixin

core.trainers.async_grpo.weight_sync.WeightSyncMixin(*args, **kwargs)

TRL’s weight sync, skipped after steps that trained nothing, merging out of place and retiring adapters instead of deleting them.