utils.data.packing
utils.data.packing
Sample-packing dataset transforms, light enough for streaming DataLoader workers.
Functions
| Name | Description |
|---|---|
| add_position_ids | Handle both single-example and batched data. |
| filter_sequences_by_length | Filter sequences outside valid length range [min_sequence_len, sequence_len]. |
add_position_ids
utils.data.packing.add_position_ids(sample)Handle both single-example and batched data. - single example: sample[‘input_ids’] is a list[int] - batched data: sample[‘input_ids’] is a list[list[int]]
filter_sequences_by_length
utils.data.packing.filter_sequences_by_length(
sample,
sequence_len=2048,
min_sequence_len=2,
raise_on_drop=False,
)Filter sequences outside valid length range [min_sequence_len, sequence_len].
Drops samples that are either too short (< min_sequence_len) or too long (> sequence_len).
Works for both single-example (list[int]) or batched (list[list[int]]).
If raise_on_drop is set, the code raises a ValueError if a sample is encountered that is too long and would have been dropped.