Skip to main content

utils.data.packing

utils.data.packing

Sample-packing dataset transforms, light enough for streaming DataLoader workers.

Functions

Name Description
add_position_ids Handle both single-example and batched data.
filter_sequences_by_length Filter sequences outside valid length range [min_sequence_len, sequence_len].

add_position_ids

utils.data.packing.add_position_ids(sample)

Handle both single-example and batched data. - single example: sample[‘input_ids’] is a list[int] - batched data: sample[‘input_ids’] is a list[list[int]]

filter_sequences_by_length

utils.data.packing.filter_sequences_by_length(
    sample,
    sequence_len=2048,
    min_sequence_len=2,
    raise_on_drop=False,
)

Filter sequences outside valid length range [min_sequence_len, sequence_len].

Drops samples that are either too short (< min_sequence_len) or too long (> sequence_len).

Works for both single-example (list[int]) or batched (list[list[int]]).

If raise_on_drop is set, the code raises a ValueError if a sample is encountered that is too long and would have been dropped.