datasets

datasets

Module containing dataset functionality.

We want this to be a wrapper for an existing dataset that we have loaded. Lets use the concept of middlewares to wrap each dataset. We’ll use the collators later on to pad the datasets.

Classes

Name	Description
TokenizedPromptDataset	Dataset that returns tokenized prompts from a stream of text files.

TokenizedPromptDataset

datasets.TokenizedPromptDataset(
    prompt_tokenizer,
    dataset,
    process_count=None,
    keep_in_memory=False,
    **kwargs,
)

Dataset that returns tokenized prompts from a stream of text files.

Parameters

Name	Type	Description	Default
prompt_tokenizer	PromptTokenizingStrategy	The prompt tokenizing method for processing the data.	required
dataset	Dataset	Dataset with text files.	required
process_count	int | None	Number of processes to use for tokenizing.	None
keep_in_memory	bool | None	Whether to keep the tokenized dataset in memory.	False