prompt_strategies.pretrain

prompt_strategies.pretrain

pretraining prompt strategies

Classes

Name	Description
PretrainTokenizationStrategy	handles tokenization for pretraining with strides
PretrainTokenizer	basic tokenization class for pretraining

PretrainTokenizationStrategy

prompt_strategies.pretrain.PretrainTokenizationStrategy(
    *args,
    max_length=None,
    text_column='text',
    **kwargs,
)

handles tokenization for pretraining with strides

PretrainTokenizer

prompt_strategies.pretrain.PretrainTokenizer()

basic tokenization class for pretraining