Data Loaders

Data Iterator

class onmt.inputters.DynamicDatasetIter(corpora, corpora_info, transforms, vocabs, task, batch_type, batch_size, batch_size_multiple, data_type='text', bucket_size=2048, bucket_size_init=-1, bucket_size_increment=0, copy=False, skip_empty_level='warning', stride=1, offset=0)

Bases: torch.utils.data.dataset.IterableDataset

Yield batch from (multiple) plain text corpus.

Parameters

• corpora (dict[str, ParallelCorpus]) – collections of corpora to iterate;

• corpora_info (dict[str, dict]) – corpora infos correspond to corpora;

• transforms (dict[str, Transform]) – transforms may be used by corpora;

• vocabs (dict[str, Vocab]) – vocab dict for convert corpora into Tensor;

• task (str) – CorpusTask.TRAIN/VALID/INFER;

• batch_type (str) – batching type to count on, choices=[tokens, sents];

• batch_size (int) – numbers of examples in a batch;

• batch_size_multiple (int) – make batch size multiply of this;

• data_type (str) – input data type, currently only text;

• bucket_size (int) – accum this number of examples in a dynamic dataset;

• bucket_size_init (int) – initialize the bucket with this

• of examples; (amount) –

• bucket_size_increment (int) – increment the bucket

• with this amount of examples; (size) –

• copy (Bool) – if True, will add specific items for copy_attn

• skip_empty_level (str) – security level when encouter empty line;

• stride (int) – iterate data files with this stride;

• offset (int) – iterate data files with this offset.

Variables

• batch_size_fn (function) – functions to calculate batch_size;

• sort_key (function) – functions define how to sort examples;

• mixer (MixingStrategy) – the strategy to iterate corpora.

batch_iter(data, batch_size, batch_size_fn=None, batch_size_multiple=1)

Yield elements from data in chunks of batch_size, where each chunk size is a multiple of batch_size_multiple.

classmethod from_opt(corpora, transforms, vocabs, opt, task, copy, stride=1, offset=0)

Initilize DynamicDatasetIter with options parsed from opt.

class onmt.inputters.MixingStrategy(iterables, weights)

Bases: object

Mixing strategy that should be used in Data Iterator.

class onmt.inputters.SequentialMixer(iterables, weights)

Bases: onmt.inputters.dynamic_iterator.MixingStrategy

Generate data sequentially from iterables which is exhaustible.

class onmt.inputters.WeightedMixer(iterables, weights)

Bases: onmt.inputters.dynamic_iterator.MixingStrategy

A mixing strategy that mix data weightedly and iterate infinitely.

Dataset

class onmt.inputters.ParallelCorpus(name, src, tgt, align=None, src_feats=None)

Bases: object

A parallel corpus file pair that can be loaded to iterate.

load(offset=0, stride=1)

Load file and iterate by lines. offset and stride allow to iterate only on every stride example, starting from offset.

class onmt.inputters.ParallelCorpusIterator(corpus, transform, skip_empty_level='warning', stride=1, offset=0)

Bases: object

An iterator dedicated to ParallelCorpus.

Parameters

• corpus (ParallelCorpus) – corpus to iterate;

• transform (TransformPipe) – transforms to be applied to corpus;

• skip_empty_level (str) – security level when encouter empty line;

• stride (int) – iterate corpus with this line stride;

• offset (int) – iterate corpus with this line offset.