opennmt.tokenizers package

Module defining tokenizers.

Tokenizers can work on string tf.Tensor as in-graph transformation.

opennmt.tokenizers.add_command_line_arguments(parser)[source]

Adds command line arguments to select the tokenizer.

opennmt.tokenizers.build_tokenizer(args)[source]

Returns a new tokenizer based on command line arguments.