opennmt.tokenizers

Module defining tokenizers.

Tokenizers can work on string tf.Tensor as in-graph transformation.

• CharacterTokenizer
• OpenNMTTokenizer
• SentencePieceTokenizer
• SpaceTokenizer
• Tokenizer
• make_tokenizer
• register_tokenizer