CharacterTokenizer

class opennmt.tokenizers.CharacterTokenizer[source]

A tokenizer that splits unicode characters.

Inherits from: opennmt.tokenizers.Tokenizer

property in_graph: Returns True if this tokenizer can be run in graph (i.e. uses TensorFlow ops).