SpaceTokenizer

class opennmt.tokenizers.SpaceTokenizer(in_graph=True)[source]

A tokenizer that splits on spaces.

Inherits from: opennmt.tokenizers.Tokenizer

__init__(in_graph=True)[source]

Initializes the tokenizer.

Parameters: in_graph – If True, this tokenizer should be integrated in the exported graph.

property in_graph: Returns True if this tokenizer can be run in graph (i.e. uses TensorFlow ops).