opennmt.tokenizers.make_tokenizer

opennmt.tokenizers.make_tokenizer(config=None)[source]

Creates a tokenizer instance from the configuration.

Parameters

config – Path to a configuration file or the configuration dictionary.

Returns

A opennmt.tokenizers.Tokenizer instance.

Raises

ValueError – if config is invalid.