opennmt.bin.tokenize_text module

Standalone script to tokenize a corpus.

opennmt.bin.tokenize_text.main()[source]