tokens_to_chars

opennmt.data.tokens_to_chars(tokens)

Splits tokens into unicode characters.

Example

>>> opennmt.data.tokens_to_chars(["hello", "world"])
<tf.RaggedTensor [[b'h', b'e', b'l', b'l', b'o'], [b'w', b'o', b'r', b'l', b'd']]>

Parameters

tokens – A string tf.Tensor of shape \([T]\).

Returns

The characters as a 2D string tf.RaggedTensor.