add_sequence_controls

opennmt.inputters.add_sequence_controls(ids, length=None, start_id=None, end_id=None, pad_id=0)[source]

Adds sequence control tokens.

Parameters

  • ids – Sequence of token ids as a 1D tf.Tensor, a 2D tf.Tensor, or a 2D tf.RaggedTensor.

  • length – Length of each sequence in the batch as a 1D tf.Tensor. Required if ids is a 2D tf.Tensor.

  • start_id – Token id to prepend to the sequences (set None to disable).

  • end_id – Token id to append to the sequences (set None to disable).

  • pad_id – Padding token ID.

Returns

The updated token ids, and optionally the updated length if it was initially passed as argument.

Raises

  • ValueError – if ids is a dense 2D tf.Tensor, but length is not set.

  • ValueError – if the rank of ids is not 1 or 2.