In addition to standard dimension settings like the number of layers, the hidden dimension size, etc., OpenNMT also provides various model architecture.
The default encoder is a simple recurrent neural network (LSTM or GRU).
The bidirectional encoder (
-brnn) consists of two independent encoders: one encoding the normal sequence and the other the reversed sequence. The output and final states are concatenated or summed depending on the
Deep bidirectional encoder¶
The deep bidirectional encoder (
-dbrnn) is an alternative bidirectional encoder where the output of every layers are summed (or concatenated) prior feeding to the next layer.
Pyramidal deep bidirectional encoder¶
The pyramidal deep bidirectional encoder (
-pdbrnn) is an alternative deep bidirectional encoder that reduces the time dimension at each layer based on
The default decoder applies attention over the source sequence and implements input feeding by default.
Input feeding is an approach to feed attentional vectors "as inputs to the next time steps to inform the model about past alignment decisions" (Luong et al. (2015)). This can be disabled by setting
With residual connections the input of a layer is element-wise added to the output before feeding to the next layer. This approach proved to be useful for the gradient flow with deep RNN stacks (more than 4 layers).
The following components support residual connections with the
- default encoder
- bidirectional encoder
- default decoder
A bridge is an additional layer between the encoder and the decoder that defines how to pass the encoder states to the decoder. It can be one of the following:
-bridge copy(default): the encoder states are copied
-bridge dense: the encoder states are forwaded through a dense layer
-bridge dense_nonlinear: the encoder states are forwaded through a dense layer followed by a non-linearity, here
-bridge none: the encoder states are not passed and the decoder initial states are set to zero
copy bridge, encoder and decoder should have the same structure (number of layers, final hidden size, etc.).
Different models are available from Luong (2015) "Global Attention Model".
and the score function is one of these:
The model is selected using
-global_attention option or can be disabled with
-attention none option. The default attention model is