SelfAttentionEncoderLayer
- class opennmt.layers.SelfAttentionEncoderLayer(*args, **kwargs)[source]
Implements one self-attention encoding layer.
Inherits from:
keras.src.engine.base_layer.Layer- __init__(num_units, num_heads, ffn_inner_dim, dropout=0.1, attention_dropout=0.1, ffn_dropout=0.1, ffn_activation=<function relu>, mha_bias=True, maximum_relative_position=None, pre_norm=True, **kwargs)[source]
Initializes the layer.
- Parameters
num_units – The number of hidden units.
num_heads – The number of heads in the multi-head attention.
ffn_inner_dim – The number of units of the inner linear transformation in the feed forward layer.
dropout – The probability to drop units from the outputs.
attention_dropout – The probability to drop units from the attention.
ffn_dropout – The probability to drop units from the activation output in the feed forward layer.
ffn_activation – The activation function to apply between the two linear transformations of the feed forward layer.
mha_bias – Add bias after linear layers in the multi-head attention.
maximum_relative_position – Maximum relative position representation (from https://arxiv.org/abs/1803.02155).
pre_norm – If
True, layer normalization is applied before each sub-layer. Otherwise it is applied after.**kwargs – Additional layer arguments.