MultiHeadAttentionReduction

class opennmt.layers.MultiHeadAttentionReduction(value)

Enumeration defining how to reduce multi-head attention matrices into a single attention matrix.

Possible values are:

• NONE: do not reduce and return all attention heads.

• FIRST_HEAD_LAST_LAYER: take the first attention head of the last layer.

• AVERAGE_LAST_LAYER: average of all attention heads of the last layer.

• AVERAGE_ALL_LAYERS: average of all attention heads.

Inherits from: enum.Enum

NONE = 0

FIRST_HEAD_LAST_LAYER = 1

AVERAGE_LAST_LAYER = 2

AVERAGE_ALL_LAYERS = 3

static reduce(attention, policy)

Reduces a list of multi-head attention matrices into a single attention matrix.

Parameters

• attention – A list of multi-head attention matrices, with shape \([B, H, T_t, T_s]\).

• policy – A opennmt.layers.MultiHeadAttentionReduction value.

Returns

A single attention matrix with shape \([B, T_t, T_s]\).