TransformerEncoderSpec

class ctranslate2.specs.TransformerEncoderSpec

Inherits from: ctranslate2.specs.LayerSpec

Methods:

__init__(num_layers: int, num_heads: int, pre_norm: bool = True, no_final_norm: bool = False, activation: Activation = Activation.RELU, num_source_embeddings: int = 1, embeddings_merge: EmbeddingsMerge = EmbeddingsMerge.CONCAT, layernorm_embedding: bool = False, relative_position: bool = False, relative_attention_bias: bool = False, ffn_glu: bool = False, rms_norm: bool = False, multi_query_attention: bool = False)

Initializes a Transformer encoder specification.

Parameters

  • num_layers – Number of layers.

  • num_heads – Number of attention heads.

  • pre_norm – Enable the pre-norm Transformer architecture.

  • no_final_norm – Disable the final layer norm in the pre-norm architecture.

  • activation – Activation to apply in the feed-forward network.

  • num_source_embeddings – Number of source embeddings.

  • embeddings_merge – When num_source_embeddings > 1, specify how the embeddings are merged.

  • layernorm_embedding – Apply layer normalization after the embedding layer.

  • relative_position – Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155.

  • relative_attention_bias – Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683.

  • ffn_glu – Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202.

  • rms_norm – Use the root mean square layer normalization.

  • multi_query_attention – Use multi-query attention.

optimize(quantization: Optional[str] = None) → None

Recursively applies some optimizations to this layer:

  • Alias variables with the same shape and value.

  • Quantize weights.

Parameters

quantization – Weight quantization scheme (possible values are: int8, int8_float32, int8_float16, int8_bfloat16, int16, float16, bfloat16, float32).

validate() → None

Verify that the required weights are set.

Raises

ValueError – If a required weight is not set in the specification.

variables(prefix: str = '', ordered: bool = False) → Dict[str, ndarray]

Recursively returns the weights from this layer and its children.

Parameters

  • prefix – Prefix to prepend to all variable names.

  • ordered – If set, an ordered list is returned instead.

Returns

Dictionary mapping variables name to value.