FairseqConverter

class ctranslate2.converters.FairseqConverter

Converts models trained with Fairseq.

Inherits from: ctranslate2.converters.Converter

Methods:

__init__(model_path: str, data_dir: str, source_lang: Optional[str] = None, target_lang: Optional[str] = None, fixed_dictionary: Optional[str] = None, no_default_special_tokens: bool = False, user_dir: Optional[str] = None)

Initializes the Fairseq converter.

Parameters

  • model_path – Path to the Fairseq PyTorch model (.pt file).

  • data_dir – Path to the Fairseq data directory containing vocabulary files.

  • source_lang – Source language (may be required if not declared in the model).

  • target_lang – Target language (may be required if not declared in the model).

  • fixed_dictionary – Path to the fixed dictionary for multilingual models.

  • no_default_special_tokens – Require all special tokens to be provided by the user (e.g. encoder end token, decoder start token).

  • user_dir – Path to the user directory containing custom extensions.

convert(output_dir: str, vmap: Optional[str] = None, quantization: Optional[str] = None, force: bool = False) → str

Converts the model to the CTranslate2 format.

Parameters

  • output_dir – Output directory where the CTranslate2 model is saved.

  • vmap – Optional path to a vocabulary mapping file that will be included in the converted model directory.

  • quantization – Weight quantization scheme (possible values are: int8, int8_float32, int8_float16, int8_bfloat16, int16, float16, bfloat16, float32).

  • force – Override the output directory if it already exists.

Returns

Path to the output directory.

Raises

  • RuntimeError – If the output directory already exists and force is not set.

  • NotImplementedError – If the converter cannot convert this model to the CTranslate2 format.

convert_from_args(args: Namespace) → str

Helper function to call ctranslate2.converters.Converter.convert() with the parsed command line options.

Parameters

args – Namespace containing parsed arguments.

Returns

Path to the output directory.

static declare_arguments(parser: ArgumentParser) → ArgumentParser

Adds common conversion options to the command line parser.

Parameters

parser – Command line argument parser.