TransformersConverter

class ctranslate2.converters.TransformersConverter

Converts models from Hugging Face Transformers.

Inherits from: ctranslate2.converters.Converter

Methods:

__init__(model_name_or_path: str, activation_scales: Optional[str] = None, copy_files: Optional[List[str]] = None, load_as_float16: bool = False, revision: Optional[str] = None, low_cpu_mem_usage: bool = False, trust_remote_code: bool = False)

Initializes the converter.

Parameters

  • model_name_or_path – Name of the pretrained model to download, or path to the directory containing the pretrained model.

  • activation_scales – Path to the pre-computed activation scales. Models may use them to rescale some weights to smooth the intermediate activations and improve the quantization accuracy. See https://github.com/mit-han-lab/smoothquant.

  • copy_files – List of filenames to copy from the Hugging Face model to the converted model directory.

  • load_as_float16 – Load the model weights as float16. More precisely, the model will be loaded with from_pretrained(..., torch_dtype=torch.float16).

  • revision – Revision of the model to download from the Hugging Face Hub.

  • low_cpu_mem_usage – Enable the flag low_cpu_mem_usage when loading the model with from_pretrained.

  • trust_remote_code – Allow converting models using custom code.

convert(output_dir: str, vmap: Optional[str] = None, quantization: Optional[str] = None, force: bool = False) → str

Converts the model to the CTranslate2 format.

Parameters

  • output_dir – Output directory where the CTranslate2 model is saved.

  • vmap – Optional path to a vocabulary mapping file that will be included in the converted model directory.

  • quantization – Weight quantization scheme (possible values are: int8, int8_float32, int8_float16, int8_bfloat16, int16, float16, bfloat16, float32).

  • force – Override the output directory if it already exists.

Returns

Path to the output directory.

Raises

  • RuntimeError – If the output directory already exists and force is not set.

  • NotImplementedError – If the converter cannot convert this model to the CTranslate2 format.

convert_from_args(args: Namespace) → str

Helper function to call ctranslate2.converters.Converter.convert() with the parsed command line options.

Parameters

args – Namespace containing parsed arguments.

Returns

Path to the output directory.

static declare_arguments(parser: ArgumentParser) → ArgumentParser

Adds common conversion options to the command line parser.

Parameters

parser – Command line argument parser.

get_model_file(filename)
load_model(model_class, model_name_or_path, **kwargs)
load_tokenizer(tokenizer_class, model_name_or_path, **kwargs)