opennmt.utils.evaluator module

Evaluation related classes and functions.

class opennmt.utils.evaluator.ExternalEvaluator(labels_file=None, output_dir=None, scorers=None)[source]

Bases: object

Base class for external evaluators.

add_scorer(scorer)[source]

Adds a scorer to this evaluator.

__call__(step, predictions_path)[source]

Scores the predictions and logs the result.

Parameters:
  • step – The step at which this evaluation occurs.
  • predictions_path – The path to the saved predictions.
score(labels_file, predictions_path)[source]

Kept for backward compatibility: calls the first scorer.

class opennmt.utils.evaluator.Scorer(name)[source]

Bases: object

Scores hypotheses against references.

name

The scorer name.

__call__(labels_file, predictions_path)[source]

Scores the predictions against the true output labels.

lower_is_better()[source]

Returns True if a lower score is better.

summarize(writer, step, score)[source]

Summarizes the score in TensorBoard.

Parameters:
  • writer – A summary writer.
  • step – The current step.
  • scorer – The score to summarize.
log(score)[source]

Logs the score in the console output.

class opennmt.utils.evaluator.ROUGEScorer[source]

Bases: opennmt.utils.evaluator.Scorer

ROUGE scorer based on https://github.com/pltrdy/rouge.

summarize(writer, step, score)[source]

Summarizes the score in TensorBoard.

Parameters:
  • writer – A summary writer.
  • step – The current step.
  • scorer – The score to summarize.
log(score)[source]

Logs the score in the console output.

class opennmt.utils.evaluator.SacreBLEUScorer[source]

Bases: opennmt.utils.evaluator.Scorer

Scorer using sacreBLEU.

class opennmt.utils.evaluator.BLEUScorer(bleu_script='multi-bleu.perl', name='BLEU')[source]

Bases: opennmt.utils.evaluator.Scorer

Scorer calling multi-bleu.perl.

class opennmt.utils.evaluator.BLEUDetokScorer[source]

Bases: opennmt.utils.evaluator.BLEUScorer

Evaluator calling multi-bleu-detok.perl.

opennmt.utils.evaluator.make_scorers(names)[source]

Returns a list of scorers.

Parameters:names – A list of scorer names or a single name.
Returns:A list of opennmt.utils.evaluator.Scorer instances.
Raises:ValueError – if a scorer name is invalid.
opennmt.utils.evaluator.external_evaluation_fn(evaluators_name, labels_file, output_dir=None)[source]

Returns a callable to be used in opennmt.utils.hooks.SaveEvaluationPredictionHook that calls one or more scorers.

Parameters:
  • evaluators_name – A scorer name or a list of scorer names.
  • labels_file – The true output labels.
  • output_dir – The run directory.
Returns:

A callable or None if evaluators_name is None or empty.

Raises:

ValueError – if an scorer name is invalid.

class opennmt.utils.evaluator.ROUGEEvaluator(labels_file=None, output_dir=None)[source]

Bases: opennmt.utils.evaluator.ExternalEvaluator

ROUGE evaluator based on https://github.com/pltrdy/rouge.

class opennmt.utils.evaluator.SacreBLEUEvaluator(labels_file=None, output_dir=None)[source]

Bases: opennmt.utils.evaluator.ExternalEvaluator

Evaluator using sacreBLEU.

class opennmt.utils.evaluator.BLEUEvaluator(labels_file=None, output_dir=None)[source]

Bases: opennmt.utils.evaluator.ExternalEvaluator

Evaluator calling multi-bleu.perl.

class opennmt.utils.evaluator.BLEUDetokEvaluator(labels_file=None, output_dir=None)[source]

Bases: opennmt.utils.evaluator.ExternalEvaluator

Evaluator calling multi-bleu-detok.perl.