InvSqrtDecay

class opennmt.schedules.InvSqrtDecay(learning_rate, warmup_steps, initial_learning_rate=0)

Decay based on the reciprocal of the step square root. This corresponds to inverse_sqrt in Fairseq and --lr-decay-inv-sqrt in Marian.

During warmup (linear increase of the learning rate):

$$\text{schedule}(\text{step})=\text{init\_lr}+(\text{lr}-\text{init\_lr})\times \frac{\text{step}}{\text{warmup\_steps}}$$

After warmup:

$$\text{schedule}(\text{step})=\text{lr}\times \sqrt{\frac{\text{warmup\_steps}}{\text{step}}}$$

See also

• opennmt.schedules.RsqrtDecay

Inherits from: keras.src.optimizers.schedules.learning_rate_schedule.LearningRateSchedule

__init__(learning_rate, warmup_steps, initial_learning_rate=0)

Initializes the decay function.

Parameters

• learning_rate – The base learning rate.

• warmup_steps – The number of warmup steps.

• initial_learning_rate – Initial learning rate during warmup.