mila-udem / blocks

    """Accumulates step with exponential discount.

    Combines :class:`BasicMomentum` and :class:`Scale` to form the

    usual momentum step rule.

    Parameters

    ----------

    learning_rate : float, optional

        The learning rate by which the previous step scaled. Defaults to 1.

    momentum : float, optional

        The momentum coefficient. Defaults to 0.

    Attributes

    ----------

    learning_rate : :class:`~tensor.SharedVariable`

        A variable for learning rate.

    momentum : :class:`~tensor.SharedVariable`

        A variable for momentum.

    See Also

    --------

    :class:`SharedVariableModifier`

    """

    def __init__(self, learning_rate=1.0, momentum=0.):

        scale = Scale(learning_rate=learning_rate)

        basic_momentum = BasicMomentum(momentum=momentum)

        self.learning_rate = scale.learning_rate

        self.momentum = basic_momentum.momentum

        self.components = [scale, basic_momentum]

class AdaDelta(StepRule):

    """Adapts the step size over time using only first order information.

    Parameters

    ----------

    decay_rate : float, optional

        Decay rate in [0, 1]. Defaults to 0.95.

    epsilon : float, optional

        Stabilizing constant for RMS. Defaults to 1e-6.

    Notes

    -----

    For more information, see [ADADELTA]_.

    .. [ADADELTA] Matthew D. Zeiler, *ADADELTA: An Adaptive Learning

       Rate Method*, arXiv:1212.5701.

    """

    def __init__(self, decay_rate=0.95, epsilon=1e-6):

        if not 0.0 <= decay_rate <= 1.0:

            norm = l2_norm(previous_steps.values())

            multiplier = tensor.switch(norm < self.threshold,

                                       1, self.threshold / norm)

            steps = OrderedDict(

                (parameter, step * multiplier)

                for parameter, step in previous_steps.items())

        return steps, []

class VariableClipping(StepRule):

    """Clip the maximum norm of individual variables along certain axes.

    This :class:`StepRule` can be used to implement L2 norm constraints on

    e.g. the weight vectors of individual hidden units, convolutional

    filters or entire weight tensors. Combine with :class:`Restrict`

    (and possibly :class:`CompositeRule`), to apply such constraints only

    to certain variables and/or apply different norm constraints to

    different variables.

    Parameters

    ----------

    threshold : float

        Maximum norm for a given (portion of a) tensor.

    axis : int or iterable, optional

        An integer single axis, or an iterable collection of integer

        axes over which to sum in order to calculate the L2 norm. If

        `None` (the default), the norm is computed over all elements

        of the tensor.

    Notes

    -----

    Because of the way the :class:`StepRule` API works, this particular

    rule implements norm clipping of the value *after* update in the

    following way: it computes ``parameter - previous_step``, scales it

    to have (possibly axes-wise) norm(s) of at most `threshold`,

    then subtracts *that* value from `parameter` to yield an 'equivalent

    step' that respects the desired norm constraints. This procedure

    implicitly assumes one is doing simple (stochastic) gradient descent,

    and so steps computed by this step rule may not make sense for use

    in other contexts.

    Investigations into max-norm regularization date from [Srebro2005]_.

    The first appearance of this technique as a regularization method

    for the weight vectors of individual hidden units in feed-forward

    neural networks may be [Hinton2012]_.

    .. [Srebro2005] Nathan Srebro and Adi Shraibman.