Skip to content

Dropout

simplegrad.nn.dropout.Dropout

Bases: Module

Apply dropout regularization during training.

During training, randomly zeroes elements of the input tensor with probability p. Disabled automatically in evaluation mode.

Parameters:

Name Type Description Default
p float

Probability of zeroing each element. Must be in [0, 1). Defaults to 0.5.

 0.5

Raises:

Type Description
ValueError

If p is not in [0, 1).
Source code in simplegrad/nn/dropout.py 
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
class Dropout(Module):
    """Apply dropout regularization during training.

    During training, randomly zeroes elements of the input tensor with
    probability ``p``. Disabled automatically in evaluation mode.

    Args:
        p: Probability of zeroing each element. Must be in ``[0, 1)``.
            Defaults to 0.5.

    Raises:
        ValueError: If ``p`` is not in ``[0, 1)``.
    """

    def __init__(self, p: float = 0.5):
        super().__init__()
        if not 0 <= p < 1:
            raise ValueError("Dropout probability must be in the range [0, 1).")
        self.p = p

    def forward(self, x: Tensor) -> Tensor:
        """Apply dropout to the input.

        In eval mode (or when ``p=0``), the input passes through unchanged.
        The random mask is generated at realize time, so this method is
        compatible with lazy execution mode.

        Args:
            x: Input tensor.

        Returns:
            Tensor with random elements zeroed (training) or unchanged (eval).
        """
        if not isinstance(x, Tensor):
            raise TypeError("Input must be a Tensor.")

        if self.eval_mode or self.p == 0:
            return _DropoutEval.apply(x)

        return _DropoutTrain.apply(x, self.p, oper=f"Dropout(p={self.p})")

    def __str__(self):
        return f"Dropout(p={self.p})"

forward(x: Tensor) -> Tensor

Apply dropout to the input.

In eval mode (or when p=0), the input passes through unchanged. The random mask is generated at realize time, so this method is compatible with lazy execution mode.

Parameters:

Name Type Description Default
x Tensor

Input tensor.

 required

Returns:

Type Description
Tensor

Tensor with random elements zeroed (training) or unchanged (eval).
Source code in simplegrad/nn/dropout.py 
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
def forward(self, x: Tensor) -> Tensor:
    """Apply dropout to the input.

    In eval mode (or when ``p=0``), the input passes through unchanged.
    The random mask is generated at realize time, so this method is
    compatible with lazy execution mode.

    Args:
        x: Input tensor.

    Returns:
        Tensor with random elements zeroed (training) or unchanged (eval).
    """
    if not isinstance(x, Tensor):
        raise TypeError("Input must be a Tensor.")

    if self.eval_mode or self.p == 0:
        return _DropoutEval.apply(x)

    return _DropoutTrain.apply(x, self.p, oper=f"Dropout(p={self.p})")