Skip to content

Activations

simplegrad.functions.activations.relu(x: Tensor) -> Tensor

Apply ReLU activation element-wise: max(0, x).

Parameters:

Name Type Description Default
x Tensor

Input tensor.

 required

Returns:

Type Description
Tensor

Tensor with negative values replaced by zero.
Source code in simplegrad/functions/activations.py 
48
49
50
51
52
53
54
55
56
57
def relu(x: Tensor) -> Tensor:
    """Apply ReLU activation element-wise: max(0, x).

    Args:
        x: Input tensor.

    Returns:
        Tensor with negative values replaced by zero.
    """
    return _Relu.apply(x)

simplegrad.functions.activations.tanh(x: Tensor) -> Tensor

Apply hyperbolic tangent element-wise.

Parameters:

Name Type Description Default
x Tensor

Input tensor.

 required

Returns:

Type Description
Tensor

Tensor with values in (-1, 1).
Source code in simplegrad/functions/activations.py 
60
61
62
63
64
65
66
67
68
69
def tanh(x: Tensor) -> Tensor:
    """Apply hyperbolic tangent element-wise.

    Args:
        x: Input tensor.

    Returns:
        Tensor with values in (-1, 1).
    """
    return _Tanh.apply(x)

simplegrad.functions.activations.sigmoid(x: Tensor) -> Tensor

Apply sigmoid activation element-wise: 1 / (1 + exp(-x)).

Parameters:

Name Type Description Default
x Tensor

Input tensor.

 required

Returns:

Type Description
Tensor

Tensor with values in (0, 1).
Source code in simplegrad/functions/activations.py 
72
73
74
75
76
77
78
79
80
81
def sigmoid(x: Tensor) -> Tensor:
    """Apply sigmoid activation element-wise: 1 / (1 + exp(-x)).

    Args:
        x: Input tensor.

    Returns:
        Tensor with values in (0, 1).
    """
    return _Sigmoid.apply(x)

simplegrad.functions.activations.softmax(x: Tensor, dim: int | None = None) -> Tensor

Apply softmax along the given dimension.

Parameters:

Name Type Description Default
x Tensor

Input tensor.

 required
dim int | None

Dimension to normalize over. If None, normalizes over all elements.

 None

Returns:

Type Description
Tensor

Tensor where values along dim sum to 1.
Source code in simplegrad/functions/activations.py 
84
85
86
87
88
89
90
91
92
93
94
95
96
@compound_op
def softmax(x: Tensor, dim: int | None = None) -> Tensor:
    """Apply softmax along the given dimension.

    Args:
        x: Input tensor.
        dim: Dimension to normalize over. If None, normalizes over all elements.

    Returns:
        Tensor where values along ``dim`` sum to 1.
    """
    exps = exp(x)
    return exps / sum(exps, dim)