Operators

Tensor operations module for tiny-pytorch implementation.

This module provides a collection of fundamental tensor operations that form the building blocks of the computational graph in tiny-pytorch. Each operation is implemented as a class that inherits from the Op base class, with corresponding helper functions for easier usage.

The module includes element-wise operations, matrix operations, and various mathematical functions commonly used in deep learning.

Classes:

• ScalarAdd : Op –

  Addition of a scalar to a tensor.

• EWiseAdd : Op –

  Element-wise addition of two tensors.

• ScalarMul : Op –

  Multiplication of a tensor by a scalar.

• EWiseMul : Op –

  Element-wise multiplication of two tensors.

• Negate : Op –

  Negation of a tensor.

• ScalarPower : Op –

  Raising tensor elements to a scalar power.

• EWisePower : Op –

  Element-wise power operation between two tensors.

• ScalarDivide : Op –

  Division of a tensor by a scalar.

• EWiseDivide : Op –

  Element-wise division of two tensors.

• Reshape : Op –

  Reshaping a tensor to a new shape.

• Summation : Op –

  Summing tensor elements along specified axes.

• BroadcastTo : Op –

  Broadcasting a tensor to a larger shape.

• Transpose : Op –

  Transposing a tensor along specified axes.

• MatMul : Op –

  Matrix multiplication between two tensors.

• Log : Op –

  Natural logarithm of tensor elements.

• Exp : Op –

  Exponential of tensor elements.

• ReLU : Op –

  Rectified Linear Unit activation function.

• LogSumExp : Op –

  Log-sum-exp operation, commonly used in softmax computation.

Functions:

• add_scalar –

  Add a scalar to a tensor.

• add –

  Add two tensors element-wise.

• mul_scalar –

  Multiply a tensor by a scalar.

• multiply –

  Multiply two tensors element-wise.

• negate –

  Negate a tensor.

• power_scalar –

  Raise tensor elements to a scalar power.

• power –

  Element-wise power operation.

• divide_scalar –

  Divide a tensor by a scalar.

• divide –

  Element-wise division of tensors.

• reshape –

  Reshape a tensor.

• summation –

  Sum tensor elements along specified axes.

• broadcast_to –

  Broadcast tensor to a larger shape.

• transpose –

  Transpose tensor axes.

• matmul –

  Matrix multiplication.

• log –

  Natural logarithm.

• exp –

  Exponential function.

• relu –

  ReLU activation function.

• logsumexp –

  Log-sum-exp operation.

Notes

All operations support automatic differentiation through their gradient methods, making them suitable for building and training neural networks.

BroadcastTo

Bases: Op

Broadcast a tensor to a larger shape.

Parameters:

• shape (tuple) –

  Target shape to broadcast to.

Methods:

• compute –

  Compute the broadcast operation.

• gradient –

  Compute the gradient of the operation.

EWiseAdd

Bases: Op

Element-wise addition of two tensors.

Methods:

• compute –

  Compute element-wise addition.

• gradient –

  Compute the gradient with respect to both inputs.

EWiseDivide

Bases: Op

Element-wise division of two tensors.

Methods:

• compute –

  Compute element-wise division.

• gradient –

  Compute the gradient with respect to both inputs.

EWiseMul

Bases: Op

Element-wise multiplication of two tensors.

Methods:

• compute –

  Compute element-wise multiplication.

• gradient –

  Compute the gradient with respect to both inputs.

EWisePower

Bases: Op

Element-wise power operation between two tensors.

Methods:

• compute –

  Compute element-wise power operation.

• gradient –

  Compute the gradient with respect to both inputs.

Exp

Bases: Op

Exponential of tensor elements.

Methods:

• compute –

  Compute exponential.

• gradient –

  Compute the gradient of the operation.

Log

Bases: Op

Natural logarithm of tensor elements.

Methods:

• compute –

  Compute natural logarithm.

• gradient –

  Compute the gradient of the operation.

LogSumExp

Bases: Op

Log-sum-exp operation, commonly used in softmax computation.

Parameters:

• axes (tuple or None, default: None ) –

  Axes along which to perform the operation. If None, use all axes.

Methods:

• compute –

  Compute log-sum-exp operation.

• gradient –

  Compute the gradient of the operation.

MatMul

Bases: Op

Matrix multiplication between two tensors.

Methods:

• compute –

  Compute matrix multiplication.

• gradient –

  Compute the gradient with respect to both inputs.

Negate

Bases: Op

Negate a tensor element-wise.

Methods:

• compute –

  Compute the negation operation.

• gradient –

  Compute the gradient of the operation.

ReLU

Bases: Op

Rectified Linear Unit activation function.

Methods:

• compute –

  Compute ReLU activation.

• gradient –

  Compute the gradient of the operation.

Reshape

Bases: Op

Reshape a tensor to a new shape.

Parameters:

• shape (tuple) –

  The target shape for the tensor.

Methods:

• compute –

  Compute the reshape operation.

• gradient –

  Compute the gradient of the operation.

ScalarAdd

Bases: Op

Add a scalar to a tensor.

Parameters:

• scalar (float) –

  The scalar value to add to the tensor.

Methods:

• compute –

  Compute the scalar addition operation.

• gradient –

  Compute the gradient of the operation.

ScalarDivide

Bases: Op

Divide a tensor by a scalar.

Parameters:

• scalar (float) –

  The scalar value to divide by.

Methods:

• compute –

  Compute the scalar division.

• gradient –

  Compute the gradient of the operation.

ScalarMul

Bases: Op

Multiply a tensor by a scalar.

Parameters:

• scalar (float) –

  The scalar value to multiply with the tensor.

Methods:

• compute –

  Compute the scalar multiplication.

• gradient –

  Compute the gradient of the operation.

ScalarPower

Bases: Op

Raise tensor elements to a scalar power.

Parameters:

• scalar (float) –

  The power to raise tensor elements to.

Methods:

• compute –

  Compute the power operation.

• gradient –

  Compute the gradient of the operation.

Summation

Bases: Op

Sum tensor elements along specified axes.

Parameters:

• axes (tuple or None, default: None ) –

  Axes along which to perform summation. If None, sum over all axes.

Methods:

• compute –

  Compute the summation operation.

• gradient –

  Compute the gradient of the operation.

Transpose

Bases: Op

Transpose a tensor along specified axes.

Parameters:

• axes (tuple or None, default: None ) –

  Permutation of the dimensions. If None, reverse the dimensions.

Methods:

• compute –

  Compute the transpose operation.

• gradient –

  Compute the gradient of the operation.

add(a, b)

Add two tensors element-wise.

Parameters:

• a (Tensor) –

  First input tensor.

• b (Tensor) –

  Second input tensor.

Returns:

• Tensor –

  Element-wise sum of the input tensors.

add_scalar(a, scalar)

Add a scalar value to a tensor.

Parameters:

• a (Tensor) –

  Input tensor.

• scalar (float) –

  Scalar value to add.

Returns:

• Tensor –

  A new tensor with the scalar added to each element.

broadcast_to(a, shape)

Broadcast a tensor to a larger shape.

Parameters:

• a (Tensor) –

  Input tensor.

• shape (tuple) –

  Target shape to broadcast to.

Returns:

• Tensor –

  Broadcasted tensor with the specified shape.

divide(a, b)

Divide two tensors element-wise.

Parameters:

• a (Tensor) –

  Numerator tensor.

• b (Tensor) –

  Denominator tensor.

Returns:

• Tensor –

  Element-wise division of the input tensors.

divide_scalar(a, scalar)

Divide a tensor by a scalar value.

Parameters:

• a (Tensor) –

  Input tensor.

• scalar (float) –

  Scalar value to divide by.

Returns:

• Tensor –

  A new tensor with each element divided by the scalar.

exp(a)

Compute the exponential of tensor elements.

Parameters:

• a (Tensor) –

  Input tensor.

Returns:

• Tensor –

  Exponential of input tensor elements.

log(a)

Compute the natural logarithm of tensor elements.

Parameters:

• a (Tensor) –

  Input tensor.

Returns:

• Tensor –

  Natural logarithm of input tensor elements.

logsumexp(a, axes=None)

Compute log-sum-exp along specified axes.

Parameters:

• a (Tensor) –

  Input tensor.

• axes (tuple or None, default: None ) –

  Axes along which to perform the operation. If None, use all axes.

Returns:

• Tensor –

  Result of log-sum-exp operation.

matmul(a, b)

Perform matrix multiplication between two tensors.

Parameters:

• a (Tensor) –

  First input tensor.

• b (Tensor) –

  Second input tensor.

Returns:

• Tensor –

  Result of matrix multiplication.

mul_scalar(a, scalar)

Multiply a tensor by a scalar value.

Parameters:

• a (Tensor) –

  Input tensor.

• scalar (float) –

  Scalar value to multiply with.

Returns:

• Tensor –

  A new tensor with each element multiplied by the scalar.

multiply(a, b)

Multiply two tensors element-wise.

Parameters:

• a (Tensor) –

  First input tensor.

• b (Tensor) –

  Second input tensor.

Returns:

• Tensor –

  Element-wise product of the input tensors.

negate(a)

Negate a tensor element-wise.

Parameters:

• a (Tensor) –

  Input tensor.

Returns:

• Tensor –

  A new tensor with each element negated.

power(a, b)

Raise elements of one tensor to powers specified by another tensor.

Parameters:

• a (Tensor) –

  Base tensor.

• b (Tensor) –

  Exponent tensor.

Returns:

• Tensor –

  Element-wise power operation result.

power_scalar(a, scalar)

Raise tensor elements to a scalar power.

Parameters:

• a (Tensor) –

  Input tensor.

• scalar (float) –

  Power to raise elements to.

Returns:

• Tensor –

  A new tensor with each element raised to the given power.

relu(a)

Apply Rectified Linear Unit (ReLU) activation function.

Parameters:

• a (Tensor) –

  Input tensor.

Returns:

• Tensor –

  Tensor with ReLU activation applied.

reshape(a, shape)

Reshape a tensor to a new shape.

Parameters:

• a (Tensor) –

  Input tensor.

• shape (tuple) –

  Target shape for the tensor.

Returns:

• Tensor –

  A new tensor with the specified shape.

summation(a, axes=None)

Sum tensor elements along specified axes.

Parameters:

• a (Tensor) –

  Input tensor.

• axes (tuple or None, default: None ) –

  Axes along which to perform summation. If None, sum over all axes.

Returns:

• Tensor –

  Sum of elements along specified axes.

transpose(a, axes=None)

Transpose a tensor along specified axes.

Parameters:

• a (Tensor) –

  Input tensor.

• axes (tuple or None, default: None ) –

  Permutation of the dimensions. If None, reverse the dimensions.

Returns:

• Tensor –

  Transposed tensor.