Save Memory When Operating on Tensors

To avoid allocating new memory when operating on tensors, we can either: - Do inplace operations

x = torch.randn(1000, 1000)
before = id(x)
x.add_(100)
id(x) == before #=> True

• Or change the current object that the tensor is pointing to

x = torch.randn(1000, 1000)
before = id(x)
x += 100
id(x) == before #=> True

• Or assign the result of the operation to the previously allocated tensor

x = torch.randn(1000, 1000)
before = id(x)
x[:] = x + 100
id(x) == before #=> True

Otherwise, if we do the following, it will create new object and make x point to it. This will put pressure on the host memory.

x = torch.randn(1000, 1000)
before = id(x)
x = x + 100
id(x) == before #=> False