PyTorch (2) Tensor autograd functions

1. autograd:

In deep learning, we need to compute the gradient of the variables. The variables we discussed here are the variables in mathematics rather than the variables in the program. Torch provides autograd module to compute the gradient of the variable.

!! The Variable API has been deprecated: Variables are no longer necessary to use autograd with tensors. Autograd automatically supports Tensors with requires_grad set to True.

x = torch.tensor([1], requires_grad=True)
y = 2*x

y.backward() // compute the differentiation

print(x.grad)
# tensor([ 2])

y.backward() computes the differentiation of y so x.grad is equal to:
$$\frac{dy}{dx}=\frac{d(2x)}{dx}=2$$ Another example:

x = torch.tensor([1], requires_grad=True)
y = 2*x
z = y**2

z.backward() // compute the differentiation

print(x.grad)
# tensor([ 8])

z.backward() computes the differentiation of z so x.grad is equal to:
$$\frac{dz}{dx}=\frac{d(4x^{2})}{dx}=8x=8$$

2.Compute the gradient of the matrix:

x = torch.tensor([[1,2,3]], requires_grad=True)
w = torch.tensor([[4],[5],[6]])
y = x.matmul(w)

print(y)
# tensor([[ 32]])

y.backward()

print(x.grad)
# tensor([[ 4,  5,  6]])

$$x^T = \begin{bmatrix} x_1 & x_2 & x_3 \end{bmatrix} = \begin{bmatrix} 1 & 2 & 3 \end{bmatrix}$$
$$w = \begin{bmatrix} w_1\\ w_2\\ w_3 \end{bmatrix}=\begin{bmatrix} 4\\ 5\\ 6 \end{bmatrix}$$
$$y = x^T*w = w_1x_1 + w_2x_2 + w_3x_3$$
$$\frac{\partial y}{\partial x_1} = \frac{\partial (w_1x_1 + w_2x_2 + w_3x_3)}{\partial x_1} = w_1 = 4$$