DEV Community

Cover image for PyTorch :: Performing Derivatives on Tensors
Gurkirat Singh
Gurkirat Singh

Posted on

PyTorch :: Performing Derivatives on Tensors

Hello everyone,

Especially to those who fear from maths :P

In this you will learn how to perform derivatives on tensors using pytorch. Let me tell you, once you will learn, it will be fun for you.

Basically the derivatives are used for generating the parameters in the neural network. For now you will learn the basics of derivatives.

Suppose you have a function f(x)=x2f(x) = x^2 , so according to calculus, the derivative of f(x)f(x) would be df(x)x=2x1\frac{d f(x)}{x} = 2x^1

Let's see how can we do this and find the value of where $x=9$

Read more about it's usage: https://medium.com/@alfonsollanes/why-do-we-use-the-derivatives-of-activation-functions-in-a-neural-network-563f2886f2ab

import torch

x = torch.tensor(9.0, requires_grad=True)
print(x)
Enter fullscreen mode Exit fullscreen mode
tensor(9., requires_grad=True)
Enter fullscreen mode Exit fullscreen mode

This time I am setting requires_grad=True, this means we are using pytorch autograd engine to compute the derivatives. Derivatives are calculated by tracing the graph (enabled when requires_grad=True) from root to leaf by using chain rule (back propagation).

Note Only Tensors of floating point dtype can require gradients

For more information you can read this wonderful post: https://towardsdatascience.com/pytorch-autograd-understanding-the-heart-of-pytorchs-magic-2686cd94ec95

Also you can read the official documentation regarding autograd on pytorch's website: https://pytorch.org/docs/stable/notes/autograd.html

Moving on, let's calculate x2x^2

y = x**2
print(y)
Enter fullscreen mode Exit fullscreen mode
tensor(81., grad_fn=<PowBackward0>)
Enter fullscreen mode Exit fullscreen mode

If you would see, this time the pytorch has attached a gradient function with the tensor.

To calculate the derivative, you need to call Tensor.backward() method.

And use .grad property on the leaf tensor (here it's $x$)

y.backward()
print(x.grad)
Enter fullscreen mode Exit fullscreen mode
tensor(18.)
Enter fullscreen mode Exit fullscreen mode

Isn't it look like a magic to you? Let's calculate the derivative of another complex equation.

f(x)=sin(x22)f(x) = \sin ( \frac{x}{2 \sqrt 2} )

Where x=100x = 100

If you would simplify the denominator 222 \sqrt 2 , it would be 2322^{\frac{3}{2}}

x = torch.tensor(100., requires_grad=True)

y = torch.sin(x / (2 ** 1.5) )

print(x)
print(y)

y.backward()
print(x.grad)
Enter fullscreen mode Exit fullscreen mode
tensor(100., requires_grad=True)
tensor(-0.7158, grad_fn=<SinBackward>)
tensor(-0.2469)
Enter fullscreen mode Exit fullscreen mode

If you want to test it, you can go to https://www.derivative-calculator.net/ and type in the above expression.

You can also calculate the partial derivatives in pytorch. For example you have f(u,v)=uv+u2+v3f(u,v) = uv + u^2 + v^3 , the derivative df(u,v)du=2u+v\frac{d f(u,v)}{du} = 2u + v and df(u,v)dv=3v2+u\frac{d f(u,v)}{dv} = 3v^2 + u

Let's calculate the partial derivative using pytorch where u=2,v=4u = 2, v = 4

u = torch.tensor(2., requires_grad=True)
v = torch.tensor(4., requires_grad=True)

f = u**2 + v**3 + u*v

print(u)
print(v)
print(f)

f.backward()
print(u.grad)
print(v.grad)
Enter fullscreen mode Exit fullscreen mode
tensor(2., requires_grad=True)
tensor(4., requires_grad=True)
tensor(76., grad_fn=<AddBackward0>)
tensor(8.)
tensor(50.)
Enter fullscreen mode Exit fullscreen mode

If you have doubts or any idea reach me out via following sources

Top comments (0)