Higher order differentiation

#1

Does mxnet allow to compute higher order differentiation like in tensorflow, pytorch? Because in their official documentation it’s written that only few operations are available for higher order differentiation.

#2

Support for higher order differentiation is limited at the moment, but coverage is improving.

You should check out the mxnet.autograd.grad function for this.

x = nd.array([1, 2, 3])
x.attach_grad()
with autograd.record():
    y = nd.exp(x)
    y_grad = autograd.grad(y, x, create_graph=True, retain_graph=True)[0]
y_grad.backward()
print(x.grad)

Simplest way to see if operators are supported is to try it, but you can tell by looking at the backend source code if you’re not scared of C++ code. One sign that an operator supports higher order gradients is that the FGradient attribute is not registered to a backward operator, e.g. exp. And an indicator that an operator doesn’t support higher order gradients is that the FGradient attribute is registered to a backward operator, and the backward operator itself doesn’t register another FGradient function, e.g. sigmoid.

2 Likes
#3

Thanks for answering, it solved my problem.