Higher order differentiation

Does mxnet allow to compute higher order differentiation like in tensorflow, pytorch? Because in their official documentation it’s written that only few operations are available for higher order differentiation.

Support for higher order differentiation is limited at the moment, but coverage is improving.

You should check out the mxnet.autograd.grad function for this.

x = nd.array([1, 2, 3])
x.attach_grad()
with autograd.record():
    y = nd.exp(x)
    y_grad = autograd.grad(y, x, create_graph=True, retain_graph=True)[0]
y_grad.backward()
print(x.grad)

Simplest way to see if operators are supported is to try it, but you can tell by looking at the backend source code if you’re not scared of C++ code. One sign that an operator supports higher order gradients is that the FGradient attribute is not registered to a backward operator, e.g. exp. And an indicator that an operator doesn’t support higher order gradients is that the FGradient attribute is registered to a backward operator, and the backward operator itself doesn’t register another FGradient function, e.g. sigmoid.

2 Likes

Thanks for answering, it solved my problem.