Does mxnet allow to compute higher order differentiation like in tensorflow, pytorch? Because in their official documentation it’s written that only few operations are available for higher order differentiation.
Support for higher order differentiation is limited at the moment, but coverage is improving.
You should check out the mxnet.autograd.grad
function for this.
x = nd.array([1, 2, 3])
x.attach_grad()
with autograd.record():
y = nd.exp(x)
y_grad = autograd.grad(y, x, create_graph=True, retain_graph=True)[0]
y_grad.backward()
print(x.grad)
Simplest way to see if operators are supported is to try it, but you can tell by looking at the backend source code if you’re not scared of C++ code. One sign that an operator supports higher order gradients is that the FGradient
attribute is not registered to a backward operator, e.g. exp. And an indicator that an operator doesn’t support higher order gradients is that the FGradient
attribute is registered to a backward operator, and the backward operator itself doesn’t register another FGradient
function, e.g. sigmoid.
2 Likes
Thanks for answering, it solved my problem.