Hello, I have a problem where I want to use
autograd.record(heads, head_grads)
with a list of heads. For that to work, all entries in heads must be part of a graph built up in autograd.record(), or must have a gradient attached directly.
Is there a way to find out whether this situation holds for an NDArray object? I can track this myself, sure, but my code would be so much cleaner.
To give context:
a = mx.nd.zeros((1,))
a.attach_grad()
with autograd.record():
b = 2 * a
b.backward()
This works fine. But:
a = mx.nd.zeros((1,))
with autograd.record():
b = 2 * a
b.backward()
This fails, and tells me that b cannot be differentiated. I totally understand that, but:
Is there a way to find out whether b is not amenable for backward, except for trying it out?
For me, this is not even a sane option, because I want to use
autograd.backward
with a whole list of heads and head_gradients.
First, I thought that
b.grad is not None
does the job. But this does not work. For example:
a = mx.nd.zeros((1,)
a.attach_grad()
with autograd.record():
b = 2 * a
a.grad is None
==> False
b.grad is None
==> True
So this test just tells me whether I myself have attached a gradient. I suppose this is due to lazy creating of the grad variable, it exists after I have called b.backward(), but that is too late.
Thanks a lot for enlightment. I just could not find it in the NDArray API docs