Is there any example of changing a parameter during testing (and not training)? Im specifically thinking of AlphaDropout - the noise should be added during the training but not testing… Just wanted to see if there was an example
Could you be more specific regarding what parameter what want to change? Did you add a custom operator for the activation layer?
No, I did (in gluon)
if mx.autograd.is_training():
noise = F.random.uniform()
else:
noise = 1
output = output * noise
I’d like to do this only during training - and not testing… How can I do that in mxnet symbol? In gluon I just use mx.autograd.is_training()
to check whether I should forward through an operation
@eric-haibin-lin is that clearer?
It’s a bit tricky to express that using Symbol. Is there any reason why Gluon is not used?
Legacy codebase - is there anything I can look at to mimic the behavior?
You can create a new module and bind with shared_module arg.
For example, let’s say the parameters/variables in net2
are all present in net1
:
def net1(...):
weight = mx.sym.var('w')
score = mx.sym.FullyConnected(data, weight)
out = mx.sym.SoftmaxOutput(score, label)
def net2(...):
weight = mx.sym.var('w')
score = mx.sym.FullyConnected(data, weight)
noise = mx.sym.random_uniform(...)
new_score = noise + score
out = mx.sym.SoftmaxOutput(new_score, label)
Then net2 can share the parameter in net1 with the following code:
module1 = mx.module.Module(net1, ...)
module2 = mx.module.Module(net2, ...)
module2.bind(ctx, ..., shared_module=module1)
How would that turn off the noise at test time?
Would this work for you:
train_test_gate = mx.sym.Variable('train_test_gate') # an array with (2,) shape
output = output * mx.sym.random.uniform() * train_test_gate[0] + output * train_test_gate[1]
Then you’d want to include train_test_gate
in your inputs in your DataIter and set it to mx.nd.array([1, 0])
for train dataset and mx.nd.array([0, 1])
for test dataset.
You can also do this:
is_training = mx.sym.Variable('is_training')
noise = mx.sym.random.uniform()
output = output * mx.sym.where(is_training, noise, mx.sym.ones_like(noise))
and set is_training
to 1 (or 0) in your training (or testing) dataset.
I should mention that topic of control flow blocks (e.g. if condition, for-loop, while-loop) in MXNet have been discussed in multiple threads before (like here and here) and the current recommendation is to use imperative if possible.