Clip_gradient setting in gluon.trainer doesn't work

Gluon
1

Setting clip gradients in Trainer doesn’t work.
Below code is used to clip gradients to [-2.0, 2.0]. With this setting, I printed the weights gradients of the last convolutional layer, and it turns out that some gradients are still much larger than 2.0.

trainer = gluon.Trainer(ssh_net.collect_params(), ‘sgd’, {‘learning_rate’:1e-4, ‘wd’:5e-4, ‘clip_gradient’: 2., ‘lr_scheduler’:lr_sch, ‘momentum’:0.9})

2

The clipping is only applied by the optimizer in its update rule, the gradients themselves are computed with their true values.

2 Likes