How to train specific layers using gluon with different learning rate?


#1

Hi!
It’s actually 2 questions.

  1. For example, I want to train the frst conv layer while the other layers are freezed. However, it seems that I can only keep the base layer unchanged while training the last layer:

for X, y in train_data:
y = y.astype(‘float32’).as_in_context(ctx)
output_features = net.features(X.as_in_context(ctx))
with autograd.record():
outputs = net.output(output_features)
l = loss(outputs, y)
l.backward()
trainer.step(batch_size)
train_l += l.mean().asscalar()

  1. Is there any way to train different block of a net with different learning rate? Fro example , the lr of the output part is still lr, while the middle layers of the model are trained with lr/3, and the base is trained with even smaller lr/10. I know that in fast.ai it’s very easy to do so:

    lrs=np.array([lr/10,lr/3,lr])
    learner.fit(lrs/4,4,cycle_len=2,use_clr=(10,20))

How can I do above tasks?
Thanks a lot!