Gluon: Fix some parameters during training

[mseeger]

Hello, I’d like to train a model in a two-stage fashion. In the first stage, I’d like to keep a certain subset of parameters fixed to initial values (say, all parameters in a certain Block). In the second stage, I’d like to train all parameters.

What is an easy way to exclude certain parameters (say, those of a Block, being part of a larger model) from training? Do I need to filter collect_params()? Example appreciated!

[gogamza]

@mseeger

I’d like to keep a certain subset of parameters fixed to initial values (say, all parameters in a certain Block)

Check “block.weight.set_data()” function to set initial value. or use mx.init.Constant().

a= nn.Dense(in_units=2, units=3, weight_initializer=mx.init.Constant(F.array([[1,2], [4,5],[ 3,6]])))

a.collect_params().initialize( ctx=ctx)

a.weight.data()

[[ 1. 2.]
[ 4. 5.]
[ 3. 6.]]
<NDArray 3x2 @gpu(0)>

“net.bolck.collect_params().setattr(‘grad_req’, ‘null’)” could be help if you fix parameter of certain Block.