SVRG Optimization on gluon (mx.module on gl.trainer)


#1

Hi,

I want to use the mxnet.contrib.svrg_optimization module with gluon.trainer but I’m having some issues with it. I’ve tried to use a simple softmax layer like

​net = nn.HybridSequential()
net.add(nn.Dense(10))
net.initialize(init.Normal(sigma=0.01))

with
svrg = SVRGModule(net)

But it obviously fails with AttributeError: 'HybridSequential' object has no attribute 'list_arguments'.

Basically, is there a way to use a module.Module optimizer on gluon.trainer?

Thanks.


#2

You get this exception, because SVRGModule is working only with Module API.

What you can do is to write your model in Gluon, export to .json file and later load it back as a Symbol. You also would need to add your expected output to the loaded back model, because original Gluon one wouldn’t have a label associated with that. Take a look into this example:

import mxnet as mx
from mxnet.contrib.svrg_optimization.svrg_module import SVRGModule
from mxnet.gluon import nn
from mxnet.initializer import Normal

ctx = mx.cpu()

net = nn.HybridSequential()
net.add(nn.Dense(10))

net.initialize(Normal(sigma=0.01))

# hybridization and forward pass needed to make model exportable
net.hybridize(static_alloc=True)
net(mx.random.uniform(shape=(10, 5)))

# this would save model to json file
net.export("net")

# We load model back and attach SoftmaxOutput
# Without attaching softmax SVRGModule would complain that it cannot find the label
deserialized_net = mx.symbol.load("net-symbol.json")
net_with_softmax_output = mx.symbol.SoftmaxOutput(data=deserialized_net)

svrg = SVRGModule(net_with_softmax_output,
                  data_names=['data'],
                  label_names=['softmaxoutput0_label'],
                  update_freq=2)

I haven’t tried to train the module, because I don’t have training data. But feel free to try, and let me know if it works.


#3

For some reason it doesn’t give me any metrics or other outputs after the .fit() method, but I’m sure that’s an unrelated bug on my code.

Thanks for the extremely complete answer, Sergey.