Converting Gluon Neural Style to C++ Feedforward

I would like to be able to save .json and .params from the networks below

from

https://github.com/tornadomeet/mxnet/blob/master/example/gluon/style_transfer

class Net(Block):
def __init__(self, input_nc=3, output_nc=3, ngf=64, 
             norm_layer=InstanceNorm, n_blocks=6, gpu_ids=[]):
    super(Net, self).__init__()
    self.gpu_ids = gpu_ids
    self.gram = GramMatrix()

    block = Bottleneck
    upblock = UpBottleneck
    expansion = 4

    with self.name_scope():
        self.model1 = nn.Sequential()
        self.ins = Inspiration(ngf*expansion)
        self.model = nn.Sequential()

        self.model1.add(ConvLayer(input_nc, 64, kernel_size=7, stride=1))
        self.model1.add(norm_layer(in_channels=64))
        self.model1.add(nn.Activation('relu'))
        self.model1.add(block(64, 32, 2, 1, norm_layer))
        self.model1.add(block(32*expansion, ngf, 2, 1, norm_layer))


        self.model.add(self.model1)
        self.model.add(self.ins)

        for i in range(n_blocks):
            self.model.add(block(ngf*expansion, ngf, 1, None, norm_layer))
    
        self.model.add(upblock(ngf*expansion, 32, 2, norm_layer))
        self.model.add(upblock(32*expansion, 16, 2, norm_layer))
        self.model.add(norm_layer(in_channels=16*expansion))
        self.model.add(nn.Activation('relu'))
        self.model.add(ConvLayer(16*expansion, output_nc, kernel_size=7, stride=1))


def setTarget(self, Xs):
    F = self.model1(Xs)
    G = self.gram(F)
    self.ins.setTarget(G)

def forward(self, input):
    return self.model(input)

So this is cool I just call swap the network ndarray for the “data” symbol and save the .json and the .params, then load the model up in C++

by hacking up the evaluate function in,

https://github.com/tornadomeet/mxnet/blob/master/example/gluon/style_transfer/main.py

As described in:

http://gluon.mxnet.io/chapter07_distributed-learning/hybridize.html#Get-the-symbolic-program

in a similar sense to

But will the data[0] from the classification predict c++ (its actually C API) example be the pixels of the image with the style transferred onto it?

Also if I save a symbolic model, can I change the size of the image input into the model?

Or is this determined by the size of the image loaded up when the model was saved?

Obviously there would need to be one .params and .json file per style, but does this also burn in the resolution of the image the symbolic network is used?

Anyway I tried out the idea and I got stuck when I added the following lines to the end of evaluate

    x = mx.sym.var('data')
    y = style_model(x)
    y_json = y.tojson()
    y.save("MODEL.json")
    y.save_params("MODEL.params")

I get the following trace:

  File "/asset/common/software/thirdparty/mxnet/1.0.0-build1/python2.7/mxnet/gluon/block.py", line 360, in __call__
    return self.forward(*args)
  File "/home/samh/dev/MXNet-Gluon-Style-Transfer/net.py", line 215, in forward
    return self.model(input)
  File "/asset/common/software/thirdparty/mxnet/1.0.0-build1/python2.7/mxnet/gluon/block.py", line 360, in __call__
    return self.forward(*args)
  File "/asset/common/software/thirdparty/mxnet/1.0.0-build1/python2.7/mxnet/gluon/nn/basic_layers.py", line 53, in forward
    x = block(x)
  File "/asset/common/software/thirdparty/mxnet/1.0.0-build1/python2.7/mxnet/gluon/block.py", line 360, in __call__
    return self.forward(*args)
  File "/asset/common/software/thirdparty/mxnet/1.0.0-build1/python2.7/mxnet/gluon/nn/basic_layers.py", line 53, in forward
    x = block(x)
  File "/asset/common/software/thirdparty/mxnet/1.0.0-build1/python2.7/mxnet/gluon/block.py", line 360, in __call__
    return self.forward(*args)
  File "/home/samh/dev/MXNet-Gluon-Style-Transfer/net.py", line 123, in forward
    x = self.pad(x)
  File "/asset/common/software/thirdparty/mxnet/1.0.0-build1/python2.7/mxnet/gluon/block.py", line 360, in __call__
    return self.forward(*args)
  File "/home/samh/dev/MXNet-Gluon-Style-Transfer/net.py", line 44, in forward
    return F.pad(x, mode='reflect', pad_width=self.pad_width)
  File "<string>", line 112, in pad
AssertionError: Argument data must have NDArray type, but got <Symbol data>

So it seems that train in Python and use in C++ with gluon has quite a few caveats.

Also if I have to rewrite the Blocks in C++ is there API support for this?

1 Like

Please excuse my stupidity, the model needs the style image to define the padding and reflection amounts. So you can save the model in this state. Then load this up in C++ and run the content image through this.

I will try this out.

Sam

I am now working through this and it seems that some of the Hybrid functions are not implementing the hybrid_forward methods once this is one the saving will be possible.

There is good information here:
http://gluon.mxnet.io/chapter07_distributed-learning/hybridize.html

and I have put in a request for help here:

But it should really be here:

If anybody want to help, let me know.

OK I have reported it as an issue:

There is an issue that the model will not even train when it is not saved symbolically.

The model training should be independent with symbolic mode. I will follow up in github issue.