Differences between mx.nd.Custom and mx.sym.Custom?

1

I am implementing a fairly complex CustomOp. The documentation is pretty incomplete there…

It does not only have ndarray input parameters, but also other arguments. I’ve been using mx.sym.Custom so far, where the following worked:
mx.sym.Custom(**kwargs)
Here, kwargs is a dict that has entries of type mx.sym (for the input args) and entries of type string. The latter are passed to the CustomOpProp constructor, while the former are passed to forward. Say:
kwargs = {‘x’: mx.sym.Variable(‘x’), ‘myarg’ : ‘foo’}
x is an input to forward,while myarg is passed to CustomOpProp.init

This works fine for what I want to do. But now, I’d like to switch to Gluon / autograd, and suddenly things break. When I write:
mx.nd.Custom(**kwargs)
where now kwargs contains either mx.nd for inputs, or string for CustomOpProp parameters, then I get an error.

More precisely, my CustomOp has three input arguments, called xmat, prior_sqprec, targets, and the CustomOpProp.init has a number of string argument. When I build kwargs and call
mx.nd.Custom(**kwargs)
I get:

Traceback (most recent call last):
File “_ctypes/callbacks.c”, line 315, in 'calling callback function’
File “/Users/matthis/ws_mxnet/src/MatthiasMXNetCode/mymxnet_env/lib/python2.7/site-packages/mxnet-0.12.1-py2.7.egg/mxnet/operator.py”, line 616, in creator
op_prop = prop_cls(**kwargs)
TypeError: init() got an unexpected keyword argument ‘prior_sqprec’

So it complains about the 2nd input argument prior_sqprec, not about xmat or targets.

I have no idea what is going on here. Maybe, my use of mx.sym.Custom is non-standard, but then I need a custom operator with (a) several inputs, and (b) with a number of additional args (which I code as string).

This seems to be a difference between mx.sym.Custom and mx.nd.Custom, which could be an issue for people switching from mx.sym to Gluon.

I can attach full code, but it is pretty long.

2

You are right, the documentation for CustomOp is not detailed enough, but that goes for most of the API documentation.

I think that inputs and auxiliary state arguments should be passed as positional args. You can wrap your CustomOp in a gluon block for a cleaner API:

# Python 3
class MyFunc(HybridBlock):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        with self.name_scope():
            self.aux0 = self.params.get('aux0', ...)

    def hybrid_forward(self, F, in0, in1, aux0=None):
        return F.Custom(in0, in1, in2, op_type='myfunc', extra_arg='foo')

Note that any extra keyword args to Custom will get converted to strings when passed to your
CustomOpProp’s __init__ method. Also note that there are a couple of open bugs on use of auxiliary states that have been fixed on master, so you should either avoid auxiliary state or build from master.

3

OK, so if I understand, you say one should pass mx.nd args (inputs, aux) first, followed by op_type, and then args for CustomOpProp.init. I will try that Monday.
Still funny that passing a single Python dict works for mx.sym.Custom, but not for mx.nd.Custom. Maybe it was never intended to work for mx.sym.Custom either …

Another (maybe stupid) question: I am implementing a loss function. Would I stlll do this as HybridBlock, even though it sits on the top, and not in the middle of the network?

4

The keyword arguments can be in any order. I have no idea why there is different behavior between nd and sym versions of Custom; perhaps the name of the arguments is different.

If you look at the implementation of the L2Loss function in Gluon, you will see that it is just a HybridBlock. It is not clear there is anything special about it unless the back-end specifically looks for that base class under the covers. If you are implementing a loss function, you should probably inherit from the Loss base class, but I don’t know if it actually matters.

5

Thanks Chris. It indeed works if I use positional args in the order you suggest.
So this solves the issue for me. I will try and implement my loss function as a HybridBlock then.

Bye,