Moving PyMC3 from Theano to MXNet

[aseyboldt]

Hi, thanks for taking the time for that meeting by the way, it was fun
One thing I forgot to mention during the meeting is support for constant values. I wrote about this here before, but I guess it got lost a bit. Just wondering if you have any thought about that. (see also the separate thread about a possible design)

Since I wrote this I also noticed that it seems you ran into this issue yourselves, while implementing random draws from different distribution. The functions seem to assume that all parameters are of the same type. This is exactly the issue I am trying to work around.

@aseyboldt

Somewhere, we need to implement the log density of the beta distribution. Each parameter, and also the value, could be a symbol, a python scalar or a numpy array. For a distribution with n parameters that gives us 3**(n+1) possible combinations of input types, that each have a different api. In Theano, we can simply cast each parameter to a theano variable. If it is a numpy array or a python scalar, it wraps it in a constant, which behaves like any other theano variable. Additionally, it performs constant propagation during graph optimisation, so that computations that are done purely on constants only happen once during compilation.

The way I’d do that in mxnet right now (using my best current understanding of the library, which I still don’t know particularly well), would be to create a new parameter / symbol for each input, block the gradient if it is a constant, and manually keep track of which of those inputs are actual parameters that we want to know the posterior for, and which ones are just constants. This should work, but it would clutter the inputs of the model, and also make any future constant-related optimisations in the graph impossible. MxNet doesn’t even know that those values are constant.

Conceptually, I think it makes a lot of sense to look at a constant as an Op that doesn’t have any inputs. The actual value can be stored as auxiliary data. A design like that would prevent the cluttering of the inputs, keep the raw graph (exported json!) clean and would at least in principle allow constant-related optimizations (readonly aux data that is known when bind is called).