Loading RNN in C++: Check failed: MXExecutorBindEX


#1

I managed to train a simple feed forward DNN in Python, store the model to disk (.json + .params) and use the model for forwarding in C++: NDArray::Load() and Symbol::Load(), then SimpleBind() and exec->Forward(). Works like a charm.

Now I would like to do the same with an RNN, so I again trained a FusedRNNCell and stored the parameters to disk. I had to modify the shape of the input data from 2D to 3D:

train_x = train_x[:,np.newaxis,:]

Everything else is pretty similar. The model is fine, I can use it in Python. Now when I try to load it in C++, something goes wrong during the SimpleBind:

terminate called after throwing an instance of 'dmlc::Error'
  what():  [06:24:29] incubator-mxnet/cpp-package/include/mxnet-cpp/executor.hpp:84: Check failed: MXExecutorBindEX(symbol.GetHandle(), context.GetDeviceType(), context.GetDeviceId(), group_to_ctx.size(), map_keys.data(), dev_types.data(), dev_ids.data(), arg_handles.size(), arg_handles.data(), grad_handles.data(), grad_reqs_uint.data(), aux_handles.size(), aux_handles.data(), shared_exec_handle, &handle_) == 0 (-1 vs. 0) 

Stack trace returned 10 entries:
[bt] (0) mytool(dmlc::StackTrace()+0x3d) [0xd32d6d]
[bt] (1) mytool(dmlc::LogMessageFatal::~LogMessageFatal()+0x18) [0xd33028]
[bt] (2) mytool(mxnet::cpp::Executor::Executor(mxnet::cpp::Symbol const&, mxnet::cpp::Context, std::vector<mxnet::cpp::NDArray, std::allocator<mxnet::cpp::NDArray> > const&, std::vector<mxnet::cpp::NDArray, std::allocator<mxnet::cpp::NDArray> > const&, std::vector<mxnet::cpp::OpReqType, std::allocator<mxnet::cpp::OpReqType> > const&, std::vector<mxnet::cpp::NDArray, std::allocator<mxnet::cpp::NDArray> > const&, std::map<std::string, mxnet::cpp::Context, std::less<std::string>, std::allocator<std::pair<std::string const, mxnet::cpp::Context> > > const&, mxnet::cpp::Executor*)+0x525) [0xd9ff05]
[bt] (3) mytool(mxnet::cpp::Symbol::SimpleBind(mxnet::cpp::Context const&, std::map<std::string, mxnet::cpp::NDArray, std::less<std::string>, std::allocator<std::pair<std::string const, mxnet::cpp::NDArray> > > const&, std::map<std::string, mxnet::cpp::NDArray, std::less<std::string>, std::allocator<std::pair<std::string const, mxnet::cpp::NDArray> > > const&, std::map<std::string, mxnet::cpp::OpReqType, std::less<std::string>, std::allocator<std::pair<std::string const, mxnet::cpp::OpReqType> > > const&, std::map<std::string, mxnet::cpp::NDArray, std::less<std::string>, std::allocator<std::pair<std::string const, mxnet::cpp::NDArray> > > const&)+0x14d) [0xda2ebd]
[bt] (4) mytool(getScore(unsigned int, unsigned int) const+0x844) [0xd9c2c4]

How can I debug this? How do I know what fails within MXExecutorBindEX()? Is there anything specific to the RNN code that I have to modify (coming from a working DNN setup)?