Problem when loading saved params

Gluon
1

This is the error I get when I tried to load saved parameters.
Traceback (most recent call last):
File “exp.py”, line 85, in
rc.predict(test_x1, test_x2, test_y, test_l)
File “/home/name/RC/model.py”, line 234, in predict
self.model.load_params(self.model_path, ctx=ctx)
File “/home/name/virtualEnv/local/lib/python2.7/site-packages/mxnet/gluon/block.py”, line 317, in load_params
self.prefix)
File “/home/name/virtualEnv/local/lib/python2.7/site-packages/mxnet/gluon/parameter.py”, line 676, in load
self[name]._load_init(arg_dict[name], ctx)
File “/home/name/virtualEnv/local/lib/python2.7/site-packages/mxnet/gluon/parameter.py”, line 209, in _load_init
assert set(ctx) == set(self._deferred_init[1]),
IndexError: tuple index out of range

I saved the parameters like this:
self.model.save_params(self.model_path)

And I load it like this:
self.model.load_params(self.model_path, ctx=ctx)

Anyone has any ideas? Or any way to interpret the error message. Thanks very much!

2

Hi, could you specify which version of mxnet you are using, and provide a minimally reproducible example? Thanks!

3

MXNet version 0.9.3a3. It might took me some time to provide an example, but you can take a look at my source code here.
Save

And load here:
Load

Thank you very much for replying!

4

Thanks, could you try upgrading to MXNet 1.1.0 or 1.2.0 using pip install mxnet-cu90 --pre for example?

Also, do you re-create your .model object prior to trying to load the params?

5

Thanks for replying! I’ll upgrade it and get back to you with the result.
Yes, basically what I did is try to load the params with a new object. So instead of initializing it like I did in the training, I just load the params when predicting.

6

I see. Could give the shape of documents and questions parameter of your model?

7

Document should be of shape (batch_size, 25, 92, embedding_size)
Question should be of shape(batch_size, 126, embedding_size)
Document will be process by CNN and feed to LSTM along with Question. Really appreciate your help!

8

My apologies. The version is already 1.1.0.

9 
embedding_size = 400
vocab_size = 100
batch_size = 1
ctx = mx.gpu()
test = RCv1(nd.ones((vocab_size, embedding_size)), vocab_size=vocab_size, batch_size=batch_size)
test.model.collect_params().initialize(mx.init.Xavier(magnitude=2.24, rnd_type='gaussian'), ctx=ctx)
test.model(nd.ones((batch_size, 25, 94, embedding_size), ctx), nd.ones((batch_size, 126, embedding_size), ctx))


MXNetError: [21:52:28] src/operator/tensor/./matrix_op-inl.h:312: Check failed: shp.ndim() == param.axes.ndim() (4 vs. 3)

Are you sure about these shapes? ( I changed 92 to 94 to fix a previous error )

these are the correct shapes:

Document should be of shape (batch_size, 25, 94, embedding_size)
Question should be of shape(batch_size, 126)
10

I think you might have hit a bug in MXNet.
Here is a possible work around, you need to re-initialize your parameters before loading them:

working example

embedding_size = 400
vocab_size = 100
batch_size = 1
ctx = mx.gpu()
​
# Create first model
test = RCv1(nd.ones((vocab_size, embedding_size)), vocab_size=vocab_size, batch_size=batch_size)
# Initialize params
test.model.collect_params().initialize(mx.init.Xavier(magnitude=2.24, rnd_type='gaussian'), ctx=ctx)
# Runs one batch
print('Model 1', test.model(nd.ones((batch_size, 25, 94, embedding_size), ctx), nd.ones((batch_size, 126), ctx)))
# Save params
test.model.save_params('test.params')
​
# Create new model
test2 = RCv1(nd.ones((vocab_size, embedding_size)), vocab_size=vocab_size, batch_size=batch_size)
# Initilize params
test2.model.collect_params().initialize(mx.init.Xavier(magnitude=2.24, rnd_type='gaussian'), ctx=ctx)
# Reload params from file
test2.model.load_params('test.params', ctx=ctx)
# Run same batch
print('Model 2', test2.model(nd.ones((batch_size, 25, 94, embedding_size), ctx), nd.ones((batch_size, 126), ctx)))

Model 1 
[[ 0.24516296  0.23849127 -0.17715654 -0.00132978  0.011528   -0.35865149
   0.14470389  0.11249124  0.40096593 -0.05768602 -0.4132649  -0.17700988
  -0.24334204  0.11367183 -0.12864728 -0.09445646  0.05795538  0.23781824
   0.37032971 -0.03531937]]
<NDArray 1x20 @gpu(0)>
Model 2 
[[ 0.24516296  0.23849127 -0.17715654 -0.00132978  0.011528   -0.35865149
   0.14470389  0.11249124  0.40096593 -0.05768602 -0.4132649  -0.17700988
  -0.24334204  0.11367183 -0.12864728 -0.09445646  0.05795538  0.23781824
   0.37032971 -0.03531937]]
<NDArray 1x20 @gpu(0)>
1 Like
11

Sorry I was just about to get back to you. Yes, I got the shape wrong because I was thinking the processed question shape. Sorry about that.

Thank you very much! It worked just like you said. You are being so helpful. I will mark your post as solution.