AssertionError / mod.forward(Batch(data = [mx.nd.array(img)]))


#1

Description

I’m following this guide https://aws.amazon.com/de/blogs/machine-learning/building-an-autonomous-vehicle-part-4-using-behavioral-cloning-with-apache-mxnet-for-your-self-driving-car/ to train a model with the provided trainingsdata and the valid.rec. When I add the code from “Evaluation and simulator” part of this guide. Following Error occurs.

Error Message

AssertionError                            Traceback (most recent call last)
<ipython-input-7-fabdcf182ee5> in <module>()
     22     img = np.swapaxes(img, 1, 2)
     23     img = img[np.newaxis, :]
---> 24     mod.forward(Batch(data = [mx.nd.array(img)]))
     25     exp = mod.get_outputs()[0].asnumpy()[0]
     26     angle = 180*exp

/usr/local/lib/python2.7/dist-packages/mxnet/module/module.pyc in forward(self, data_batch, is_train)
    588             Default is ``None``, which means ``is_train`` takes the value of ``self.for_training``.
    589         """
--> 590         assert self.binded and self.params_initialized
    591 
    592         curr_data_shapes = tuple(i.shape for i in self._data_shapes)

AssertionError: 

Can somebody help me or has some advice? I’m not really expirenced with mxnet and don’t know where to go from here now. I had more Problems with the code from this Guide and made some changes.

Code I used

import mxnet as mx
import numpy as np

data = mx.symbol.Variable(name="data")

body = mx.sym.Convolution(data=data, num_filter=24,  kernel=(5, 5), stride=(2,2)) 
body = mx.sym.Activation(data=body, act_type='relu', name='relu1')
body = mx.symbol.Pooling(data=body, kernel=(2, 2), stride=(2,2), pool_type='max')

body = mx.sym.Convolution(data=body, num_filter=32,  kernel=(5, 5), stride=(2,2))
body = mx.sym.Activation(data=body, act_type='relu')
body = mx.symbol.Pooling(data=body, kernel=(2, 2), stride=(2,2), pool_type='max')

flatten = mx.symbol.Flatten(data=body)

body = mx.symbol.FullyConnected(data=flatten, name='fc0', num_hidden=32)
body = mx.sym.Activation(data=body, act_type='relu', name='relu6')
body = mx.sym.Dropout(data=body, p=0.1)

body = mx.symbol.FullyConnected(data=body, name='fc1', num_hidden=16)
body = mx.sym.Activation(data=body, act_type='relu', name='relu7')

out = mx.symbol.FullyConnected(data=body, name='fc2', num_hidden=1)
out = mx.symbol.LinearRegressionOutput(data=out, name="softmax")

# Get Iterators

def get_iterators(batch_size, data_shape=(3, 120, 160)):
    train = mx.io.ImageRecordIter(
        path_imgrec         = 'train.rec', 
        data_name           = 'data',
        label_name          = 'softmax_label',
        batch_size          = batch_size,
        data_shape          = data_shape,
        shuffle             = True,
        rand_crop           = True,
        rand_mirror         = True)
    val = mx.io.ImageRecordIter(
        path_imgrec         = 'valid.rec',
        data_name           = 'data',
        label_name          = 'softmax_label',
        batch_size          = batch_size,
        data_shape          = data_shape,
        rand_crop           = False,
        rand_mirror         = False)
    return (train, val)

batch_size = 16
train_iter, val_iter = get_iterators(batch_size)

#Training

batch_size = 8
num_cpus = 1
num_epoch = 10
model_prefix = 'my-car2'
checkpoint = mx.callback.do_checkpoint(model_prefix)
mod = mx.mod.Module(out, context=[mx.cpu(i) for i in range(num_cpus)])
mod.fit(train_data=train_iter, 
        eval_data=val_iter, 
        eval_metric='mae', 
        optimizer='adam',
        optimizer_params={'learning_rate': 0.0001},
        num_epoch=num_epoch,
        epoch_end_callback = checkpoint,        
       )

	   
import os	
import time
import mxnet as mx
import numpy as np
%matplotlib inline
from PIL import Image
from IPython import display
from collections import namedtuple
import matplotlib.patches as patches
import matplotlib.pyplot as plt

PATH = 'trainingdata/'
all_files = sorted(os.listdir(PATH))
sym, arg_params, aux_params = mx.model.load_checkpoint('my-car2', num_epoch)
mod = mx.mod.Module(symbol=sym) 
fig, ax = plt.subplots(1)
Batch = collections.namedtuple('Batch', ['data'])
for fname in all_files:
    org_img = Image.open(PATH + fname)    
    img = np.array(org_img)
    img = np.swapaxes(img, 0, 2)
    img = np.swapaxes(img, 1, 2) 
    img = img[np.newaxis, :]
    mod.forward(Batch(data = [mx.nd.array(img)]))
    exp = mod.get_outputs()[0].asnumpy()[0]
    angle = 180*exp
    left = 80 * exp + 80
    rect = patches.Rectangle((left, 85),20,30, angle=angle,
                             linewidth=2,edgecolor='r',facecolor='none')
    patch = ax.add_patch(rect)
    display.clear_output(wait=True)
    display.display(plt.gcf())
    plt.imshow(org_img)
    time.sleep(0.1)
    patch.remove()

#Package used (Python/R/Scala/Julia):
I’m using:

  1. Ubuntu 16.04 in OracleVM
  2. Python2.7
  3. Jupyter qtconsole
  4. Mxnet v1.2.1 with cpu and pip

Steps to reproduce

  1. Downloaded trainingdata and valid.rec in one folder.
  2. Created from trainingdata train.lst and train.rec.
python im2rec.py /home/summer/summerws/train /home/summer/summerws/trainingdata/ --recursive --list --num-thread 8
python im2rec.py /train /home/summer/summerws/trainingdata --recursive --pass-through --pack-label --num-thread 8
  1. Start Jupyter QtConsole from terminal inside of the folder with train.rec, valid.rec and trainingdata/
jupyter qtconsole
  1. Copied my code inside the jupyter qtconsole

Thanks in advance for your help.


#2

Update: I checked the values of binded and params_initialized they were both false. Before

sym, arg_params, aux_params = mx.model.load_checkpoint('my-car2', num_epoch)
mod = mx.mod.Module(symbol=sym)

they were both set to true. I removed these to lines for test purpose since my module is still loaded at this point.

But now i’m getting this error:

AssertionError                            Traceback (most recent call last)
<ipython-input-3-75fba45fdb96> in <module>()
     23     img = np.swapaxes(img, 1, 2)
     24     img = img[np.newaxis, :]
---> 25     mod.forward(Batch(data = [mx.nd.array(img)]))
     26     exp = mod.get_outputs()[0].asnumpy()[0]
     27     angle = 180*exp

/usr/local/lib/python2.7/dist-packages/mxnet/module/module.pyc in forward(self, data_batch, is_train)
    608                 new_lshape = None
    609 
--> 610             self.reshape(new_dshape, new_lshape)
    611 
    612         self._exec_group.forward(data_batch, is_train)

/usr/local/lib/python2.7/dist-packages/mxnet/module/module.pyc in reshape(self, data_shapes, label_shapes)
    469             self.data_names, self.label_names, data_shapes, label_shapes)
    470 
--> 471         self._exec_group.reshape(self._data_shapes, self._label_shapes)
    472 
    473     def init_optimizer(self, kvstore='local', optimizer='sgd',

/usr/local/lib/python2.7/dist-packages/mxnet/module/executor_group.pyc in reshape(self, data_shapes, label_shapes)
    380         if self._default_execs is None:
    381             self._default_execs = [i for i in self.execs]
--> 382         self.bind_exec(data_shapes, label_shapes, reshape=True)
    383 
    384     def set_params(self, arg_params, aux_params, allow_extra=False):

/usr/local/lib/python2.7/dist-packages/mxnet/module/executor_group.pyc in bind_exec(self, data_shapes, label_shapes, shared_group, reshape)
    356             if reshape:
    357                 self.execs[i] = self._default_execs[i].reshape(
--> 358                     allow_up_sizing=True, **dict(data_shapes_i + label_shapes_i))
    359             else:
    360                 self.execs.append(self._bind_ith_exec(i, data_shapes_i, label_shapes_i,

/usr/local/lib/python2.7/dist-packages/mxnet/executor.pyc in reshape(self, partial_shaping, allow_up_sizing, **kwargs)
    428                     "This can cause the new executor to not share parameters " + \
    429                     "with the old one. Please check for error in network." +\
--> 430                     "If this is intended, set partial_shaping=True to suppress this warning.")
    431 
    432         new_aux_dict = {}

AssertionError: Shape of unspecified array arg:softmax_label changed. This can cause the new executor to not share parameters with the old one. Please check for error in network.If this is intended, set partial_shaping=True to suppress this warning.

With partial_shaping=True would follow the next error:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-5-16cd1c8e1a85> in <module>()
      7     img = np.swapaxes(img, 1, 2)
      8     img = img[np.newaxis, :]
----> 9     mod.forward(Batch(data = [mx.nd.array(img)]))
     10     exp = mod.get_outputs()[0].asnumpy()[0]
     11     angle = 180*exp

/usr/local/lib/python2.7/dist-packages/mxnet/module/module.pyc in forward(self, data_batch, is_train)
    610             self.reshape(new_dshape, new_lshape)
    611 
--> 612         self._exec_group.forward(data_batch, is_train)
    613 
    614     def backward(self, out_grads=None):

/usr/local/lib/python2.7/dist-packages/mxnet/module/executor_group.pyc in forward(self, data_batch, is_train)
    434 
    435         """
--> 436         _load_data(data_batch, self.data_arrays, self.data_layouts)
    437         if is_train is None:
    438             is_train = self.for_training

/usr/local/lib/python2.7/dist-packages/mxnet/module/executor_group.pyc in _load_data(batch, targets, major_axis)
     65 def _load_data(batch, targets, major_axis):
     66     """Load data into sliced arrays."""
---> 67     _load_general(batch.data, targets, major_axis)
     68 
     69 

/usr/local/lib/python2.7/dist-packages/mxnet/module/executor_group.pyc in _load_general(data, targets, major_axis)
     46                     if do_crop:
     47                         if axis == 0:
---> 48                             d_src[slice_idx.start:slice_idx.stop].copyto(d_dst)
     49                         else:
     50                             if d_src.context == d_dst.context:

/usr/local/lib/python2.7/dist-packages/mxnet/ndarray/ndarray.pyc in __getitem__(self, key)
    504         indexing_dispatch_code = _get_indexing_dispatch_code(key)
    505         if indexing_dispatch_code == _NDARRAY_BASIC_INDEXING:
--> 506             return self._get_nd_basic_indexing(key)
    507         elif indexing_dispatch_code == _NDARRAY_ADVANCED_INDEXING:
    508             return self._get_nd_advanced_indexing(key)

/usr/local/lib/python2.7/dist-packages/mxnet/ndarray/ndarray.pyc in _get_nd_basic_indexing(self, key)
    781                 return op.slice(self, begin=(key.start,), end=(key.stop,), step=(key.step,))
    782             elif key.start is not None or key.stop is not None:
--> 783                 return self._slice(key.start, key.stop)
    784             else:
    785                 return self

/usr/local/lib/python2.7/dist-packages/mxnet/ndarray/ndarray.pyc in _slice(self, start, stop)
    896         """
    897         handle = NDArrayHandle()
--> 898         start, stop, _ = _get_index_range(start, stop, self.shape[0])
    899 
    900         check_call(_LIB.MXNDArraySlice(

/usr/local/lib/python2.7/dist-packages/mxnet/ndarray/ndarray.pyc in _get_index_range(start, stop, length, step)
   2175             raise IndexError('Slicing stop %d exceeds limit of %d' % (stop-length, length))
   2176     elif stop > length:
-> 2177         raise IndexError('Slicing stop %d exceeds limit of %d' % (stop, length))
   2178 
   2179     return start, stop, step

IndexError: Slicing stop 16 exceeds limit of 1
´´´