Cryptic failure of SSD training with gluoncv 0.5.0

Gluon
1

Hi,
I’m using gluoncv code to train a custom detector.
Net is a net = gcv.model_zoo.get_model('ssd_512_resnet50_v1_coco', pretrained=True)

data loader:

with autograd.train_mode():
    _, _, anchors = net(mx.nd.zeros((1, 3, image_size, image_size)))
train_transform = SSDDefaultTrainTransform(image_size, image_size, anchors)
batchify_fn = Tuple(Stack(), Stack(), Stack())  # stack image, cls_targets, box_targets
train_data = gluon.data.DataLoader(train_dataset.transform(train_transform), batch_size, True, batchify_fn=batchify_fn, last_batch='rollover', num_workers=num_workers)

val loader:

val_transform = SSDDefaultValTransform(image_size, image_size)
batchify_fn = Tuple(Stack(), Pad(pad_val=-1))
val_data = gluon.data.DataLoader(validation_dataset.transform(val_transform), batch_size, False, batchify_fn=batchify_fn, last_batch='keep', num_workers=num_workers)

loss:

mbox_loss = gcv.loss.SSDMultiBoxLoss()
ce_metric = mx.metric.Loss('CrossEntropy')
smoothl1_metric = mx.metric.Loss('SmoothL1')

training loop is quite conventional gluon stuff. Training with gluoncv 0.4.0 works fine; training with gluoncv 0.5.0 returns some cryptic error:

[Epoch 0][Batch 0], Speed: 0.954 samples/sec, CrossEntropy=7.652, SmoothL1=3.561
[Epoch 0][Batch 20], Speed: 53.380 samples/sec, CrossEntropy=3.742, SmoothL1=2.974
[Epoch 0][Batch 40], Speed: 70.256 samples/sec, CrossEntropy=3.276, SmoothL1=2.688

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<timed exec> in <module>()

<ipython-input-29-7f581d9451e5> in validate(net, val_data, ctx, classes, size)
     20             gt_bboxes.append(y.slice_axis(axis=-1, begin=0, end=4))
     21 
---> 22             metric.update(det_bboxes, det_ids, det_scores, gt_bboxes, gt_ids, [None])
     23     return metric.get()

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/gluoncv/utils/metrics/voc_detection.py in update(self, pred_bboxes, pred_labels, pred_scores, gt_bboxes, gt_labels, gt_difficults)
    107 
    108         if isinstance(gt_labels, list):
--> 109             if len(gt_difficults) * gt_difficults[0].shape[0] != \
    110                     len(gt_labels) * gt_labels[0].shape[0]:
    111                 gt_difficults = [None] * len(gt_labels) * gt_labels[0].shape[0]

AttributeError: 'NoneType' object has no attribute 'shape'
2

Is it possible that you have examples with 0 targets ? I think that’s likely the culprit here.