How to use my own images for a CNN with MXNet (ImageRecordIter)?


#1

Hello,

I’ve been stuck for the past few days and don’t really understand where exactly my mistake is.

I basically want to train a CNN with my own images in Python. I’ve been following this tutorial, except for the MNIST dataset.

My images have either a positive or negative label, and I’ve arranged the folders in that way as well. I’ve done as the MXNet IO API said and created three .lst files with im2rec.py (train/test/validation). With those I created the respective .rec and .idx files.

However, I still can’t figure out how to actually use them in my CNN. I’ve had several approaches so far, currently I’m using ImageRecordIter like this:

train_iter = ImageRecordIter(
path_imgrec = ‘./test_train.rec’,
data_shape = (3, 227, 227),
path_imglist = ‘./test_train.lst’,
path_imgidx = ‘./test_train.idx’,
batch_size = 3,
shuffle = True
)

Same for validation and test sets, of course. At first I got an Error at those lines:

for i, (data, label) in enumerate(train_data):
data = data.as_in_context(ctx)
label = label.as_in_context(ctx)

TypeError: ‘DataBatch’ object is not iterable

Error, which I fixed like proposed here.

for i, batch in enumerate(data_iterator):
data = batch.data[0].as_in_context(ctx)
label = batch.label[0].as_in_context(ctx)

Which allowed me to actually run the program, however this is the result I’m getting:

Epoch 0. Loss: 0.0, Train_acc nan, Test_acc nan
Epoch 1. Loss: 0.0, Train_acc nan, Test_acc nan
Epoch 2. Loss: 0.0, Train_acc nan, Test_acc nan
Epoch 3. Loss: 0.0, Train_acc nan, Test_acc nan
Epoch 4. Loss: 0.0, Train_acc nan, Test_acc nan

I do recognize that the error probably is that I’m not correctly iterating through the data, as

def evaluate_accuracy(data_iterator, net):

simply skips over the for loop. Do I have to reset the iterator during every epoch? However, I have no idea how to properly do it and have been stuck for the past few days because of this. I’ve also tried using ImageRecordDataset, RecordFileDataset and the DataLoader in combination with any of these, and couldn’t get it to work. Maybe I’ve even made a mistake at some other step that I don’t know yet. I’d really appreciate any help or pointers.


#2

You could consider using ImageFolderDataset() which is written for Gluon specifically. But using record files should work as well. Some things to check:

  • Do the record files actually contain anything (does the file size make sense)?
  • If you simply iterate through the training data, what does it contain?
  • Try showing the dataset as an image. Does it look OK?