Custom dataset similar to Python generator?

Hi!

I am coming from Keras/Tensorflow and would like to move to mxnet. So far it looks awesome.

I am however wondering if there is any possibility to set up a custom dataset where the data is generated similar to a Python generator? I am working on time series data and reading it all into memory and storing it in a 2D array is not an option. In Tensorflow this is possible: https://www.tensorflow.org/guide/data#consuming_python_generators

Somewhere I read that using a custom dataset and dataloader this should be no problem? The API will not read all the data into memory at once.

Finally, I found this post https://guertl.me/post/183427090170/custom-dataset-and-dataloader-in-gluon on how to create custom datasets. Could you recommend me any other tutorials?

This is my first post so please let me know if I have done something wrong. Thanks!

If you search in the forum you’ll find several discussions about custom datasets. This may also help:

1 Like

Thanks! I’ll check it out. :slight_smile:

1 Like

Nice to read about the Custom dataset similar to Python generator.
If I already know what a generator function is, and want to learn about how to use it, to create a custom data generator given link by you can proceed. However, if I don’t know what a generator function is, or want a refresher, quickly go through here: https://www.cetpainfotech.com/technology/python-training