I think I’ve discovered a discrepancy in the behavior of mxnet.random.shuffle() on OSX and Ubuntu. Both systems are running mxnet 1.3.1.
Minimal reproducible example:
import random as rnd
import numpy as np
import mxnet as mx
import mxnet.ndarray as nd
rnd.seed(1)
np.random.seed(1)
mx.random.seed(1)
mx.random.shuffle(nd.arange(10))
This can have serious repercussions for SGD if you use shuffle() to shuffle the data. If results are not reproducible across operating systems, that would be bad.
This is not a bug. You cannot rely on the random seed being the same on different machines, given that the underlying random number generators may be different.
Random number generators in MXNet are device specific.
`mx.random.seed(seed_state)` sets the state of each generator using `seed_state` and the
device id. Therefore, random numbers generated from different devices can be different
even if they are seeded using the same seed.
Note the seed in MXNet is generated with the device id as well as the seed state in contrast to Numpy’s random seed behaviour which an identical seed results in identical random numbers.
I think the documentation means that:
mx.random.seed(0) may generates different numbers on GPU(0) and GPU(1). That’s fine. But if I change the platform and compare the result on GPU(0), they are expected to be consistent. No?