Default datatype


#1

Hi!
I noticed that the default data type (i.e., the one used when creating arrays without specifying the dtype argument) in MXNet is float32 when I found my loss became nan after several hundred iterations. Now I want to use float64 in stead of float32 to see the precision of the data type was responsible for the nan. However, it seems to me that that the only way to specify float64 as the data type for every array is to explicitly add dtype='float64' to every function call that creates an array from scratch. I wonder if there is a way to replace the default data type.
This reminds me of a similar question Shortcut for enabling GPU concerning context rather than dtype. The two questions put together are simply ‘Is there any way to change the default settings (ctx and dtype) for newly created arrays?’
Thanks!

UPD: I did not find a way to specify the data type of the convolutional layer parameters. Dense and Embedding both have dtype in their constructors, but Conv2D does not.

Below are excerpts from implementations of Dense and Conv2D where the parameters are defined:
Dense:

            self.weight = self.params.get('weight', shape=(units, in_units),
                                          init=weight_initializer, dtype=dtype,
                                          allow_deferred_init=True)
            if use_bias:
                self.bias = self.params.get('bias', shape=(units,),
                                            init=bias_initializer, dtype=dtype,
                                            allow_deferred_init=True)
            else:
                self.bias = None

Conv2D (actually _Conv, the superclass of Conv2D):

            self.weight = self.params.get('weight', shape=wshapes[1],
                                          init=weight_initializer,
                                          allow_deferred_init=True)
            if use_bias:
                self.bias = self.params.get('bias', shape=wshapes[2],
                                            init=bias_initializer,
                                            allow_deferred_init=True)
            else:
                self.bias = None

Note that the Dense specifies the data type for its parameters, but Conv2D does not, which in turn means that the parameters can only be of type float32, since Parameter has this default setting when dtype is not specified.

UPD2: I found that Block.cast is available: (https://mxnet.incubator.apache.org/api/python/gluon/gluon.html#mxnet.gluon.Block.cast)


#2

For your first question as you’ve answered your second, there are code points which explicitly set dtype=‘float32’ which means you won’t be able to set a default dtype via a global or the like. Note there is a mx_real_t in base.py but it isn’t referenced everywhere.

Vishaal