Hi!
I noticed that the default data type (i.e., the one used when creating arrays without specifying the dtype
argument) in MXNet is float32
when I found my loss became nan
after several hundred iterations. Now I want to use float64
in stead of float32
to see the precision of the data type was responsible for the nan
. However, it seems to me that that the only way to specify float64
as the data type for every array is to explicitly add dtype='float64'
to every function call that creates an array from scratch. I wonder if there is a way to replace the default data type.
This reminds me of a similar question Shortcut for enabling GPU concerning context rather than dtype. The two questions put together are simply ‘Is there any way to change the default settings (ctx
and dtype
) for newly created arrays?’
Thanks!
UPD: I did not find a way to specify the data type of the convolutional layer parameters. Dense
and Embedding
both have dtype
in their constructors, but Conv2D
does not.
Below are excerpts from implementations of Dense
and Conv2D
where the parameters are defined:
Dense
:
self.weight = self.params.get('weight', shape=(units, in_units),
init=weight_initializer, dtype=dtype,
allow_deferred_init=True)
if use_bias:
self.bias = self.params.get('bias', shape=(units,),
init=bias_initializer, dtype=dtype,
allow_deferred_init=True)
else:
self.bias = None
Conv2D
(actually _Conv
, the superclass of Conv2D
):
self.weight = self.params.get('weight', shape=wshapes[1],
init=weight_initializer,
allow_deferred_init=True)
if use_bias:
self.bias = self.params.get('bias', shape=wshapes[2],
init=bias_initializer,
allow_deferred_init=True)
else:
self.bias = None
Note that the Dense
specifies the data type for its parameters, but Conv2D
does not, which in turn means that the parameters can only be of type float32
, since Parameter
has this default setting when dtype
is not specified.
UPD2: I found that Block.cast
is available: (https://mxnet.incubator.apache.org/api/python/gluon/gluon.html#mxnet.gluon.Block.cast)