Batch_dot one hot vectors with embeddings results in nan

#1

Hi there,

I have a one hot tensor like this:


First dim is batch size, second dim is char, third is one hot vector. One batch looks like this:

And an embedding tensor like this:


Each element is in a certain range:

Then I use char_embed = nd.batch_dot(one_hot, bert_embed) to lookup my embedding, the result contains nan which confuses me for a whole day.


If there is no nan in bert_embed, why is nan produced when it is batch_dot with a one hot tensor?

#2

The bug only occurs on CPU:

src/operator/nn/./fully_connected-inl.h:200: float16 fully connected layer is currentlyonly supported by CuDNN version.

Well, it works fine on GPU. So, not a big deal.