Symbol block doesn't support row_sparse storage type


#1

I was using a Symbol block and had mx.symbol.Embedding layer which worked without any issues. When I changed it to sparse.Embedding I realized that SymbolBlock doesn’t seem to work with row_sparse.

        weight = mx.symbol.var("encoder_weight", stype='row_sparse')
        embed = mx.symbol.sparse.Embedding(data=data, weight=weight, input_dim=input_dim,                                    
                                                 output_dim=embedding_size, name='embed', sparse_grad=True)

From the code in gluon/block.py it looks like SymbolBlock is inherited from HybridBlock and which intern is from Block. None of them check parameter storage type.

Is there specific reason for SymbolBlock not supporting row_sparse storage type ?

        # check if any symbol is row_sparse
        row_sparse_storage = ndarray.ndarray._STORAGE_TYPE_STR_TO_ID['row_sparse']
        for i in out:
            for j in i.get_internals():
                assert(j.attr("__storage_type__") != str(row_sparse_storage)), \
                    "SymbolBlock doesn't support Parameter '%s' because its storage " \
                    "type is 'row_sparse'." % j.name

#2

@eric-haibin-lin could you shed some light here? Thanks!


#3

Sorry for the late reply, I’ve been traveling. @rrb0112 what network are you trying to run and why do you need a symbol block? You can probably use native gluon blocks such as nn.Embedding(sparse_grad=True) or contrib.nn.SparseEmbedding()