Installation of R Package Fails

Dear All,
I tried installing the R package for my box (debian stable) according to the instructions here

the first part i.e.

./install_mxnet_ubuntu_python.sh

completed without errors, but the second script

./install_mxnet_ubuntu_r.sh

failed. I enclose the final output of the first and the second script hoping that someone can help me out with this

Many thanks!
#############################################
Output of the python.sh script

[...]
-Wl,--whole-archive /home/lorenzo/mxnet/3rdparty/tvm/nnvm/lib/libnnvm.a -Wl,--no-whole-archive
a - build/src/io/iter_libsvm.o
a - build/src/common/utils.o
a - build/src/common/rtc.o
a - build/src/nnvm/gradient.o
a - build/src/nnvm/legacy_op_util.o
a - build/src/nnvm/tvm_bridge.o
a - build/src/nnvm/graph_editor.o
a - build/src/nnvm/legacy_json_util.o
a - build/src/nnvm/plan_memory.o
a - build/src/nnvm/low_precision_pass.o
a - build/src/nnvm/amp_infer_unknown.o
a - build/src/profiler/profiler.o
a - build/src/profiler/aggregate_stats.o
a - build/src/profiler/nvtx.o
a - build/src/profiler/vtune.o
a - build/src/imperative/cached_op.o
a - build/src/imperative/imperative_utils.o
a - build/src/imperative/imperative.o
a - build/src/ndarray/ndarray_function.o
a - build/src/ndarray/ndarray.o
a - build/src/operator/instance_norm.o
a - build/src/operator/subgraph_op_common.o
a - build/src/operator/grid_generator.o
a - build/src/operator/pooling_v1.o
a - build/src/operator/loss_binary_op.o
a - build/src/operator/sequence_reverse.o
a - build/src/operator/swapaxis.o
a - build/src/operator/rnn.o
a - build/src/operator/convolution_v1.o
a - build/src/operator/crop.o
a - build/src/operator/control_flow.o
a - build/src/operator/bilinear_sampler.o
a - build/src/operator/softmax_output.o
a - build/src/operator/roi_pooling.o
a - build/src/operator/sequence_last.o
a - build/src/operator/batch_norm_v1.o
a - build/src/operator/cross_device_copy.o
a - build/src/operator/svm_output.o
a - build/src/operator/regression_output.o
a - build/src/operator/l2_normalization.o
a - build/src/operator/leaky_relu.o
a - build/src/operator/c_lapack_api.o
a - build/src/operator/identity_attach_KL_sparse_reg.o
a - build/src/operator/make_loss.o
a - build/src/operator/operator.o
a - build/src/operator/operator_tune.o
a - build/src/operator/correlation.o
a - build/src/operator/operator_util.o
a - build/src/operator/spatial_transformer.o
a - build/src/operator/pad.o
a - build/src/operator/slice_channel.o
a - build/src/operator/sequence_mask.o
a - build/src/operator/optimizer_op.o
a - build/src/engine/naive_engine.o
a - build/src/engine/openmp.o
a - build/src/engine/threaded_engine_pooled.o
a - build/src/engine/threaded_engine.o
a - build/src/engine/engine.o
a - build/src/engine/threaded_engine_perdevice.o
a - build/src/storage/storage.o
a - build/src/c_api/c_api_executor.o
a - build/src/c_api/c_api_symbolic.o
a - build/src/c_api/c_api_profile.o
a - build/src/c_api/c_api_ndarray.o
a - build/src/c_api/c_api_test.o
a - build/src/c_api/c_predict_api.o
a - build/src/c_api/c_api_function.o
a - build/src/c_api/c_api.o
a - build/src/c_api/c_api_error.o
a - build/src/executor/inplace_addto_detect_pass.o
a - build/src/executor/graph_executor.o
a - build/src/executor/infer_graph_attr_pass.o
a - build/src/executor/attach_op_execs_pass.o
a - build/src/executor/attach_op_resource_pass.o
a - build/src/kvstore/gradient_compression.o
a - build/src/kvstore/kvstore_utils.o
a - build/src/kvstore/kvstore.o
a - build/src/resource.o
a - build/src/base.o
a - build/src/libinfo.o
a - build/src/initialize.o
g++ -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/home/lorenzo/mxnet/3rdparty/mshadow/ -I/home/lorenzo/mxnet/3rdparty/dmlc-core/include -fPIC -I/home/lorenzo/mxnet/3rdparty/tvm/nnvm/include -I/home/lorenzo/mxnet/3rdparty/dlpack/include -I/home/lorenzo/mxnet/3rdparty/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_F16C=0 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMSHADOW_INT64_TENSOR_SIZE=0 -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -DUSE_JEMALLOC  -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -std=c++11  -o bin/im2rec tools/im2rec.cc build/src/operator/contrib/tvmop/ufunc.o build/src/operator/subgraph/mkldnn/mkldnn_subgraph_property.o build/src/operator/subgraph/mkldnn/mkldnn_conv.o build/src/operator/subgraph/mkldnn/mkldnn_fc.o build/src/operator/subgraph/tensorrt/tensorrt.o build/src/operator/subgraph/tensorrt/onnx_to_tensorrt.o build/src/operator/subgraph/tensorrt/nnvm_to_onnx.o build/src/operator/quantization/mkldnn/mkldnn_quantized_elemwise_add.o build/src/operator/quantization/mkldnn/mkldnn_quantized_conv.o build/src/operator/quantization/mkldnn/mkldnn_quantized_act.o build/src/operator/quantization/mkldnn/mkldnn_quantized_fully_connected.o build/src/operator/quantization/mkldnn/mkldnn_quantized_pooling.o build/src/operator/quantization/mkldnn/mkldnn_quantized_concat.o build/src/operator/nn/mkldnn/mkldnn_convolution.o build/src/operator/nn/mkldnn/mkldnn_concat.o build/src/operator/nn/mkldnn/mkldnn_flatten.o build/src/operator/nn/mkldnn/mkldnn_base.o build/src/operator/nn/mkldnn/mkldnn_slice.o build/src/operator/nn/mkldnn/mkldnn_reshape.o build/src/operator/nn/mkldnn/mkldnn_transpose.o build/src/operator/nn/mkldnn/mkldnn_act.o build/src/operator/nn/mkldnn/mkldnn_softmax.o build/src/operator/nn/mkldnn/mkldnn_softmax_output.o build/src/operator/nn/mkldnn/mkldnn_deconvolution.o build/src/operator/nn/mkldnn/mkldnn_copy.o build/src/operator/nn/mkldnn/mkldnn_fully_connected.o build/src/operator/nn/mkldnn/mkldnn_pooling.o build/src/operator/nn/mkldnn/mkldnn_sum.o build/src/operator/nn/cudnn/cudnn_algoreg.o build/src/operator/nn/cudnn/cudnn_batch_norm.o build/src/operator/tensor/elemwise_binary_broadcast_op_basic.o build/src/operator/tensor/sparse_retain.o build/src/operator/tensor/square_sum.o build/src/operator/tensor/matrix_op.o build/src/operator/tensor/elemwise_binary_op_basic.o build/src/operator/tensor/broadcast_reduce_op_value.o build/src/operator/tensor/init_op.o build/src/operator/tensor/elemwise_binary_broadcast_op_extended.o build/src/operator/tensor/histogram.o build/src/operator/tensor/broadcast_reduce_op_index.o build/src/operator/tensor/dot.o build/src/operator/tensor/elemwise_scatter_op.o build/src/operator/tensor/elemwise_binary_scalar_op_extended.o build/src/operator/tensor/ordering_op.o build/src/operator/tensor/elemwise_unary_op_basic.o build/src/operator/tensor/ravel.o build/src/operator/tensor/amp_cast.o build/src/operator/tensor/control_flow_op.o build/src/operator/tensor/elemwise_binary_scalar_op_basic.o build/src/operator/tensor/indexing_op.o build/src/operator/tensor/elemwise_binary_broadcast_op_logic.o build/src/operator/tensor/la_op.o build/src/operator/tensor/diag_op.o build/src/operator/tensor/elemwise_binary_op_logic.o build/src/operator/tensor/elemwise_binary_op_extended.o build/src/operator/tensor/elemwise_sum.o build/src/operator/tensor/cast_storage.o build/src/operator/tensor/elemwise_binary_scalar_op_logic.o build/src/operator/tensor/elemwise_binary_op.o build/src/operator/tensor/elemwise_unary_op_trig.o build/src/operator/contrib/multibox_target.o build/src/operator/contrib/sync_batch_norm.o build/src/operator/contrib/dgl_graph.o build/src/operator/contrib/count_sketch.o build/src/operator/contrib/krprod.o build/src/operator/contrib/bilinear_resize.o build/src/operator/contrib/hawkes_ll.o build/src/operator/contrib/nnz.o build/src/operator/contrib/quadratic_op.o build/src/operator/contrib/roi_align.o build/src/operator/contrib/proposal.o build/src/operator/contrib/deformable_psroi_pooling.o build/src/operator/contrib/fft.o build/src/operator/contrib/multibox_prior.o build/src/operator/contrib/gradient_multiplier_op.o build/src/operator/contrib/adamw.o build/src/operator/contrib/transformer.o build/src/operator/contrib/all_finite.o build/src/operator/contrib/index_array.o build/src/operator/contrib/multi_proposal.o build/src/operator/contrib/index_copy.o build/src/operator/contrib/bounding_box.o build/src/operator/contrib/amp_graph_pass.o build/src/operator/contrib/boolean_mask.o build/src/operator/contrib/psroi_pooling.o build/src/operator/contrib/deformable_convolution.o build/src/operator/contrib/ifft.o build/src/operator/contrib/multibox_detection.o build/src/operator/contrib/optimizer_op.o build/src/operator/contrib/adaptive_avg_pooling.o build/src/operator/custom/native_op.o build/src/operator/custom/ndarray_op.o build/src/operator/custom/custom.o build/src/operator/image/crop.o build/src/operator/image/image_random.o build/src/operator/image/resize.o build/src/operator/nnpack/nnpack_util.o build/src/operator/tvmop/op_module.o build/src/operator/random/sample_multinomial_op.o build/src/operator/random/pdf_op.o build/src/operator/random/multisample_op.o build/src/operator/random/unique_sample_op.o build/src/operator/random/sample_op.o build/src/operator/random/shuffle_op.o build/src/operator/subgraph/build_subgraph.o build/src/operator/subgraph/default_subgraph_property_v2.o build/src/operator/subgraph/default_subgraph_property.o build/src/operator/quantization/quantized_elemwise_add.o build/src/operator/quantization/dequantize.o build/src/operator/quantization/quantize_graph_pass.o build/src/operator/quantization/quantized_flatten.o build/src/operator/quantization/quantized_conv.o build/src/operator/quantization/quantized_fully_connected.o build/src/operator/quantization/quantized_pooling.o build/src/operator/quantization/quantize_v2.o build/src/operator/quantization/quantized_concat.o build/src/operator/quantization/requantize.o build/src/operator/quantization/quantized_activation.o build/src/operator/quantization/quantize.o build/src/operator/nn/moments.o build/src/operator/nn/softmax.o build/src/operator/nn/group_norm.o build/src/operator/nn/pooling.o build/src/operator/nn/deconvolution.o build/src/operator/nn/activation.o build/src/operator/nn/upsampling.o build/src/operator/nn/ctc_loss.o build/src/operator/nn/fully_connected.o build/src/operator/nn/convolution.o build/src/operator/nn/lrn.o build/src/operator/nn/layer_norm.o build/src/operator/nn/concat.o build/src/operator/nn/softmax_activation.o build/src/operator/nn/batch_norm.o build/src/operator/nn/dropout.o build/src/io/io.o build/src/io/image_aug_default.o build/src/io/iter_image_det_recordio.o build/src/io/image_io.o build/src/io/image_det_aug_default.o build/src/io/iter_csv.o build/src/io/iter_image_recordio.o build/src/io/iter_mnist.o build/src/io/iter_image_recordio_2.o build/src/io/iter_libsvm.o build/src/common/utils.o build/src/common/rtc.o build/src/nnvm/gradient.o build/src/nnvm/legacy_op_util.o build/src/nnvm/tvm_bridge.o build/src/nnvm/graph_editor.o build/src/nnvm/legacy_json_util.o build/src/nnvm/plan_memory.o build/src/nnvm/low_precision_pass.o build/src/nnvm/amp_infer_unknown.o build/src/profiler/profiler.o build/src/profiler/aggregate_stats.o build/src/profiler/nvtx.o build/src/profiler/vtune.o build/src/imperative/cached_op.o build/src/imperative/imperative_utils.o build/src/imperative/imperative.o build/src/ndarray/ndarray_function.o build/src/ndarray/ndarray.o build/src/operator/instance_norm.o build/src/operator/subgraph_op_common.o build/src/operator/grid_generator.o build/src/operator/pooling_v1.o build/src/operator/loss_binary_op.o build/src/operator/sequence_reverse.o build/src/operator/swapaxis.o build/src/operator/rnn.o build/src/operator/convolution_v1.o build/src/operator/crop.o build/src/operator/control_flow.o build/src/operator/bilinear_sampler.o build/src/operator/softmax_output.o build/src/operator/roi_pooling.o build/src/operator/sequence_last.o build/src/operator/batch_norm_v1.o build/src/operator/cross_device_copy.o build/src/operator/svm_output.o build/src/operator/regression_output.o build/src/operator/l2_normalization.o build/src/operator/leaky_relu.o build/src/operator/c_lapack_api.o build/src/operator/identity_attach_KL_sparse_reg.o build/src/operator/make_loss.o build/src/operator/operator.o build/src/operator/operator_tune.o build/src/operator/correlation.o build/src/operator/operator_util.o build/src/operator/spatial_transformer.o build/src/operator/pad.o build/src/operator/slice_channel.o build/src/operator/sequence_mask.o build/src/operator/optimizer_op.o build/src/engine/naive_engine.o build/src/engine/openmp.o build/src/engine/threaded_engine_pooled.o build/src/engine/threaded_engine.o build/src/engine/engine.o build/src/engine/threaded_engine_perdevice.o build/src/storage/storage.o build/src/c_api/c_api_executor.o build/src/c_api/c_api_symbolic.o build/src/c_api/c_api_profile.o build/src/c_api/c_api_ndarray.o build/src/c_api/c_api_test.o build/src/c_api/c_predict_api.o build/src/c_api/c_api_function.o build/src/c_api/c_api.o build/src/c_api/c_api_error.o build/src/executor/inplace_addto_detect_pass.o build/src/executor/graph_executor.o build/src/executor/infer_graph_attr_pass.o build/src/executor/attach_op_execs_pass.o build/src/executor/attach_op_resource_pass.o build/src/kvstore/gradient_compression.o build/src/kvstore/kvstore_utils.o build/src/kvstore/kvstore.o build/src/resource.o build/src/base.o build/src/libinfo.o build/src/initialize.o /home/lorenzo/mxnet/3rdparty/dmlc-core/libdmlc.a /home/lorenzo/mxnet/3rdparty/tvm/nnvm/lib/libnnvm.a -pthread -lm -lopenblas -fopenmp -lrt  -lopencv_highgui -lopencv_imgcodecs -lopencv_imgproc -lopencv_core /usr/lib/x86_64-linux-gnu/libjemalloc.so 

and output of the R.sh script

[...]
      ^~~~~~~
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I../inst/include -I"/usr/local/lib/R/site-library/Rcpp/include"   -fpic  -g -O2 -fdebug-prefix-map=/home/jranke/git/r-backports/buster/r-base-3.6.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c symbol.cc -o symbol.o
In file included from symbol.cc:28:
./symbol.h: In destructor ‘mxnet::R::Symbol::~Symbol()’:
./base.h:102:45: warning: throw will always call terminate() [-Wterminate]
       throw Rcpp::exception(MXGetLastError());                     \
                                             ^
./symbol.h:138:5: note: in expansion of macro ‘MX_CALL’
     MX_CALL(MXSymbolFree(handle_));
     ^~~~~~~
./base.h:102:45: note: in C++11 destructors default to noexcept
       throw Rcpp::exception(MXGetLastError());                     \
                                             ^
./symbol.h:138:5: note: in expansion of macro ‘MX_CALL’
     MX_CALL(MXSymbolFree(handle_));
     ^~~~~~~
g++ -std=gnu++11 -shared -L/usr/lib/R/lib -Wl,-z,relro -o mxnet.so executor.o export.o im2rec.o io.o kvstore.o mxnet.o ndarray.o symbol.o -llapack -lblas -L/usr/lib/R/lib -lR
make[1]: Leaving directory '/home/lorenzo/mxnet/R-package/src'
installing to /usr/local/lib/R/site-library/00LOCK-R-package/00new/mxnet/libs
** R
** demo
** inst
** byte-compile and prepare package for lazy loading
Note: ... may be used in an incorrect context 
** help
No man pages found in package  ‘mxnet’ 
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
[1] "Loading local: inst/libs/libmxnet.so"
Error: package or namespace load failed for ‘mxnet’:
 .onLoad failed in loadNamespace() for 'mxnet', details:
  call: dyn.load("R-package/inst/libs/libmxnet.so", local = FALSE)
  error: unable to load shared object '/home/lorenzo/mxnet/R-package/R-package/inst/libs/libmxnet.so':
  /home/lorenzo/mxnet/R-package/R-package/inst/libs/libmxnet.so: cannot open shared object file: No such file or directory
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/usr/local/lib/R/site-library/mxnet’
make: *** [Makefile:701: rpkg] Error 1

I have just tried to install MXNet CPU version with R on a fresh EC2 instance, and it worked.

From this line I assume that there something went wrong with the installation of MXNet itself. The R installation part cannot find libmxnet.so, which is the main MXNet file. I am also surprised how small your ./install_mxnet_ubuntu_python.sh output is. In my case it was pages and it took about half an hour or so. Could you locate libmxnet.so manually? If not, then it is definitely not installed, and you need to try again.

When I was installing R part, I have noticed that MXNet requires R of version >= 3.4.4. I followed the instructions from here https://askubuntu.com/a/436491/353902 to obtain it, as by default I could install only 3.2 version.

Hello,
And thanks.
I have R 3.6.1 installed.
I did not paste all the output of `./install_mxnet_ubuntu_python.sh, but just the end to show that there is no error message.
I have the files

~$ ls mxnet/lib/libmxnet.so 
-rwxr-xr-x 1 lorenzo lorenzo 71740832 Jul 24 17:25 mxnet/lib/libmxnet.so

~$ ls mxnet/R-package/inst/libs/libmxnet.so 
-rwxr-xr-x 1 root root 71740832 Jul 24 21:57 mxnet/R-package/inst/libs/libmxnet.so

So the library gets built, but for some reason R looks for it in the wrong place

/home/lorenzo/mxnet/R-package/R-package/inst/libs/libmxnet.so

which does not exist (I do not know why it nests R-package inside another R-package directory).
Any idea about how to fix this?

It’s not actually looking in the wrong place, that’s just a fallback attempt.
It tries the real location which presumably fails and then it tries again inside R-package/R-package.
Look at the source code in R/zzz.R to see what’s happening.
Sorry that doesn’t explain why it can’t load inst/libs/libmxnet.so in the first place.
But the nested R-package is a red herring :wink: