When doing single-node, multi-gpu training in gluon, such as here https://gluon.mxnet.io/chapter07_distributed-learning/multiple-gpus-gluon.html what is the parameter update mode? async? sync?
As far as I know, when you do multigpu training on a single host, parameter updates is always synchronized, and you cannot really change it. In multigpu with single host mode you can effectively set only
local option for kvstore depending on where the gradients are going to be stored - GPUs or CPU.
By synchronous parameter update I mean, that kvstore would accumulate gradients from all GPUs first, and then do parameter updates and distribute new values for parameters across GPUs. Only then GPUs will start to process the next batch.