Is there a way to allocate fixed amount of gpu memory for a program in mxnet (like tensorflow)? Currently, I have a training script which take days to run and its gpu memory usage fluctuates from time to time. When other users accidentally use the same gpu, my program will crash because of OOM.
There is currently a limited number of things you can do to manage GPU memory. Take a look into possible environment variables - https://mxnet.incubator.apache.org/faq/env_var.html#memory-options
If you are using gluon, you can try