Is autograd thread safe?


#1

To be more concrete about the question, suppose I have two treads running, thread 1 and thread 2. Thread 1 calls with autograd.record() which happens first. Then, thread 2 calls with autograd.pause() while thread one is still executing code in its record block.

Is that safe, or will the code being exectued in the record block now be detached because thread 2 called a pause before it was finished?

Additionally, if it’s not safe, is there a pattern recommended for managing multiple threads with autograd use? Or are there plans to make it thread safe in the future?


#2

Autograd is not thread safe, so I would discourage running autograd in 2 separate threads.

Alternative way would be to use separate processes, so each autograd would work in a separate process, and not cause any issues.

There aren’t any plans to change it, because it would require a lot of efforts.


#3

Okay, thanks for the confirmation!