How to ensure the sequences of mutating the same variable


During the training process, the communication operations include the push() and pull() operations. They are dispatched through the _update_params_on_kvstore function, and each push is followed by a pull. To execute the push and pull operations, both of them will change the value of comm_buf_ in kvstore_dist.h, and push will read comm_buf_[key] and pull will mutate comm_buf_[key]. However, how their sequences are ensured? Because push operation is dispatched before pull operation? Other operations like CopyFromTo(), comm_->Broadcast() and so on, if we want to access the same variable, what should we do ensure the right execution sequence? Is it feasible to send operations in the order in which the variable is accessed?