When and where should use `detach()`?

1

I am working on training a GAN, where Image Pool is used to train the discriminator. But I got a weird error:

Check failed: type_ != nullptr: The any container is empty requested=N5mxnet10Imperative6AGInfoE

Any one can give me some advise ?

2

Can share the concerned code?

3

I am not sure whether your question in the subject is related to the error.
Detach removes the NDArray from the graph, so no gradients will be computed. In the context of GANs you need to do this because you update first the discriminator and then the generator. To compute the discriminator loss, you need to create fake data (by the generator). To not update generator twice, you need to detach this fake data from the graph. Here a small example:

with autograd.record():
   output_real,_,_ = discriminator(real_data)
   d_error_real    = loss1(output_real, real_label)
                
   # create fake image and input it to discriminator
   fake_image      = generator(g_input)
   output_fake = discriminator(fake_image.detach())
   d_error_fake    = loss1(output_fake, fake_label)
                
   # total discriminator error
   d_error         = d_error_real + d_error_fake
d_error.backward()
d_trainer.step(batch_size)

with autograd.record():
    fake_image = generator(g_input)
    output_fake, category_prob, continuous_mean = discriminator(fake_image)
    g_error = loss1(output_fake, real_label) 

g_error.backward()
4

Sure, @NRauschmayr posted exactly same code as mine ~

5

Thanks for your great explanations.

But is the graph constructed within with autograd.record() ? or discrimnator.hybridize() ? Or the truth is : discriminator.hybridize() will create a forward graph, but with autograd.record() create a backward graph, and the calculation of gradients and update of weights depend on the backward graph ?

I used

with autograd.record();
       output_real,_,_ = discriminator(real_data)
   d_error_real    = loss1(output_real, real_label)
                
   # create fake image and input it to discriminator
   with autograd.pause():
       fake_image      = generator(g_input)
   output_fake = discriminator(fake_image.detach())
   d_error_fake    = loss1(output_fake, fake_label)
                
   # total discriminator error
   d_error         = d_error_real + d_error_fake
   autograd.backward(d_error)
d_trainer.step(batch_size)
...

At last, is there any difference between autograd.backward(d_error) and d_error.backward() ?