RuntimeError: Trying to backward through the graph a second time

[Keiku]

I'm migrating my repository to pytorch-lightning and I get the following error:

RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling .backward() or autograd.grad() the first time.

The CNNLSTM model seems to be the problem, what should I do?

[My repository]
Keiku/Action-Recognition-CNN-LSTM: Action recognition tutorial using UCF-101 dataset. https://github.com/Keiku/Action-Recognition-CNN-LSTM

[ethanwharris]

Hi @Keiku This error happens if you try to call backward on something twice in a row without calling optimizer.step. Are you able to share your LightningModule code? It looks like the code in your repo just uses vanilla pytorch.

Thanks 😃

[Keiku]

@ethanwharris I tried upgrading to the latest version, but the error is still the same. Could you tell me your environment?

[Keiku]

@ethanwharris @tchaton I added the following to the forward part and it worked. I would like to verify once that the result is reproduced.

        # Reset LSTM hidden state
        self.model.lstm.reset_hidden_state()

[tchaton]

Hey @Keiku,

I don't have access to the repo anymore, but if you are using Lightning built-in tbptt, you would need to pass hiddens from the training_step and Lightning will detach automatically, preventing the double backward.

Best,
T.C

[ethanwharris]

Hi @Keiku, I re-ran the code using more than one step and do now observe the error. I suspect that clearning the hidden state isn't what you want here as your hidden state input to the LSTM will always be zero. You could either create a model which takes the full sequence in each batch and passes it through the LSTM, or use truncated backprop through time following the documentation here: https://pytorch-lightning.readthedocs.io/en/stable/advanced/sequences.html?highlight=truncated#truncated-backpropagation-through-time

Thanks 😃

[Keiku]

@ethanwharris It seems that the original implementation was already resetting the hidden state. I will consider if it can be fixed using TBTT. Thank you for your references.

https://github.com/eriklindernoren/Action-Recognition/blob/b43ec097ab293ac8d83aac02b5addd27686ebe8c/train.py#L125