loss not decreasing in AutoModelForQuestionAnswering (Transformers)

[mmgxa]

I am training my Model on SQUAD dataset. I get the same problem for either distilbert-base-uncased or bert-base-uncased checkpoint - the loss does not decrease.

In my forward method, I have this line

output = self.model(input_ids, attention_mask=attention_mask, start_positions=start_positions, end_positions=end_positions)

loss = output.loss
start_scores = output.start_logits
end_scores = output.end_logits

This should work for backpropagation, right?

Also, it gives the following warning

LightningDeprecationWarning: One of the returned values {'end_scores', 'start_scores'} has a grad_fn. We will detach it automatically but this behaviour will change in v1.6. Please detach it manually: return {'loss': ..., 'something': something.detach()} f"One of the returned values {set(extra.keys())} has a grad_fn. We will detach it automatically"

Is this affecting the loss/backprop?

[justusschock]

How does your training_step look like?

[mmgxa]

This is my training_step

def training_step(self, batch, batch_idx):
    input_ids = batch['input_ids']
    attention_mask = batch['attention_mask']
    start_positions = batch['start_positions']
    end_positions = batch['end_positions']
    output = self(input_ids, attention_mask=attention_mask, start_positions=start_positions, end_positions=end_positions) 
    
    loss = output.loss
    start_scores = output.start_logits
    end_scores = output.end_logits
    
    return {"loss": loss, 'start_scores': start_scores, 'end_scores':end_scores}
    ```

[justusschock]

can you try to only return the loss there?

[mmgxa]

Did it

return loss

No effect on the loss. But the following message disappeared!

LightningDeprecationWarning: One of the returned values {'end_scores', 'start_scores'} has a grad_fn. We will detach it automatically but this behaviour will change in v1.6. ...

[justusschock]

That's strange. Do you have optimizers setup that also hold your model's parameters?

[tchaton]

Dear @mmgxa,

Mind sharing a reproducible script with the trainer arguments you are using and make sure to update to latest PyTorch Lightning version ?

Best,
T.C

[mmgxa]

Here ...

https://colab.research.google.com/drive/13g4Onz3nUPVlG-r3tkM7WDtqtI_VNUgQ?usp=sharing

Actually it does train, but ever so slightly. The learning rate (lr=2e-5) seems to work with other models, but not for this one.