Is it possible to implement my own retry strategy?

[shvedovvladimir]

I've tried sending notifications using firebase-admin. On my server side, there were network problems and I got a response from firebase with a 503 code (or some like), but messages were sent to devices anyway. According to the default retry strategy, my server resubmitted the request. As a result, I received 2 push notifications.
I want to set retry attempts lower. How can I fix that?

[google-oss-bot]

I found a few problems with this issue:

• I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.
• This issue does not seem to follow the issue template. Make sure you provide all the required information.

[hiranya911]

The problem is FCM sending back 503 errors for some notifications, when they were actually delivered. It should be possible to safely retry a request on 503. @chong-shao WDYT?

[chong-shao]

We acknowledge that the increase on 503 is a backend problem. We are currently prioritizing resolving it.

Let's keep this thread open, I'll update once the backend problem is resolved and we can look at the retry mechanism but I guess at that time there will be a very small number of 503s.

[cati-net]

We've opened up a support ticket with you regarding this issue, as we have experienced it when sending to 100-1000 batches of 200-500 tokens each. In some cases around 500 of our batches reaches a timeout and are getting several duplicated notifications.

What we've seen is that when a batch reaches the internal 10 second limit (seen here

firebase-admin-node/src/messaging/batch-request-internal.ts

Line 23 in c6e9ef7

const TEN_SECONDS_IN_MILLIS = 10000;

) a retry is done, even though the notification has been sent out.

Our work-around at the moment is to remove ETIMEDOUT from ioErrorCodes in the defaultRetryConfig.
Though if some of our users in the batch are not getting the notification because of our "no retry on timed out"-policy, a better work-around would be appreciated. Would increasing the time-out be better than our solution right now? :)

[hiranya911]

@cati-net yours issue is different from the one discussed in this thread. We can look into increasing the timeout as a potential resolution to your problem. But do you know what causes this many requests to timeout? Have you ruled out any possible network or hardware issues?

[chong-shao]

Hi @shvedovvladimir our backend has recovered and the error rate have dropped. Is this duplicated delivery problem still happening to you?

[shvedovvladimir]

Hi @shvedovvladimir our backend has recovered and the error rate have dropped. Is this duplicated delivery problem still happening to you?

We reduced to 100 the number of sent messages in the request (on next day), and the problem did not occur again

[chong-shao]

Thanks for the reply! I'll go ahead and close this issue. Please let me know if you have other questions.