[ScheduleMessage] Async deliver message will resend forever when service not avaliable

[HScarb]

BUG REPORT

1. Please describe the issue you observed:

• What did you do (The steps to reproduce)?

Send schedule messages to dledger cluster and cause message resend.
Then change master to slave.

• What is expected to see?

Message stop resending

• What did you see instead?

Message still resending while the store is unavailable

2. Please tell us about your environment:

4.9.3 DLedger cluster

3. Other information (e.g. detailed explanation, logs, related issues, suggestions on how to fix, etc):

[dongeforever]

@HScarb Skip the message will cause message loss.

The root cause of this problem is the infinite recursion of handleResult().

PutResultProcess.thenProcess is like the below:

            this.future.thenAccept(result -> {
                this.handleResult(result);
            });

The core problem is when to call the "thenAccept ".

If the "thenAccept" is called by the background thread(for example, in dledger mode, by the QuorumAckChecker) and get an error, the handleResult will resend the message and get a new future.

As the future needs to be completed one by one.
At this time, the old future will depend on the new future to complete, and the new future depend on the old future too.

It will cause a deadlock.

@Git-Yang @duhengforever Please help a check.

The suggested solution is to use a new thread to execute the callback instead of in the "ack thread"(GroupCommitService or QuorumAckChecker).

[Git-Yang]

@HScarb Skip the message will cause message loss.

The root cause of this problem is the infinite recursion of handleResult().

PutResultProcess.thenProcess is like the below:

            this.future.thenAccept(result -> {
                this.handleResult(result);
            });

The core problem is when to call the "thenAccept ".

If the "thenAccept" is called by the background thread(for example, in dledger mode, by the QuorumAckChecker) and get an error, the handleResult will resend the message and get a new future.

As the future needs to be completed one by one. At this time, the old future will depend on the new future to complete, and the new future depend on the old future too.

It will cause a deadlock.

@Git-Yang @duhengforever Please help a check.

The suggested solution is to use a new thread to execute the callback instead of in the "ack thread"(GroupCommitService or QuorumAckChecker).

The handleResult will resend the message, bug not get a new future. Resending is via putMessage(), so Futrues are not infinitely nested.
I think, to avoid infinite recursion, the retry logic is no longer executed in onException() , but in HandlePutResultTask .

[dongeforever]

Yeah, Futures are not infinitely nested. The putMessage will create a future internally, and the newly created future will timeout and resend again.

It seems ok to do the retry in HandlePutResultTask.