dsync: race condition where done/finish is received after one side closes #10
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…s closed We do not tell the remote we are closing if they have already told us because they close the connection after sending ITEM_FINISH or ITEM_DONE and will not be ever receive anything else from us unless it just happens to get combined into the same packet as a previous response and is already in the buffer.
|
Merged. |
DovecotSync
pushed a commit
that referenced
this pull request
Jul 9, 2019
This fixes a race condition where the http_client_host_shared_idle_timeout()
function would get called with an already freed hshared argument.
Specifically, the situation arises from the hshared idle timeout calling
http_client_host_shared_free(), which removes the timeout and then proceeds to
free the client queue. The client queue freeing code indirectly calls
http_client_host_shared_check_idle(), which notices that there is no idle
timeout and allocates one.
The backtrace at the point of this new timeout allocation:
frame #3: 0x00007f0c775897f0 libdovecot.so.0`timeout_add_to(...) ioloop.c:280
frame #4: 0x00007f0c7751a45f libdovecot.so.0`http_client_host_shared_check_idle(hshared=<unavailable>) at http-client-host.c:69
frame #5: 0x00007f0c7750de89 libdovecot.so.0`http_client_request_error(_req=<unavailable>, status=9000, error="") at http-client-request.c:1525
frame #6: 0x00007f0c77517f38 libdovecot.so.0`http_client_queue_fail_full(queue=0x000055e13cff0e10, status=9000, error="", all=<unavailable>) at http-client-queue.c:183
frame #7: 0x00007f0c77518baa libdovecot.so.0`http_client_queue_free(queue=0x000055e13cff0e10) at http-client-queue.c:141
frame #8: 0x00007f0c7751a8bc libdovecot.so.0`http_client_host_free_shared(_host=<unavailable>) at http-client-host.c:391
frame #9: 0x00007f0c7751ab4c libdovecot.so.0`http_client_host_shared_free(_hshared=0x00007ffdac109e48) at http-client-host.c:294
frame #10: 0x00007f0c7751ace8 libdovecot.so.0`http_client_host_shared_idle_timeout(hshared=<unavailable>) at http-client-host.c:40
frame #11: 0x00007f0c7758a1a4 libdovecot.so.0`io_loop_handle_timeouts at ioloop.c:682
frame #12: 0x00007f0c7758a089 libdovecot.so.0`io_loop_handle_timeouts(ioloop=0x000055e13cfc8d80) at ioloop.c:696
frame #13: 0x00007f0c7758befc libdovecot.so.0`io_loop_handler_run_internal(ioloop=0x000055e13cfc8d80) at ioloop-select.c:126
frame #14: 0x00007f0c7758a56d libdovecot.so.0`io_loop_handler_run(ioloop=<unavailable>) at ioloop.c:767
frame #15: 0x00007f0c7758a798 libdovecot.so.0`io_loop_run(ioloop=0x000055e13cfc8d80) at ioloop.c:740
frame #16: 0x00007f0c774f61eb libdovecot.so.0`master_service_run(service=0x000055e13cfc8c10, callback=<unavailable>) at master-service.c:782
frame #17: 0x000055e13b48e3a5 stats`main(argc=<unavailable>, argv=<unavailable>) at main.c:99
frame #18: 0x00007f0c771092e1 libc.so.6`__libc_start_main + 241
frame #19: 0x000055e13b48e41a stats`_start + 42
DovecotSync
pushed a commit
that referenced
this pull request
Jul 9, 2019
This fixes a race condition where the http_client_host_shared_idle_timeout()
function would get called with an already freed hshared argument.
Specifically, the situation arises from the hshared idle timeout calling
http_client_host_shared_free(), which removes the timeout and then proceeds to
free the client queue. The client queue freeing code indirectly calls
http_client_host_shared_check_idle(), which notices that there is no idle
timeout and allocates one.
The backtrace at the point of this new timeout allocation:
frame #3: 0x00007f0c775897f0 libdovecot.so.0`timeout_add_to(...) ioloop.c:280
frame #4: 0x00007f0c7751a45f libdovecot.so.0`http_client_host_shared_check_idle(hshared=<unavailable>) at http-client-host.c:69
frame #5: 0x00007f0c7750de89 libdovecot.so.0`http_client_request_error(_req=<unavailable>, status=9000, error="") at http-client-request.c:1525
frame #6: 0x00007f0c77517f38 libdovecot.so.0`http_client_queue_fail_full(queue=0x000055e13cff0e10, status=9000, error="", all=<unavailable>) at http-client-queue.c:183
frame #7: 0x00007f0c77518baa libdovecot.so.0`http_client_queue_free(queue=0x000055e13cff0e10) at http-client-queue.c:141
frame #8: 0x00007f0c7751a8bc libdovecot.so.0`http_client_host_free_shared(_host=<unavailable>) at http-client-host.c:391
frame #9: 0x00007f0c7751ab4c libdovecot.so.0`http_client_host_shared_free(_hshared=0x00007ffdac109e48) at http-client-host.c:294
frame #10: 0x00007f0c7751ace8 libdovecot.so.0`http_client_host_shared_idle_timeout(hshared=<unavailable>) at http-client-host.c:40
frame #11: 0x00007f0c7758a1a4 libdovecot.so.0`io_loop_handle_timeouts at ioloop.c:682
frame #12: 0x00007f0c7758a089 libdovecot.so.0`io_loop_handle_timeouts(ioloop=0x000055e13cfc8d80) at ioloop.c:696
frame #13: 0x00007f0c7758befc libdovecot.so.0`io_loop_handler_run_internal(ioloop=0x000055e13cfc8d80) at ioloop-select.c:126
frame #14: 0x00007f0c7758a56d libdovecot.so.0`io_loop_handler_run(ioloop=<unavailable>) at ioloop.c:767
frame #15: 0x00007f0c7758a798 libdovecot.so.0`io_loop_run(ioloop=0x000055e13cfc8d80) at ioloop.c:740
frame #16: 0x00007f0c774f61eb libdovecot.so.0`master_service_run(service=0x000055e13cfc8c10, callback=<unavailable>) at master-service.c:782
frame #17: 0x000055e13b48e3a5 stats`main(argc=<unavailable>, argv=<unavailable>) at main.c:99
frame #18: 0x00007f0c771092e1 libc.so.6`__libc_start_main + 241
frame #19: 0x000055e13b48e41a stats`_start + 42
DovecotSync
pushed a commit
that referenced
this pull request
Jul 12, 2019
This fixes a race condition where the http_client_host_shared_idle_timeout()
function would get called with an already freed hshared argument.
Specifically, the situation arises from the hshared idle timeout calling
http_client_host_shared_free(), which removes the timeout and then proceeds to
free the client queue. The client queue freeing code indirectly calls
http_client_host_shared_check_idle(), which notices that there is no idle
timeout and allocates one.
The backtrace at the point of this new timeout allocation:
frame #3: 0x00007f0c775897f0 libdovecot.so.0`timeout_add_to(...) ioloop.c:280
frame #4: 0x00007f0c7751a45f libdovecot.so.0`http_client_host_shared_check_idle(hshared=<unavailable>) at http-client-host.c:69
frame #5: 0x00007f0c7750de89 libdovecot.so.0`http_client_request_error(_req=<unavailable>, status=9000, error="") at http-client-request.c:1525
frame #6: 0x00007f0c77517f38 libdovecot.so.0`http_client_queue_fail_full(queue=0x000055e13cff0e10, status=9000, error="", all=<unavailable>) at http-client-queue.c:183
frame #7: 0x00007f0c77518baa libdovecot.so.0`http_client_queue_free(queue=0x000055e13cff0e10) at http-client-queue.c:141
frame #8: 0x00007f0c7751a8bc libdovecot.so.0`http_client_host_free_shared(_host=<unavailable>) at http-client-host.c:391
frame #9: 0x00007f0c7751ab4c libdovecot.so.0`http_client_host_shared_free(_hshared=0x00007ffdac109e48) at http-client-host.c:294
frame #10: 0x00007f0c7751ace8 libdovecot.so.0`http_client_host_shared_idle_timeout(hshared=<unavailable>) at http-client-host.c:40
frame #11: 0x00007f0c7758a1a4 libdovecot.so.0`io_loop_handle_timeouts at ioloop.c:682
frame #12: 0x00007f0c7758a089 libdovecot.so.0`io_loop_handle_timeouts(ioloop=0x000055e13cfc8d80) at ioloop.c:696
frame #13: 0x00007f0c7758befc libdovecot.so.0`io_loop_handler_run_internal(ioloop=0x000055e13cfc8d80) at ioloop-select.c:126
frame #14: 0x00007f0c7758a56d libdovecot.so.0`io_loop_handler_run(ioloop=<unavailable>) at ioloop.c:767
frame #15: 0x00007f0c7758a798 libdovecot.so.0`io_loop_run(ioloop=0x000055e13cfc8d80) at ioloop.c:740
frame #16: 0x00007f0c774f61eb libdovecot.so.0`master_service_run(service=0x000055e13cfc8c10, callback=<unavailable>) at master-service.c:782
frame #17: 0x000055e13b48e3a5 stats`main(argc=<unavailable>, argv=<unavailable>) at main.c:99
frame #18: 0x00007f0c771092e1 libc.so.6`__libc_start_main + 241
frame #19: 0x000055e13b48e41a stats`_start + 42
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We do not tell the remote we are closing if they have
already told us because they close the
connection after sending ITEM_FINISH or ITEM_DONE and will
not be ever receive anything else from us unless
it just happens to get combined into the same packet
as a previous response and is already in the buffer.
This solved the problem I reported here
http://www.dovecot.org/list/dovecot/2016-May/104337.html
I was getting a EPIPE about 1:30 dsync runs over the socket. I have received 0 in 10000 runs with this change.