Fix 'Address already in use' error with TCP sockets #141

Totktonada · 2019-02-18T16:34:45Z

TcpPortDispatcher was introduced to eliminate the race condition when two workers trying to use the same port: the idea is that each worker use its own ports range. Really these ports could race with client ports (from, say, net.box or replication), which typically don't use bind() and so binds to a random available port (despite any dispatched ranges). So this approach is broken by design.

There is the workaround with use_unix_sockets and use_unix_sockets_iproto options: setting them to True should eliminate the problem. However it is possible that some tests are designed to test tcp connections, so I think we should fix the problem.

The way I see is no-bind / bind to zero port and grep logs for a LISTEN / CONSOLE port (or write it from a script to a predefined location). Remove TcpPortDispatcher and find_port() at all.

Hmm, box.cfg{listen = 0} shows binary: bound to 0.0.0.0:0. Need to elaborate how to get the port.

The discussion had initially arisen in #115.

The text was updated successfully, but these errors were encountered:

Totktonada · 2019-11-11T16:49:34Z

See tarantool/tarantool#4620

Totktonada · 2020-04-23T13:42:25Z

NB: use_unix_sockets_iproto does not work with core = 'app' test suites.

Totktonada · 2020-05-13T20:33:04Z

Related: PR #210.

Support 'use_unix_sockets_iproto' suite.ini option in 'core = app' test suites. It'll allow to workaround TCP port choosing race (see more in [1]) by using Unix sockets instead of TCP ones: we did the same for 'core = tarantool' test suites in [2]. [1]: tarantool/test-run#141 [2]: 60f84cb ('test: use unix sockets for iproto connections') Part of #4459.

Support 'use_unix_sockets_iproto' suite.ini option in 'core = app' test suites. It'll allow to workaround TCP port choosing race (see more in [1]) by using Unix sockets instead of TCP ones: we did the same for 'core = tarantool' test suites in [2]. [1]: tarantool/test-run#141 [2]: 60f84cb ('test: use unix sockets for iproto connections') Part of #4459. (cherry picked from commit b7b9a59)

sergos · 2022-02-15T22:15:55Z

Hmm.. strange, for me it looks like tarantool> box.cfg{listen=0} yields 2022-02-16 01:07:32.744 [19216] main/103/interactive I> tx_binary: bound to [::]:53061 on MacOS and 2022-02-16 01:06:33.753 [10090] main/103/interactive I> tx_binary: bound to 0.0.0.0:36267 on Linux (Ubuntu 18.04)
The problem might be in the difference:

tarantool> box.cfg.listen
---
- 0
...

tarantool> box.info.listen
---
- '[::]:53061'
...

Anyways, letting the Tarantool instance to pick a port by itself looks way more robust, since no delay between assigning a port and binding it.

Totktonada · 2022-02-16T09:56:54Z

@sergos It is because tarantool/tarantool#4620 was fixed. I filed this issue against tarantool specifically to have a change to use it here, in test-run.

To exclude the chance to encounter the tarantool/test-run#141 issue in replication-py tests let's switch to using unix sockets instead of TCP ports. NO_DOC=testing stuff NO_TEST=testing stuff NO_CHANGELOG=testing stuff

To reduce the chance to encounter the tarantool/test-run#141 issue in tests let's switch to using unix sockets instead of TCP ports where it is possible. NO_DOC=testing stuff NO_TEST=testing stuff NO_CHANGELOG=testing stuff

To reduce the chance to encounter the tarantool/test-run#141 issue in replication-py/swim tests, let's switch to using unix sockets instead of TCP ports for tarantool console. NO_DOC=testing stuff NO_TEST=testing stuff NO_CHANGELOG=testing stuff

To reduce the chance to encounter the tarantool/test-run#141 issue in replication-py/swim tests, let's switch to using unix sockets instead of TCP ports for tarantool console. NO_DOC=testing stuff NO_TEST=testing stuff NO_CHANGELOG=testing stuff (cherry picked from commit cb6fc4a)

To reduce the chance to encounter the tarantool/test-run#141 issue in replication-py tests, let's switch to using unix sockets instead of TCP ports for tarantool console. NO_DOC=testing stuff NO_TEST=testing stuff NO_CHANGELOG=testing stuff (cherry picked from commit cb6fc4a)

Detect specific failure reasons first. If none found in the whole log, look for generic. This way, specific reasons aren't overshadowed by generic ones, when those are found first in the logs. Introduce one specific reason: "Address already in use", subject of tarantool/test-run#141

To reduce the chance to encounter the tarantool/test-run#141 issue in replication-py/swim tests, let's switch to using unix sockets instead of TCP ports for tarantool console. NO_DOC=testing stuff NO_TEST=testing stuff NO_CHANGELOG=testing stuff

This patch makes test-run use only Unix sockets for admin console connection. The feature to use TCP sockets for it is dropped. Actually, admin console connection is a purely internal thing of test-run, and we can use what is more convenient for such a connection. Using only Unix sockets gives us significant advantages over TCP sockets like connection speed and eliminating issue with getting a free port for TCP connection. Part of #141

This change replaces manual choosing an iproto port for TarantoolServer by the auto resolving mechanism. In two words, test-run always provides '127.0.0.1:0' as a value for LISTEN env variable that is used in a lua file to start a tarantool instance. In this way, the port will be picked automatically, and we are getting the real value of it via the admin console by executing `box.info.listen` that is available for tarantool version >= 2.4.1, and special lua script intended for tarantool version < 2.4.1. Part of #141

This change replaces manual choosing a free iproto port for AppServer by the auto resolving mechanism. In two words, test-run always provides '127.0.0.1:0' as a value for LISTEN env variable that is used in a lua test script. In this way, the iproto port will be picked automatically, but if the test needs the real value of the port, it has to execute `box.info.listen` for tarantool version >= 2.4.1, or other lua code for tarantool version < 2.4.1. Part of #141

Part of #141

Now this functionality is not needed anymore due to added free port auto resolving mechanism. Part of #141

Now these functions are not needed anymore due to added free port auto resolving mechanism. Closes #141

This patch makes test-run use only Unix sockets for admin console connection. The feature to use TCP sockets for it is dropped. Actually, admin console connection is a purely internal thing of test-run, and we can use what is more convenient for such a connection. Using only Unix sockets gives us significant advantages over TCP sockets like connection speed and eliminating issue with getting a free port for TCP connection. Part of #141

This change replaces manual choosing an iproto port for TarantoolServer by the auto resolving mechanism. In two words, test-run always provides '127.0.0.1:0' as a value for LISTEN env variable that is used in a lua file to start a tarantool instance. In this way, the port will be picked automatically, and we are getting the real value of it via the admin console by executing `box.info.listen` that is available for tarantool version >= 2.4.1, and special lua script intended for tarantool version < 2.4.1. Part of #141

This change replaces manual choosing a free iproto port for AppServer by the auto resolving mechanism. In two words, test-run always provides '127.0.0.1:0' as a value for LISTEN env variable that is used in a lua test script. In this way, the iproto port will be picked automatically, but if the test needs the real value of the port, it has to execute `box.info.listen` for tarantool version >= 2.4.1, or other lua code for tarantool version < 2.4.1. Part of #141

Part of #141

Now this functionality is not needed anymore due to added free port auto resolving mechanism. Part of #141

Now these functions are not needed anymore due to added free port auto resolving mechanism. Closes #141

This patch makes test-run use only Unix sockets for admin console connection. The feature to use TCP sockets for it is dropped. Actually, admin console connection is a purely internal thing of test-run, and we can use what is more convenient for such a connection. Using only Unix sockets gives us significant advantages over TCP sockets like connection speed and eliminating issue with getting a free port for TCP connection. Part of #141

This change replaces manual choosing an iproto port for TarantoolServer by the auto resolving mechanism. In two words, test-run always provides '127.0.0.1:0' as a value for LISTEN env variable that is used in a lua file to start a tarantool instance. In this way, the port will be picked automatically, and we are getting the real value of it via the admin console by executing `box.info.listen` that is available for tarantool version >= 2.4.1, and special lua script intended for tarantool version < 2.4.1. Part of #141

This change replaces manual choosing a free iproto port for AppServer by the auto resolving mechanism. In two words, test-run always provides '127.0.0.1:0' as a value for LISTEN env variable that is used in a lua test script. In this way, the iproto port will be picked automatically, but if the test needs the real value of the port, it has to execute `box.info.listen` for tarantool version >= 2.4.1, or other lua code for tarantool version < 2.4.1. Part of #141

Part of #141

Now this functionality is not needed anymore due to added free port auto resolving mechanism. Part of #141

Now these functions are not needed anymore due to added free port auto resolving mechanism. Closes #141

This patch improves getting the iproto port for tarantool < 2.4.1. The previous revision of the lua script might give unstable result (more than 1 port) and test-run failed. Now it is fixed. Follows up #141

NickZay · 2024-03-13T16:52:31Z

I have some problems with ports when I use luatest with tarantool

Totktonada added the bug Something isn't working label Feb 18, 2019

This was referenced Feb 18, 2019

Sometimes results of some tests disappear if an error occured #115

Closed

test: fix 'address already in use' flaky fails tarantool/tarantool#4008

Closed

Totktonada mentioned this issue Mar 12, 2019

Run tests in parallel tarantool/vshard#174

Closed

Totktonada mentioned this issue Sep 23, 2020

Enable test reruns on failed fragiled tests #217

Merged

Totktonada mentioned this issue Mar 30, 2021

test: add initial unit tests and code coverage support #283

Merged

kyukhin added this to the wishlist milestone Oct 15, 2021

Totktonada mentioned this issue Feb 11, 2022

test: flaky often replication-py/init_storage.test.py tarantool/tarantool-qa#228

Open

kyukhin added the teamQ label Apr 11, 2022

kyukhin removed this from the wishlist milestone Apr 11, 2022

Totktonada mentioned this issue May 27, 2022

Create a helper to find a free TCP port tarantool/tt#71

Closed

ylobankov mentioned this issue Jun 2, 2022

Fail *.test.py tests in the case of catching server start errors #333

Closed

ylobankov mentioned this issue Jun 6, 2022

test: use unix sockets in replication-py tests tarantool/tarantool#7246

Merged

NickVolynkin assigned ylobankov Jul 18, 2022

NickVolynkin mentioned this issue Aug 30, 2022

gather_job_data: detect specific before generic tarantool/multivac#40

Merged

ylobankov mentioned this issue Sep 7, 2022

Free port auto resolving for TarantoolServer and AppServer #348

Merged

ylobankov added a commit that referenced this issue Sep 12, 2022

Free port auto resolving for TarantoolInspector

a47569b

Part of #141

ylobankov added a commit that referenced this issue Sep 12, 2022

Eliminate TcpPortDispatcher and port ranges

2da6bb0

Now this functionality is not needed anymore due to added free port auto resolving mechanism. Part of #141

ylobankov added a commit that referenced this issue Sep 12, 2022

Eliminate find_port() and check_port() functions

b70d148

Now these functions are not needed anymore due to added free port auto resolving mechanism. Closes #141

ylobankov added a commit that referenced this issue Sep 12, 2022

Free port auto resolving for TarantoolInspector

7e9f016

Part of #141

ylobankov added a commit that referenced this issue Sep 12, 2022

Eliminate TcpPortDispatcher and port ranges

b36799c

Now this functionality is not needed anymore due to added free port auto resolving mechanism. Part of #141

ylobankov added a commit that referenced this issue Sep 12, 2022

Eliminate find_port() and check_port() functions

f91c0dd

Now these functions are not needed anymore due to added free port auto resolving mechanism. Closes #141

ylobankov closed this as completed in #348 Sep 12, 2022

ylobankov added a commit that referenced this issue Sep 12, 2022

Free port auto resolving for TarantoolInspector

19e409b

Part of #141

ylobankov added a commit that referenced this issue Sep 12, 2022

Eliminate TcpPortDispatcher and port ranges

5a0c9bd

Now this functionality is not needed anymore due to added free port auto resolving mechanism. Part of #141

ylobankov added a commit that referenced this issue Sep 12, 2022

Eliminate find_port() and check_port() functions

9375b98

Now these functions are not needed anymore due to added free port auto resolving mechanism. Closes #141

ylobankov mentioned this issue Sep 15, 2022

Improve getting iproto port for tarantool < 2.4.1 #349

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix 'Address already in use' error with TCP sockets #141

Fix 'Address already in use' error with TCP sockets #141

Totktonada commented Feb 18, 2019

Totktonada commented Nov 11, 2019

Uh oh!

Totktonada commented Apr 23, 2020

Uh oh!

Totktonada commented May 13, 2020

Uh oh!

sergos commented Feb 15, 2022

Uh oh!

Totktonada commented Feb 16, 2022

Uh oh!

NickZay commented Mar 13, 2024

Uh oh!

Fix 'Address already in use' error with TCP sockets #141

Fix 'Address already in use' error with TCP sockets #141

Comments

Totktonada commented Feb 18, 2019

Totktonada commented Nov 11, 2019

Uh oh!

Totktonada commented Apr 23, 2020

Uh oh!

Totktonada commented May 13, 2020

Uh oh!

sergos commented Feb 15, 2022

Uh oh!

Totktonada commented Feb 16, 2022

Uh oh!

NickZay commented Mar 13, 2024

Uh oh!