Skip to content

Fix test-run crash when default server is crashed #405

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Aug 1, 2023

Conversation

ylobankov
Copy link
Contributor

@ylobankov ylobankov commented Aug 1, 2023

Before this change, test-run crashed if the default tarantool server
had crashed with a segmentation fault, and as a result, this caused an
unhandled error with the broken tarantool console handshake. So such a
situation wasn't properly processed by test-run and it failed with the
following traceback:

Traceback (most recent call last):
  File "/tarantool/test-run/lib/worker.py", line 283, in __init__
    self.inspector = suite.start_server(self.server)
  File "/tarantool/test-run/lib/test_suite.py", line 244, in start_server
    server.deploy(silent=False)
  File "/tarantool/test-run/lib/tarantool_server.py", line 784, in deploy
    self.start(silent=silent, **kwargs)
  File "/tarantool/test-run/lib/tarantool_server.py", line 911, in start
    self.wait_until_started(wait_load, deadline)
  File "/tarantool/test-run/lib/tarantool_server.py", line 1157, in wait_until_started
    ans = yaml.safe_load(temp.execute('box.info.status'))[0]
  File "/tarantool/test-run/lib/tarantool_connection.py", line 161, in execute
    self.opt_reconnect()
  File "/tarantool/test-run/lib/tarantool_connection.py", line 150, in opt_reconnect
    self.reconnect()
  File "/tarantool/test-run/lib/tarantool_connection.py", line 138, in reconnect
    self.connect()
  File "/tarantool/test-run/lib/admin_connection.py", line 96, in connect
    raise RuntimeError('Broken tarantool console handshake')
RuntimeError: Broken tarantool console handshake

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/tarantool/test-run/dispatcher.py", line 386, in _run_worker
    worker = self.gen_worker(worker_id)
  File "/tarantool/test-run/lib/worker.py", line 297, in __init__
    self.stop_server(cleanup=False)
  File "/tarantool/test-run/lib/worker.py", line 301, in stop_server
    self.suite.stop_server(self.server, self.inspector, silent=silent,
  File "/tarantool/test-run/lib/test_suite.py", line 249, in stop_server
    server.stop(silent=silent)
  File "/tarantool/test-run/lib/tarantool_server.py", line 1092, in stop
    save_join(self.crash_detector)
  File "/tarantool/test-run/lib/tarantool_server.py", line 57, in save_join
    green_obj.get(timeout=timeout)
  File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get
  File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception
  File "/home/tarantool/.local/lib/python3.10/site-packages/gevent/_compat.py", line 65, in reraise
    raise value.with_traceback(tb)
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/tarantool/test-run/lib/test.py", line 35, in _run
    self.callable(*self.callable_args, **self.callable_kwargs)
  File "/tarantool/test-run/lib/tarantool_server.py", line 967, in crash_detect
    self.kill_current_test()
  File "/tarantool/test-run/lib/tarantool_server.py", line 1032, in kill_current_test
    if self.current_test.current_test_greenlet:
AttributeError: 'NoneType' object has no attribute 'current_test_greenlet'

Now test-run handles the segmentation fault of the default server and
the error with the broken tarantool console handshake properly.

Fixes #398

Copy link
Member

@Totktonada Totktonada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No objections from me.

Copy link

@ochaplashkin ochaplashkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

@ylobankov ylobankov force-pushed the ylobankov/fix-crash-detect branch from 1ebcac1 to 8292e14 Compare August 1, 2023 13:42
Before this change, test-run crashed if the default tarantool server
had crashed with a segmentation fault, and as a result, this caused an
unhandled error with the broken tarantool console handshake. So such a
situation wasn't properly processed by test-run and it failed with the
following traceback:

    Traceback (most recent call last):
      File "/tarantool/test-run/lib/worker.py", line 283, in __init__
        self.inspector = suite.start_server(self.server)
      File "/tarantool/test-run/lib/test_suite.py", line 244, in start_server
        server.deploy(silent=False)
      File "/tarantool/test-run/lib/tarantool_server.py", line 784, in deploy
        self.start(silent=silent, **kwargs)
      File "/tarantool/test-run/lib/tarantool_server.py", line 911, in start
        self.wait_until_started(wait_load, deadline)
      File "/tarantool/test-run/lib/tarantool_server.py", line 1157, in wait_until_started
        ans = yaml.safe_load(temp.execute('box.info.status'))[0]
      File "/tarantool/test-run/lib/tarantool_connection.py", line 161, in execute
        self.opt_reconnect()
      File "/tarantool/test-run/lib/tarantool_connection.py", line 150, in opt_reconnect
        self.reconnect()
      File "/tarantool/test-run/lib/tarantool_connection.py", line 138, in reconnect
        self.connect()
      File "/tarantool/test-run/lib/admin_connection.py", line 96, in connect
        raise RuntimeError('Broken tarantool console handshake')
    RuntimeError: Broken tarantool console handshake

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
        self.run()
      File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
        self._target(*self._args, **self._kwargs)
      File "/tarantool/test-run/dispatcher.py", line 386, in _run_worker
        worker = self.gen_worker(worker_id)
      File "/tarantool/test-run/lib/worker.py", line 297, in __init__
        self.stop_server(cleanup=False)
      File "/tarantool/test-run/lib/worker.py", line 301, in stop_server
        self.suite.stop_server(self.server, self.inspector, silent=silent,
      File "/tarantool/test-run/lib/test_suite.py", line 249, in stop_server
        server.stop(silent=silent)
      File "/tarantool/test-run/lib/tarantool_server.py", line 1092, in stop
        save_join(self.crash_detector)
      File "/tarantool/test-run/lib/tarantool_server.py", line 57, in save_join
        green_obj.get(timeout=timeout)
      File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get
      File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception
      File "/home/tarantool/.local/lib/python3.10/site-packages/gevent/_compat.py", line 65, in reraise
        raise value.with_traceback(tb)
      File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
      File "/tarantool/test-run/lib/test.py", line 35, in _run
        self.callable(*self.callable_args, **self.callable_kwargs)
      File "/tarantool/test-run/lib/tarantool_server.py", line 967, in crash_detect
        self.kill_current_test()
      File "/tarantool/test-run/lib/tarantool_server.py", line 1032, in kill_current_test
        if self.current_test.current_test_greenlet:
    AttributeError: 'NoneType' object has no attribute 'current_test_greenlet'

Now test-run handles the segmentation fault of the default server and
the error with the broken tarantool console handshake properly.

Fixes #398
Now it is an irrelevant feature for us because we restart Tarantool
server before each test [1]. So the order of tests shouldn't affect
anything.

This patch just disables reproduce content printing. It is planned
to drop the related code later.

[1] #309
@ylobankov ylobankov force-pushed the ylobankov/fix-crash-detect branch from 8292e14 to 1a8d4ca Compare August 1, 2023 14:07
@ylobankov ylobankov merged commit 31f0ced into master Aug 1, 2023
@ylobankov ylobankov deleted the ylobankov/fix-crash-detect branch August 1, 2023 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

test-run is crashed when default tarantool server is crashed
3 participants