test failure in 1.18.3

[asarubbo]

Hello,

I can reproduce a test failure in a RockyLinux-9 docker container.

Elixir version: 1.18.3
Erlang version: 27.3.3

This is the output of make -j2 && make -j2 test:

#7 1641.6 Recompile: src/iex
#7 1641.9 Recompile: src/elixir_utils
#7 1642.2 Recompile: src/elixir_tokenizer
#7 1644.8 Recompile: src/elixir_sup
#7 1644.9 Recompile: src/elixir_rewrite
#7 1645.4 Recompile: src/elixir_quote
#7 1645.9 Recompile: src/elixir_parser
#7 1657.7 Recompile: src/elixir_overridable
#7 1657.8 Recompile: src/elixir_module
#7 1658.4 Recompile: src/elixir_map
#7 1658.6 Recompile: src/elixir_lexical
#7 1658.7 Recompile: src/elixir_json
#7 1664.1 Recompile: src/elixir_interpolation
#7 1664.7 Recompile: src/elixir_import
#7 1664.9 Recompile: src/elixir_fn
#7 1665.1 Recompile: src/elixir_expand
#7 1666.3 Recompile: src/elixir_errors
#7 1666.6 Recompile: src/elixir_erl_var
#7 1666.7 Recompile: src/elixir_erl_try
#7 1666.9 Recompile: src/elixir_erl_pass
#7 1667.7 Recompile: src/elixir_erl_for
#7 1668.1 Recompile: src/elixir_erl_compiler
#7 1668.2 Recompile: src/elixir_erl_clauses
#7 1668.3 Recompile: src/elixir_erl
#7 1668.9 Recompile: src/elixir_env
#7 1669.0 Recompile: src/elixir_dispatch
#7 1669.3 Recompile: src/elixir_def
#7 1669.7 Recompile: src/elixir_config
#7 1669.8 Recompile: src/elixir_compiler
#7 1669.9 Recompile: src/elixir_code_server
#7 1670.0 Recompile: src/elixir_clauses
#7 1670.4 Recompile: src/elixir_bootstrap
#7 1670.4 Recompile: src/elixir_bitstring
#7 1670.7 Recompile: src/elixir_aliases
#7 1670.8 Recompile: src/elixir
#7 1672.0 Generated elixir app
#7 1672.0 ==> bootstrap (compile)
#7 1674.9 Compiled kernel.ex
#7 1675.2 Compiled kernel/utils.ex
#7 1675.5 Compiled macro/env.ex
#7 1676.0 Compiled keyword.ex
#7 1677.0 Compiled module.ex
#7 1677.4 Compiled list.ex
#7 1679.1 Compiled macro.ex
#7 1680.6 Compiled kernel/typespec.ex
#7 1681.1 Compiled code.ex
#7 1681.3 Compiled code/identifier.ex
#7 1681.9 Compiled protocol.ex
#7 1682.2 Compiled stream/reducers.ex
#7 1685.3 Compiled enum.ex
#7 1685.8 Compiled regex.ex
#7 1686.6 Compiled inspect/algebra.ex
#7 1687.7 Compiled inspect.ex
#7 1689.1 Compiled string.ex
#7 1689.3 Compiled string/chars.ex
#7 1691.8 Compiled kernel.ex
#7 1692.0 Compiled list/chars.ex
#7 1692.1 Compiled bitwise.ex
#7 1692.6 Compiled module/parallel_checker.ex
#7 1693.0 Compiled module/behaviour.ex
#7 1693.2 Compiled module/types/helpers.ex
#7 1695.4 Compiled module/types/descr.ex
#7 1695.8 Compiled module/types/of.ex
#7 1696.7 Compiled module/types/pattern.ex
#7 1698.0 Compiled module/types/apply.ex
#7 1699.1 Compiled module/types/expr.ex
#7 1699.5 Compiled module/types.ex
#7 1702.3 Compiled exception.ex
#7 1702.7 Compiled path.ex
#7 1703.4 Compiled file.ex
#7 1703.7 Compiled map.ex
#7 1703.7 Compiled function.ex
#7 1704.1 Compiled range.ex
#7 1704.7 Compiled access.ex
#7 1705.0 Compiled io.ex
#7 1705.6 Compiled system.ex
#7 1707.9 Compiled code/formatter.ex
#7 1708.5 Compiled code/normalizer.ex
#7 1709.2 Compiled kernel/cli.ex
#7 1709.3 Compiled kernel/error_handler.ex
#7 1710.2 Compiled kernel/parallel_compiler.ex
#7 1710.4 Compiled kernel/lexical_tracker.ex
#7 1710.4 make[1]: Entering directory '/b/elixir-1.18.3'
#7 1710.4 ==> unicode (compile)
#7 1714.8 [Unicode] Break on 25 whitespace codepoints
#7 1716.4 [Unicode] Tokenizing 112700 non-ascii codepoints
#7 1716.8 [Unicode] Tokenizing 65 scriptsets
#7 1723.4 make[1]: Leaving directory '/b/elixir-1.18.3'
#7 1723.4 ==> elixir (compile)
#7 1754.9 Generated elixir app
#7 1754.9 ==> eex (compile)
#7 1756.9 ==> mix (compile)
#7 1767.3 Generated mix app
#7 1767.3 ==> ex_unit (compile)
#7 1773.3 Generated ex_unit app
#7 1773.3 ==> logger (compile)
#7 1777.0 Generated logger app
#7 1778.2 Generated eex app
#7 1778.2 ==> iex (compile)
#7 1783.1 Generated iex app
#7 1794.4 ==> elixir (eunit)
#7 1795.6   All 126 tests passed.
#7 1795.6 
#7 1795.6 ==> elixir (ex_unit)
#7 1858.4 Running ExUnit with seed: 739527, max_cases: 4
#7 1858.4 Excluding tags: [windows: true]
#7 1858.4 
#7 1858.4 .................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
#7 1920.7 Finished in 123.1 seconds (60.8s on load, 45.9s async, 16.4s sync)
#7 1920.7 2040 doctests, 4550 tests, 0 failures, 13 excluded
#7 1920.7 ==> ex_unit (ex_unit)
#7 1929.5 Running ExUnit with seed: 847311, max_cases: 4
#7 1929.5 
#7 1929.6 ........................................................................................................................................................................................................................................................................................................................................................................................................................................................
#7 1933.8 Finished in 11.8 seconds (7.6s on load, 0.6s async, 3.6s sync)
#7 1933.8 48 doctests, 392 tests, 0 failures
#7 1933.9 ==> logger (ex_unit)
#7 1939.4 Running ExUnit with seed: 755924, max_cases: 4
#7 1939.4 
#7 1939.4 ................................................................................................................................................................
#7 1940.4 Finished in 5.1 seconds (4.1s on load, 0.04s async, 0.9s sync)
#7 1940.4 3 doctests, 157 tests, 0 failures
#7 1940.4 ==> eex (ex_unit)
#7 1942.4 Running ExUnit with seed: 753143, max_cases: 4
#7 1942.4 
#7 1942.4 .................................................................................................................
#7 1942.5 Finished in 0.9 seconds (0.8s on load, 0.1s async, 0.00s sync)
#7 1942.5 7 doctests, 106 tests, 0 failures
#7 1942.6 ==> iex (ex_unit)
#7 1946.5 Running ExUnit with seed: 871427, max_cases: 4
#7 1946.5 
#7 1946.5 .........................................................................................................................................................................................................................................................................
#7 1963.3 Finished in 19.5 seconds (2.7s on load, 0.7s async, 16.0s sync)
#7 1963.3 265 tests, 0 failures
#7 1963.4 ==> mix (ex_unit)
#7 1979.1 Running ExUnit with seed: 340699, max_cases: 4
#7 1979.1 Excluding tags: [windows: true]
#7 1979.1 
#7 1979.2 .
#7 1979.3 
#7 1979.3   1) test delivers broadcast to subscribers for different keys (Mix.Sync.PubSubTest)
#7 1979.3      /b/elixir-1.18.3/lib/mix/test/mix/sync/pubsub_test.exs:39
#7 1979.3      Assertion failed, no matching message after 100ms
#7 1979.3      The process mailbox is empty.
#7 1979.3      code: assert_receive :subscribed1
#7 1979.3      stacktrace:
#7 1979.3        test/mix/sync/pubsub_test.exs:60: (test)
#7 1979.3 
#7 1979.3 .............................................................................................................................................................................................................................................................................................................    warning: redefining module Mix.Tasks.Local.Sample (current version defined in memory)
#7 2038.6     │
#7 2038.6   1 │ defmodule Mix.Tasks.Local.Sample do
#7 2038.6     │ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#7 2038.6     │
#7 2038.6     └─ lib/local.sample.ex:1: Mix.Tasks.Local.Sample (module)
#7 2038.6 
#7 2038.6 ................................
#7 2044.7 09:17:03.156 [notice] Application parsetools exited: :stopped
#7 2044.7 .
#7 2044.8 09:17:03.200 [notice] Application parsetools exited: :stopped
#7 2044.8 .
#7 2044.8 09:17:03.239 [notice] Application parsetools exited: :stopped
#7 2044.8 .....
#7 2047.1 09:17:05.523 [notice] Application parsetools exited: :stopped
#7 2047.1 ..
#7 2047.2 09:17:05.573 [notice] Application parsetools exited: :stopped
#7 2047.2 
#7 2047.2 09:17:05.622 [notice] Application parsetools exited: :stopped
#7 2047.2 .
#7 2047.3 09:17:05.688 [notice] Application parsetools exited: :stopped
#7 2047.3 ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
#7 2408.6 Finished in 443.6 seconds (14.2s on load, 0.5s async, 428.9s sync)
#7 2408.6 6 doctests, 855 tests, 1 failure
#7 2408.6 make: *** [Makefile:116: test_mix] Error 2

Is this a know issue?
Is there something else I can provide to help debug the issue?

Thanks

[josevalim]

This is a timing issue, so I suspect the machine has low resources. You can add ExUnit.configure(assert_receive_timeout: 1000) to lib/mix/test/test_helper.exs. I will add an env var to make it easier.

[asarubbo]

Thanks for the fast response. Can you clarify in which part of test_helper.exs I can put ExUnit.configure(assert_receive_timeout: 1000)

Anyway this is a shared vserver with 2 cores, so not a lot of resources, maybe -j1 could help here

Thanks

[josevalim]

@asarubbo Elixir tests already run concurrently whenever possible, so you don't need -j 1 for running tests. Also, some of those tests may start subprocesses or test concurrency itself (as it is a concurrent language), so there is certainly a high amount of concurrency going on. You can see the linked commit / PR for the changes necessary. We already had an env var for the different suites and now we are applying it consistently. :)

[asarubbo]

test the fixed version is not really simple to me right now, but test the future (1.18.4) is really straightforward to me, so if you don't mind I will re-test after new version will be out.

Thanks.