Failure in SimpleCov test suite

[eregon]

From https://github.com/simplecov-ruby/simplecov/pull/1079/files#r1566262539

The failure is:

$ bundle exec rake spec

Failures:

  1) return codes inside fixtures/frameworks when running testunit_bad.rb behaves like bad tests with default configuration prints a message to STDERR
     Failure/Error: expect(@stderr).to match(/stopped.+SimpleCov.+previous.+error/i)
     
       expected "" to match /stopped.+SimpleCov.+previous.+error/i
       Diff:
       @@ -1 +1 @@
       -/stopped.+SimpleCov.+previous.+error/i
       +""
       
     Shared Example Group: "bad tests" called from ./spec/return_codes_spec.rb:69
     # ./spec/return_codes_spec.rb:37:in `block (5 levels) in <top (required)>'
     # ./spec/return_codes_spec.rb:12:in `block (4 levels) in <top (required)>'
     # ./spec/return_codes_spec.rb:10:in `block (3 levels) in <top (required)>'

Finished in 40.45 seconds (files took 1.59 seconds to load)
389 examples, 1 failure

Failed examples:

rspec './spec/return_codes_spec.rb[1:1:3:1:1:2]' # return codes inside fixtures/frameworks when running testunit_bad.rb behaves like bad tests with default configuration prints a message to STDERR

Note the exit code is already correct (the spec above), but for some reason there is no output.

[andrykonchin]

The failing test checks that simplecov prints a warning about a previous error (exception) before reporting its own results. So in the failing test a test-unit test case fails and test-unit calls exit 1 in its at_exit callback (what raises SystemExit exception). simplecov checks $! in its own at_exit callback and prints the warning.

It seems the issue is that SystemExit raised by test-unit isn't visible in the simplecov's at_exit callback so no warning is printed.

This codesnippet reproduces the issue:

at_exit do
  puts "#1"
  puts $!.inspect
end

at_exit do
  puts "#2"
  exit 42
end

Ruby 3.2

ruby -v test.rb
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-darwin21]
#2
#1
#<SystemExit: exit>

TruffleRuby master

jt -q ruby test.rb
#2
#1
nil

[PragTob]

@andrykonchin great reproduction work, thank you very much! 💚

[andrykonchin]

Fixed in cc2d014

[eregon]

@andrykonchin Could you make a PR to SimpleCov to unexclude that test? (see link in description)

[andrykonchin]

The enabled back test case is supposed to pass on TruffleRuby head (that is used in the scheduled builds) and still fail on TruffleRuby 24.0.0 (that is used in builds triggered in PRs).

So what is better to do - to unexclude the test now and don't run specs on TruffleRuby 24.0.0 or unexclude it later when the fix is released?

[PragTob]

@andrykonchin unexclude it later when the fix is released and available on github actions (imo)

[eregon]

But it will be like 6 months until the next release.
I think it makes sense to alter the condition to only skip the test on TruffleRuby < 24.1.

[andrykonchin]

Right, makes sense.