Options that change how tests are run should not cause a rebuild

[erikd]

Describe the bug
Building my project and the running the tests, causes the test command to rebuild everything.

I like to see the output of my test suites so I use --test-show-details=direct and I have multiple test suites and to avoid them all running at the same time (and having their output all come out at once) I use -j1.

To Reproduce
Steps to reproduce the behavior:
Using cabal version 3.0:

$ cabal build all
$ cabal test all -j1 --test-show-details=direct

If I run the same two commands again, the build command will rebuild everything and then the test command will again rebuild everything.

Expected behavior
The test command should only build the stuff that has not been built.

System informataion

• Linuix
• cabal 3.0
• ghc 8.6.5

Related: #6129 #6069

[robx]

I see something similar with test-options, which makes cabal test rather painful to use as a test runner. Changing a test filter passed through test-options raises the test overhead from 1s to 5s. (Happy to file separately if desired.)

test log

$ time cabal test --test-options='-t noexist' # ran the same thing before
Build profile: -w ghc-8.10.4 -O1
In order, the following will be built (use -v for more details):
 - configurator-pg-0.2.5 (test:tests) (first run)
Preprocessing test suite 'tests' for configurator-pg-0.2.5..
Building test suite 'tests' for configurator-pg-0.2.5..
Running 1 test suites...
Test suite tests: RUNNING...
Test suite tests: PASS
Test suite logged to:
/s/configurator-pg/dist-newstyle/build/x86_64-osx/ghc-8.10.4/configurator-pg-0.2.5/t/tests/test/configurator-pg-0.2.5-tests.log
1 of 1 test suites (1 of 1 test cases) passed.

real	0m0.850s
user	0m0.639s
sys	0m0.167s
$ time cabal test --test-options='-t reallynoexist'
Build profile: -w ghc-8.10.4 -O1
In order, the following will be built (use -v for more details):
 - configurator-pg-0.2.5 (lib) (configuration changed)
 - configurator-pg-0.2.5 (test:tests) (configuration changed)
Configuring library for configurator-pg-0.2.5..
Preprocessing library for configurator-pg-0.2.5..
Building library for configurator-pg-0.2.5..
Configuring test suite 'tests' for configurator-pg-0.2.5..
Warning: The package has an extraneous version range for a dependency on an
internal library: configurator-pg >=0 && ==0.2.5. This version range includes
the current package but isn't needed as the current package's library will
always be used.
Preprocessing test suite 'tests' for configurator-pg-0.2.5..
Building test suite 'tests' for configurator-pg-0.2.5..
Running 1 test suites...
Test suite tests: RUNNING...
Test suite tests: PASS
Test suite logged to:
/s/configurator-pg/dist-newstyle/build/x86_64-osx/ghc-8.10.4/configurator-pg-0.2.5/t/tests/test/configurator-pg-0.2.5-tests.log
1 of 1 test suites (1 of 1 test cases) passed.

real	0m4.885s
user	0m3.253s
sys	0m1.235s

[Mikolaj]

To help you (and people searching and finding this issue) out until this is solved in cabal itself, is building and then repeatedly running the test executable directly from its path, with suitable options, a sufficient workaround?

[fgaz]

Duplicate of #6114 ?

[robx]

Duplicate of #6114 ?

Seems like this is a part of #6114, though by my reading this might be better off as a separate issue? (Namely test-options shouldn't be a solver input, while enable-tests should be?)

[robx]

This is particularly bad on cabal itself:

$ time cabal test unit-tests --test-options="-h"
Build profile: -w ghc-8.10.4 -O1
In order, the following will be built (use -v for more details):
 - Cabal-3.7.0.0 (lib) (configuration changed)
 - Cabal-described-3.7.0.0 (lib) (configuration changed)
 - Cabal-QuickCheck-3.7.0.0 (lib) (configuration changed)
 - Cabal-tests-3 (test:unit-tests) (configuration changed)
Configuring library for Cabal-3.7.0.0..
Preprocessing library for Cabal-3.7.0.0..
Building library for Cabal-3.7.0.0..
Configuring library for Cabal-QuickCheck-3.7.0.0..
Configuring library for Cabal-described-3.7.0.0..
Preprocessing library for Cabal-QuickCheck-3.7.0.0..
Building library for Cabal-QuickCheck-3.7.0.0..
Preprocessing library for Cabal-described-3.7.0.0..
Building library for Cabal-described-3.7.0.0..
Configuring test suite 'unit-tests' for Cabal-tests-3..
Preprocessing test suite 'unit-tests' for Cabal-tests-3..
Building test suite 'unit-tests' for Cabal-tests-3..
Running 1 test suites...
Test suite unit-tests: RUNNING...
Test suite unit-tests: PASS
Test suite logged to:
/s/cabal/dist-newstyle/build/x86_64-osx/ghc-8.10.4/Cabal-tests-3/t/unit-tests/test/Cabal-tests-3-unit-tests.log
1 of 1 test suites (1 of 1 test cases) passed.

real	0m15.737s
user	0m13.792s
sys	0m4.300s

[robx]

To help you (and people searching and finding this issue) out until this is solved in cabal itself, is building and then repeatedly running the test executable directly from its path, with suitable options, a sufficient workaround?

It's not really a good work-around, because I usually want recompiles if anything changed. So this will require me to keep track of whether I edited anything myself. (Nevermind having to figure out how to execute the test binary, which again isn't something I usually have at the top of my mind.)

[fgaz]

A better workaround for the --test-options case could be to use run, since it supports running tests too. cabal run sometest -- --some-option shouldn't cause a rebuild when test options (after the --) are changed