cabal-tests --with-cabal should have sensible default

[andreasabel]

As suggested in #8392 (comment), the result of cabal list-bin cabal-install should be the default for --with-cabal (rather than skipping all tests that require cabal-install).

In general, a --with-PROG should only overwrite an existing default for PROG, at least that is the custom (followed by configure and cabal).

[fgaz]

Then we'll also need a way to skip cabal-install tests for the lib-only part of the testsuite

[ulysses4ever]

This may be convenient indeed but I have two reservations.

A mild one: this looks like a case where you may want to be explicit about the argument, because it’s the single most important one for the whole test suite.

Next, technical: is there a precedent in the whole project where some tool calls some other tool to infer a default value of its parameter. This looks like an extremely fragile scheme. What if there’s no cabal built yet? What if it’s stale? That’s why I’m slightly leaning towards explicit argument passing in this case: it raises the chance that the user actually realizes that the value makes any sense.

The current behavior is not satisfying either. I think I’d prefer it to fail if no --with-cabal provided (for lib tests that are okay with missing cabal there may be another argument, e.g. --only-lib or something). But I’m not actively against this proposal either.

[andreasabel]

Then we'll also need a way to skip cabal-install tests for the lib-only part of the testsuite

As I suggested in another thread, I find --with-cabal=- intuitive to state that the testsuite should be run without cabal-install.

I haven't thought through the cabal list-bin cabal-install which @Mikolaj suggested. If fact, I don't like cabal list-bin because it doesn't straight out print the last generated executable, but starts musing about whether it needs to reconfigure the project. And if you switched GHC to another version in the meantime, it cries "trouble". I prefer cabal-plan list-bin which doesn't make any fuss.

@ulysses4ever : If --with-cabal should be mandatory, then the use of option syntax --OPT is wrong. Options are for optional things. The correct syntax is then:

cabal-tests CABAL FILES ...

CABAL  The cabal-install executable to use in the test suite.

[Mikolaj]

I prefer cabal-plan list-bin which doesn't make any fuss.

Agreed. If we go for it, we definitely should not try to configure or build the in-tree cabal, but instead fail hard if it's not present (all this assuming the user did not specify the cabal to use).

[andreasabel]

Just fell into this trap again today. Maybe the --with-cabal option could be added to the example in the docs here:

cabal/cabal-testsuite/README.md

Lines 8 to 11 in bcfc79c

	2. Run the `cabal-tests` executable. It will scan for all tests
	in your current directory and subdirectories and run them.
	To run a specific set of tests, use `cabal-tests PATH ...`.
	(e.g. `cabal run cabal-testsuite:cabal-tests -- cabal-testsuite/PackageTests/TestOptions/setup.test.hs`)

Also, cabal-tests could emit a clear warning if --with-cabal was not supplied. The current communication looks like this:

$ cabal-tests -- cabal-testsuite/PackageTests/ListBin/Script/cabal.test.hs 
/usr/local/bin/runghc-9.4.4 -- '--ghc-arg=-i' '--ghc-arg=-no-user-package-db' '--ghc-arg=-package-db' '--ghc-arg=/Users/abel/.cabal/store/ghc-9.4.4/package.db' '--ghc-arg=-package-db' '--ghc-arg=/Users/abel/bin/src/cabal/dist-newstyle/packagedb/ghc-9.4.4' '--ghc-arg=-package-id' '--ghc-arg=Cabal-3.9.0.0-inplace' '--ghc-arg=-package-id' '--ghc-arg=Cabal-syntax-3.9.0.0-inplace' '--ghc-arg=-package-id' '--ghc-arg=sync-2.2.4-67e91ea5' '--ghc-arg=-package-id' '--ghc-arg=base-4.17.0.0' '--ghc-arg=-package-id' '--ghc-arg=cabal-testsuite-3-inplace' '--ghc-arg=-package-id' '--ghc-arg=clck-0.8.3-0601bb17' '--ghc-arg=-package-id' '--ghc-arg=exceptions-0.10.5' '--ghc-arg=-package-id' '--ghc-arg=filepath-1.4.2.2' '--ghc-arg=-package-id' '--ghc-arg=ptprs-pplctv-0.16.1.0-d424affa' '--ghc-arg=-package-id' '--ghc-arg=process-1.6.16.0' '--ghc-arg=-package-id' '--ghc-arg=transformers-0.5.6.2' cabal-testsuite/PackageTests/ListBin/Script/cabal.test.hs --builddir /Users/abel/bin/src/cabal/dist-newstyle/build/x86_64-osx/ghc-9.4.4/cabal-testsuite-3 cabal-testsuite/PackageTests/ListBin/Script/cabal.test.hs
SKIP no cabal-install
cabal.test.hs: TestCodeSkip "no cabal-install"
cabal-tests: callProcess: /usr/local/bin/runghc-9.4.4 "--" "--ghc-arg=-i" "--ghc-arg=-no-user-package-db" "--ghc-arg=-package-db" "--ghc-arg=/Users/abel/.cabal/store/ghc-9.4.4/package.db" "--ghc-arg=-package-db" "--ghc-arg=/Users/abel/bin/src/cabal/dist-newstyle/packagedb/ghc-9.4.4" "--ghc-arg=-package-id" "--ghc-arg=Cabal-3.9.0.0-inplace" "--ghc-arg=-package-id" "--ghc-arg=Cabal-syntax-3.9.0.0-inplace" "--ghc-arg=-package-id" "--ghc-arg=sync-2.2.4-67e91ea5" "--ghc-arg=-package-id" "--ghc-arg=base-4.17.0.0" "--ghc-arg=-package-id" "--ghc-arg=cabal-testsuite-3-inplace" "--ghc-arg=-package-id" "--ghc-arg=clck-0.8.3-0601bb17" "--ghc-arg=-package-id" "--ghc-arg=exceptions-0.10.5" "--ghc-arg=-package-id" "--ghc-arg=filepath-1.4.2.2" "--ghc-arg=-package-id" "--ghc-arg=ptprs-pplctv-0.16.1.0-d424affa" "--ghc-arg=-package-id" "--ghc-arg=process-1.6.16.0" "--ghc-arg=-package-id" "--ghc-arg=transformers-0.5.6.2" "cabal-testsuite/PackageTests/ListBin/Script/cabal.test.hs" "--builddir" "/Users/abel/bin/src/cabal/dist-newstyle/build/x86_64-osx/ghc-9.4.4/cabal-testsuite-3" "cabal-testsuite/PackageTests/ListBin/Script/cabal.test.hs" (exit 1): failed

[andreasabel]

Because cabal run cabal-testsuite:cabal-tests puts the correct cabal into the PATH, the default for --with-cabal could simply be cabal. The test runner could inform about the choice of cabal when starting up (by printing the full path of the resolved cabal).

If one wants to skip the cabal-install tests, we could have an option --skip-cabal-install-tests, in the same way as we have a --skip-setup-tests already.