Implement --benchmark-options for v2-bench #6224

RyanGlScott · 2019-09-06T01:09:06Z

This implements lots of plumbing to allow the --benchmark-option(s) flags to be used with v2-bench, analgous to v2-test's --test-option(s) flag.

Fixes #6209.

Please include the following checklist in your PR:

Patches conform to the coding conventions.
Any changes that could be relevant to users have been recorded in the changelog.
The documentation has been updated, if necessary.
If the change is docs-only, [ci skip] is used to avoid triggering the build bots.

Please also shortly describe how you tested your change. Bonus points for added tests!

RyanGlScott · 2019-09-06T01:09:49Z

I have confirmed that I can run cabal v2-bench --benchmark-options="--help" locally, and it does what you would expect...

$ /home/rgscott/Documents/Hacking/Haskell/cabal/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cabal-install-3.1.0.0/build/cabal/cabal bench --enable-benchmarks --benchmark-options="--help"
Resolving dependencies...
Build profile: -w ghc-8.6.5 -O1
In order, the following will be built (use -v for more details):
 - text-show-3.8.2 (lib) (configuration changed)
 - text-show-3.8.2 (bench:bench) (configuration changed)
Configuring library for text-show-3.8.2..
Preprocessing library for text-show-3.8.2..
Building library for text-show-3.8.2..
Configuring benchmark 'bench' for text-show-3.8.2..
Warning: The package has an extraneous version range for a dependency on an
internal library: text-show -any && ==3.8.2, text-show -any && ==3.8.2. This
version range includes the current package but isn't needed as the current
package's library will always be used.
Preprocessing benchmark 'bench' for text-show-3.8.2..
Building benchmark 'bench' for text-show-3.8.2..
Running 1 benchmarks...
Benchmark bench: RUNNING...
Microbenchmark suite - built with criterion 1.5.5.0

Usage: bench [-I|--ci CI] [-L|--time-limit SECS] [--resamples COUNT]
             [--regress RESP:PRED..] [--raw FILE] [-o|--output FILE]
             [--csv FILE] [--json FILE] [--junit FILE] [-v|--verbosity LEVEL]
             [-t|--template FILE] [[-m|--match MATCH] [NAME...] |
             (-n|--iters ITERS) [-m|--match MATCH] [NAME...] | (-l|--list) |
             --version]

Available options:
  -h,--help                Show this help text
  -I,--ci CI               Confidence interval
  -L,--time-limit SECS     Time limit to run a benchmark
  --resamples COUNT        Number of bootstrap resamples to perform
  --regress RESP:PRED..    Regressions to perform
  --raw FILE               File to write raw data to
  -o,--output FILE         File to write report to
  --csv FILE               File to write CSV summary to
  --json FILE              File to write JSON summary to
  --junit FILE             File to write JUnit summary to
  -v,--verbosity LEVEL     Verbosity level
  -t,--template FILE       Template to use for report
  -m,--match MATCH         How to match benchmark names ("prefix", "glob",
                           "pattern", or "ipattern")
  -n,--iters ITERS         Run benchmarks, don't analyse
  -m,--match MATCH         How to match benchmark names ("prefix", "glob",
                           "pattern", or "ipattern")
  -l,--list                List benchmarks
  --version                Show version info

Regression metrics (for use with --regress):
  time                     wall-clock time
  cpuTime                  CPU time
  cycles                   CPU cycles
  iters                    loop iterations
  allocated                (+RTS -T) bytes allocated
  numGcs                   (+RTS -T) number of garbage collections
  bytesCopied              (+RTS -T) number of bytes copied during GC
  mutatorWallSeconds       (+RTS -T) wall-clock time for mutator threads
  mutatorCpuSeconds        (+RTS -T) CPU time spent running mutator threads
  gcWallSeconds            (+RTS -T) wall-clock time spent doing GC
  gcCpuSeconds             (+RTS -T) CPU time spent doing GC
Benchmark bench: FINISH

Well, mostly. What is up with this part?

Warning: The package has an extraneous version range for a dependency on an
internal library: text-show -any && ==3.8.2, text-show -any && ==3.8.2. This
version range includes the current package but isn't needed as the current
package's library will always be used.

The package whose benchmark I'm running (text-show) does not even have an internal library, so I have no idea why it's warning about that.

cabal-install/Distribution/Client/ProjectConfig/Legacy.hs

sjakobi · 2019-09-06T01:17:12Z

Well, mostly. What is up with this part?
Warning: The package has an extraneous version range for a dependency on an
internal library: text-show -any && ==3.8.2, text-show -any && ==3.8.2. This
version range includes the current package but isn't needed as the current
package's library will always be used.
The package whose benchmark I'm running (text-show) does not even have an internal library, so I have no idea why it's warning about that.

That looks like #5119.

RyanGlScott · 2019-09-06T01:21:56Z

While it's possible that it's "just" #5119, it's worth noting that with this patch, the extraneous warning now appears any time you run cabal v2-bench (even without --benchmark-options). This is a borderline regression in behavior, which makes me rather nervous.

RyanGlScott · 2019-09-06T01:32:00Z

Actually, I retract my claim in #6224 (comment) — I don't think my patch caused this warning. That's because I get the same warning if I run cabal v2-bench using an unmodified cabal-install from commit 3042371. So while the reappearance of this warning is probably cause for concern, it seems doubtful that this patch caused it.

sjakobi · 2019-09-06T01:34:35Z

Thanks a lot for implementing this @RyanGlScott! The new options work very nicely for me! 👍

RyanGlScott · 2019-09-06T01:55:40Z

On certain Travis configurations (e.g., GHCVER=8.2.2 SCRIPT=script USE_GOLD=YES), I'm getting test suite failures like this:

[17 of 34] Compiling UnitTests.Distribution.Client.ProjectConfig ( tests/UnitTests/Distribution/Client/ProjectConfig.hs, /home/travis/build/haskell/cabal/dist-newstyle/build/x86_64-linux/ghc-8.2.2/cabal-install-3.1.0.0/x/cabal/build/cabal/cabal-tmp/UnitTests/Distribution/Client/ProjectConfig.o )

tests/UnitTests/Distribution/Client/ProjectConfig.hs:561:7: error:

    • Couldn't match type ‘[PathTemplate] -> PackageConfig’

                     with ‘PackageConfig’

      Expected type: Gen PackageConfig

        Actual type: Gen ([PathTemplate] -> PackageConfig)

    • In the expression:

        PackageConfig

          <$>

            (MapLast . Map.fromList

               <$>

                 shortListOf

                   10 ((,) <$> arbitraryProgramName <*> arbitraryShortToken))

          <*>

            (MapMappend . Map.fromList

               <$>

                 shortListOf

                   10 ((,) <$> arbitraryProgramName <*> listOf arbitraryShortToken))

          <*> (toNubList <$> listOf arbitraryShortToken)

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> shortListOf 5 arbitraryShortToken

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> shortListOf 5 arbitraryShortToken

          <*> shortListOf 5 arbitraryShortToken

          <*> shortListOf 5 arbitraryShortToken

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitraryFlag arbitraryShortToken

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitraryFlag arbitraryShortToken

          <*> arbitrary

          <*> arbitrary

          <*> arbitraryFlag arbitraryShortToken

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitraryFlag arbitraryShortToken

          <*> arbitrary

          <*> shortListOf 5 arbitrary

      In an equation for ‘arbitrary’:

          arbitrary

            = PackageConfig

                <$>

                  (MapLast . Map.fromList

                     <$>

                       shortListOf

                         10 ((,) <$> arbitraryProgramName <*> arbitraryShortToken))

                <*>

                  (MapMappend . Map.fromList

                     <$>

                       shortListOf

                         10 ((,) <$> arbitraryProgramName <*> listOf arbitraryShortToken))

                <*> (toNubList <$> listOf arbitraryShortToken)

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> shortListOf 5 arbitraryShortToken

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> shortListOf 5 arbitraryShortToken

                <*> shortListOf 5 arbitraryShortToken

                <*> shortListOf 5 arbitraryShortToken

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitraryFlag arbitraryShortToken

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitraryFlag arbitraryShortToken

                <*> arbitrary

                <*> arbitrary

                <*> arbitraryFlag arbitraryShortToken

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitraryFlag arbitraryShortToken

                <*> arbitrary

                <*> shortListOf 5 arbitrary

            where

                arbitraryProgramName :: Gen String

                arbitraryProgramName

                  = elements

                      [programName prog | (prog, _) <- knownPrograms (defaultProgramDb)]

      In the instance declaration for ‘Arbitrary PackageConfig’

    |

561 |       PackageConfig

    |       ^^^^^^^^^^^^^...
The command "./travis-${SCRIPT}.sh -j2" exited with 1.

The thing is, I'm not actually sure how to reproduce this failure locally. I'm testing with cabal build all --enable-tests --enable-benchmarks. What should I run to build this file?

RyanGlScott · 2019-09-06T13:30:34Z

With @hvr's help (thanks!), I believe I figured out how to fix the issue in #6224 (comment). Let's see if Travis agrees with me.

RyanGlScott · 2019-09-06T14:14:08Z

The AppVeyor build is segfaulting:

cabal-testsuite\PackageTests\CustomPreProcess\setup.test.hs                                                                       FAIL (18.96s)
$ "C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\runghc.exe" "--" "--ghc-arg=-i" "--ghc-arg=-no-user-package-db" "--ghc-arg=-package-db" "--ghc-arg=C:\SR\ghc-8.0.2\package.db" "--ghc-arg=-package-db" "--ghc-arg=C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "--ghc-arg=-package-id" "--ghc-arg=Cabal-3.1.0.0-inplace" "--ghc-arg=-package-id" "--ghc-arg=async-2.2.2-1efad130fd1a9aa0bcbad23df9d6e3b7f4f52833" "--ghc-arg=-package-id" "--ghc-arg=base-4.9.1.0" "--ghc-arg=-package-id" "--ghc-arg=cabal-testsuite-3.1.0.0-inplace" "--ghc-arg=-package-id" "--ghc-arg=clock-0.7.2-088dcc63def76a4701b4ae9b8e5e4e0d49b5f553" "--ghc-arg=-package-id" "--ghc-arg=exceptions-0.10.3-ad54f44e07a565173d7d1654d483c10a94670bca" "--ghc-arg=-package-id" "--ghc-arg=filepath-1.4.1.1" "--ghc-arg=-package-id" "--ghc-arg=optparse-appl_-0.14.3.0-9b251e4813f894e07f1c2aac43d0da5797a64d59" "--ghc-arg=-package-id" "--ghc-arg=process-1.4.3.0" "--ghc-arg=-package-id" "--ghc-arg=transformers-0.5.2.0" "cabal-testsuite\PackageTests\CustomPreProcess\setup.test.hs" "--builddir" "C:\projects\cabal\dist-newstyle\build\x86_64-windows\ghc-8.0.2\cabal-testsuite-3.1.0.0" "cabal-testsuite\PackageTests\CustomPreProcess\setup.test.hs" "--with-cabal" "dist-newstyle\build\x86_64-windows\ghc-8.0.2\cabal-install-3.1.0.0\x\cabal\build\cabal\cabal.exe"
stdout:
# Setup configure
+ "C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\runghc.exe" "--" "--ghc-arg=-i" "--ghc-arg=-no-user-package-db" "--ghc-arg=-package-db" "--ghc-arg=C:\SR\ghc-8.0.2\package.db" "--ghc-arg=-package-db" "--ghc-arg=C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "--ghc-arg=-package-id" "--ghc-arg=Cabal-3.1.0.0-inplace" "--ghc-arg=-package-id" "--ghc-arg=async-2.2.2-1efad130fd1a9aa0bcbad23df9d6e3b7f4f52833" "--ghc-arg=-package-id" "--ghc-arg=base-4.9.1.0" "--ghc-arg=-package-id" "--ghc-arg=cabal-testsuite-3.1.0.0-inplace" "--ghc-arg=-package-id" "--ghc-arg=clock-0.7.2-088dcc63def76a4701b4ae9b8e5e4e0d49b5f553" "--ghc-arg=-package-id" "--ghc-arg=exceptions-0.10.3-ad54f44e07a565173d7d1654d483c10a94670bca" "--ghc-arg=-package-id" "--ghc-arg=filepath-1.4.1.1" "--ghc-arg=-package-id" "--ghc-arg=optparse-appl_-0.14.3.0-9b251e4813f894e07f1c2aac43d0da5797a64d59" "--ghc-arg=-package-id" "--ghc-arg=process-1.4.3.0" "--ghc-arg=-package-id" "--ghc-arg=transformers-0.5.2.0" "C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\.\.\Setup.hs" "configure" "-vverbose +markoutput +nowrap" "--distdir" "setup.dist\work\dist" "--global" "--with-ghc" "C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--with-haddock" "C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\haddock.exe" "--enable-deterministic" "--prefix=C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\usr" "--package-db=clear" "--package-db=global" "--package-db=C:\SR\ghc-8.0.2\package.db" "--package-db=C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2"
Using Parsec parser
-----BEGIN CABAL OUTPUT-----
Configuring internal-preprocessor-test-0.1.0.0...
-----END CABAL OUTPUT-----
Dependency base -any: using base-4.9.1.0
Dependency base -any: using base-4.9.1.0
Dependency directory -any: using directory-1.3.0.0
Dependency base -any: using base-4.9.1.0
Dependency internal-preprocessor-test -any: using internal-preprocessor-test-0.1.0.0
Source component graph:
    component exe:my-custom-preprocessor
    component lib dependency exe:my-custom-preprocessor
    component exe:hello-world dependency lib
Configured component graph:
    component internal-preprocessor-test-0.1.0.0-my-custom-preprocessor
        include base-4.9.1.0
        include directory-1.3.0.0
    component internal-preprocessor-test-0.1.0.0 include base-4.9.1.0
    component internal-preprocessor-test-0.1.0.0-hello-world
        include base-4.9.1.0
        include internal-preprocessor-test-0.1.0.0
Linked component graph:
    unit internal-preprocessor-test-0.1.0.0-my-custom-preprocessor
        include base-4.9.1.0
        include directory-1.3.0.0
    unit internal-preprocessor-test-0.1.0.0
        include base-4.9.1.0
        A=internal-preprocessor-test-0.1.0.0:A
    unit internal-preprocessor-test-0.1.0.0-hello-world
        include base-4.9.1.0
        include internal-preprocessor-test-0.1.0.0
Ready component graph:
    definite internal-preprocessor-test-0.1.0.0-my-custom-preprocessor
        depends base-4.9.1.0
        depends directory-1.3.0.0
    definite internal-preprocessor-test-0.1.0.0
        depends base-4.9.1.0
        depends internal-preprocessor-test-0.1.0.0-my-custom-preprocessor
    definite internal-preprocessor-test-0.1.0.0-hello-world
        depends base-4.9.1.0
        depends internal-preprocessor-test-0.1.0.0
Using Cabal-3.1.0.0 compiled by ghc-8.0
Using compiler: ghc-8.0.2
Using install prefix: C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\usr
Executables installed in: C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\usr\bin
Libraries installed in: C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\usr\x86_64-windows-ghc-8.0.2\internal-preprocessor-test-0.1.0.0
Dynamic Libraries installed in: C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\usr\x86_64-windows-ghc-8.0.2
Private executables installed in: C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\usr\internal-preprocessor-test-0.1.0.0\x86_64-windows-ghc-8.0.2\internal-preprocessor-test-0.1.0.0
Data files installed in: C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\usr\x86_64-windows-ghc-8.0.2\internal-preprocessor-test-0.1.0.0
Documentation installed in: C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\usr\doc\x86_64-windows-ghc-8.0.2\internal-preprocessor-test-0.1.0.0
Configuration files installed in: C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\usr\etc
Using alex version 3.2.4 found on system at: C:\Users\appveyor\AppData\Roaming\cabal\bin\alex.exe
Using ar found on system at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\mingw\bin\ar.exe
No c2hs found
No cpphs found
No doctest found
Using gcc version 5.2.0 found on system at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\mingw\bin\gcc.exe
Using ghc version 8.0.2 given by user at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe
Using ghc-pkg version 8.0.2 found on system at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc-pkg.exe
No ghcjs found
No ghcjs-pkg found
No greencard found
Using haddock version 2.17.3 given by user at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\haddock.exe
Using happy version 1.19.12 found on system at: C:\Users\appveyor\AppData\Roaming\cabal\bin\happy.exe
Using haskell-suite found on system at: haskell-suite-dummy-location
Using haskell-suite-pkg found on system at: haskell-suite-pkg-dummy-location
No hmake found
Using hpc version 0.67 found on system at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\hpc.exe
Using hsc2hs version 0.68.1 found on system at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\hsc2hs.exe
No hscolour found
No jhc found
Using ld found on system at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\mingw\bin\ld.exe
Using pkg-config version 0.29.2 found on system at: C:\msys64\usr\bin\pkg-config.exe
Using runghc version 8.0.2 found on system at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\runghc.exe
Using strip version 2.25 found on system at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\mingw\bin\strip.exe
Using tar found on system at: C:\msys64\usr\bin\tar.exe
No uhc found
# Setup build
+ "C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\runghc.exe" "--" "--ghc-arg=-i" "--ghc-arg=-no-user-package-db" "--ghc-arg=-package-db" "--ghc-arg=C:\SR\ghc-8.0.2\package.db" "--ghc-arg=-package-db" "--ghc-arg=C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "--ghc-arg=-package-id" "--ghc-arg=Cabal-3.1.0.0-inplace" "--ghc-arg=-package-id" "--ghc-arg=async-2.2.2-1efad130fd1a9aa0bcbad23df9d6e3b7f4f52833" "--ghc-arg=-package-id" "--ghc-arg=base-4.9.1.0" "--ghc-arg=-package-id" "--ghc-arg=cabal-testsuite-3.1.0.0-inplace" "--ghc-arg=-package-id" "--ghc-arg=clock-0.7.2-088dcc63def76a4701b4ae9b8e5e4e0d49b5f553" "--ghc-arg=-package-id" "--ghc-arg=exceptions-0.10.3-ad54f44e07a565173d7d1654d483c10a94670bca" "--ghc-arg=-package-id" "--ghc-arg=filepath-1.4.1.1" "--ghc-arg=-package-id" "--ghc-arg=optparse-appl_-0.14.3.0-9b251e4813f894e07f1c2aac43d0da5797a64d59" "--ghc-arg=-package-id" "--ghc-arg=process-1.4.3.0" "--ghc-arg=-package-id" "--ghc-arg=transformers-0.5.2.0" "C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\.\.\Setup.hs" "build" "-vverbose +markoutput +nowrap" "--distdir" "setup.dist\work\dist"
Component build order: executable 'my-custom-preprocessor', library, executable 'hello-world'
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc-pkg.exe" "init" "setup.dist\work\dist\package.conf.inplace"
creating setup.dist\work\dist\build\my-custom-preprocessor
creating setup.dist\work\dist\build\my-custom-preprocessor\autogen
creating setup.dist\work\dist\build\my-custom-preprocessor\autogen
-----BEGIN CABAL OUTPUT-----
Preprocessing executable 'my-custom-preprocessor' for internal-preprocessor-test-0.1.0.0..
-----END CABAL OUTPUT-----
-----BEGIN CABAL OUTPUT-----
Building executable 'my-custom-preprocessor' for internal-preprocessor-test-0.1.0.0..
-----END CABAL OUTPUT-----
creating setup.dist\work\dist\build\my-custom-preprocessor
creating setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--make" "-no-link" "-fbuilding-cabal-package" "-O" "-static" "-outputdir" "setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-odir" "setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-hidir" "setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-stubdir" "setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-i" "-isetup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-i." "-isetup.dist\work\dist\build\my-custom-preprocessor\autogen" "-isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build\my-custom-preprocessor\autogen" "-Isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-optP-include" "-optPsetup.dist\work\dist\build\my-custom-preprocessor\autogen\cabal_macros.h" "-hide-all-packages" "-no-user-package-db" "-package-db" "C:\SR\ghc-8.0.2\package.db" "-package-db" "C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "-package-db" "setup.dist\work\dist\package.conf.inplace" "-package-id" "base-4.9.1.0" "-package-id" "directory-1.3.0.0" "-XHaskell2010" ".\MyCustomPreprocessor.hs"
[1 of 1] Compiling Main             ( MyCustomPreprocessor.hs, setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp\Main.o )
Linking...
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--make" "-fbuilding-cabal-package" "-O" "-static" "-outputdir" "setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-odir" "setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-hidir" "setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-stubdir" "setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-i" "-isetup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-i." "-isetup.dist\work\dist\build\my-custom-preprocessor\autogen" "-isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build\my-custom-preprocessor\autogen" "-Isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-optP-include" "-optPsetup.dist\work\dist\build\my-custom-preprocessor\autogen\cabal_macros.h" "-hide-all-packages" "-no-user-package-db" "-package-db" "C:\SR\ghc-8.0.2\package.db" "-package-db" "C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "-package-db" "setup.dist\work\dist\package.conf.inplace" "-package-id" "base-4.9.1.0" "-package-id" "directory-1.3.0.0" "-XHaskell2010" ".\MyCustomPreprocessor.hs" "-o" "setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor.exe"
Linking setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor.exe ...
creating setup.dist\work\dist\build
creating setup.dist\work\dist\build\autogen
creating setup.dist\work\dist\build\autogen
-----BEGIN CABAL OUTPUT-----
Preprocessing library for internal-preprocessor-test-0.1.0.0..
-----END CABAL OUTPUT-----
creating setup.dist\work\dist\build
Preprocessing A.pre to setup.dist\work\dist\build\A.hs
-----BEGIN CABAL OUTPUT-----
Building library for internal-preprocessor-test-0.1.0.0..
-----END CABAL OUTPUT-----
creating setup.dist\work\dist\build
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--make" "-fbuilding-cabal-package" "-O" "-outputdir" "setup.dist\work\dist\build" "-odir" "setup.dist\work\dist\build" "-hidir" "setup.dist\work\dist\build" "-stubdir" "setup.dist\work\dist\build" "-i" "-isetup.dist\work\dist\build" "-i." "-isetup.dist\work\dist\build\autogen" "-isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build\autogen" "-Isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build" "-optP-include" "-optPsetup.dist\work\dist\build\autogen\cabal_macros.h" "-this-unit-id" "internal-preprocessor-test-0.1.0.0" "-hide-all-packages" "-no-user-package-db" "-package-db" "C:\SR\ghc-8.0.2\package.db" "-package-db" "C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "-package-db" "setup.dist\work\dist\package.conf.inplace" "-package-id" "base-4.9.1.0" "-XHaskell2010" "A"
[1 of 1] Compiling A                ( setup.dist\work\dist\build\A.hs, setup.dist\work\dist\build\A.o )
Linking...
[(DefiniteUnitId (DefUnitId {unDefUnitId = UnitId "base-4.9.1.0"}),DefaultRenaming)]
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\mingw\bin\ar.exe" "-r" "setup.dist\work\dist\build\objs-672\libHSinternal-preprocessor-test-0.1.0.0.a" "@setup.dist\work\dist\build\objs-672\ar2696224464.rsp"
C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\mingw\bin\ar.exe: creating setup.dist\work\dist\build\objs-672\libHSinternal-preprocessor-test-0.1.0.0.a
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\mingw\bin\ld.exe" "-x" "-r" "-o" "setup.dist\work\dist\build\HSinternal-preprocessor-test-0.1.0.0.o" "@setup.dist\work\dist\build\ld570528145.rsp"
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc-pkg.exe" "recache" "--package-db=setup.dist\work\dist\package.conf.inplace"
creating setup.dist\work\dist\build\hello-world
creating setup.dist\work\dist\build\hello-world\autogen
creating setup.dist\work\dist\build\hello-world\autogen
-----BEGIN CABAL OUTPUT-----
Preprocessing executable 'hello-world' for internal-preprocessor-test-0.1.0.0..
-----END CABAL OUTPUT-----
-----BEGIN CABAL OUTPUT-----
Building executable 'hello-world' for internal-preprocessor-test-0.1.0.0..
-----END CABAL OUTPUT-----
creating setup.dist\work\dist\build\hello-world
creating setup.dist\work\dist\build\hello-world\hello-world-tmp
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--make" "-no-link" "-fbuilding-cabal-package" "-O" "-static" "-outputdir" "setup.dist\work\dist\build\hello-world\hello-world-tmp" "-odir" "setup.dist\work\dist\build\hello-world\hello-world-tmp" "-hidir" "setup.dist\work\dist\build\hello-world\hello-world-tmp" "-stubdir" "setup.dist\work\dist\build\hello-world\hello-world-tmp" "-i" "-isetup.dist\work\dist\build\hello-world\hello-world-tmp" "-i." "-isetup.dist\work\dist\build\hello-world\autogen" "-isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build\hello-world\autogen" "-Isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build\hello-world\hello-world-tmp" "-optP-include" "-optPsetup.dist\work\dist\build\hello-world\autogen\cabal_macros.h" "-hide-all-packages" "-no-user-package-db" "-package-db" "C:\SR\ghc-8.0.2\package.db" "-package-db" "C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "-package-db" "setup.dist\work\dist\package.conf.inplace" "-package-id" "base-4.9.1.0" "-package-id" "internal-preprocessor-test-0.1.0.0" "-XHaskell2010" ".\Hello.hs"
[1 of 1] Compiling Main             ( Hello.hs, setup.dist\work\dist\build\hello-world\hello-world-tmp\Main.o )
Linking...
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--make" "-fbuilding-cabal-package" "-O" "-static" "-outputdir" "setup.dist\work\dist\build\hello-world\hello-world-tmp" "-odir" "setup.dist\work\dist\build\hello-world\hello-world-tmp" "-hidir" "setup.dist\work\dist\build\hello-world\hello-world-tmp" "-stubdir" "setup.dist\work\dist\build\hello-world\hello-world-tmp" "-i" "-isetup.dist\work\dist\build\hello-world\hello-world-tmp" "-i." "-isetup.dist\work\dist\build\hello-world\autogen" "-isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build\hello-world\autogen" "-Isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build\hello-world\hello-world-tmp" "-optP-include" "-optPsetup.dist\work\dist\build\hello-world\autogen\cabal_macros.h" "-hide-all-packages" "-no-user-package-db" "-package-db" "C:\SR\ghc-8.0.2\package.db" "-package-db" "C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "-package-db" "setup.dist\work\dist\package.conf.inplace" "-package-id" "base-4.9.1.0" "-package-id" "internal-preprocessor-test-0.1.0.0" "-XHaskell2010" ".\Hello.hs" "-o" "setup.dist\work\dist\build\hello-world\hello-world.exe"
Linking setup.dist\work\dist\build\hello-world\hello-world.exe ...
# hello-world
+ "C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\work\.\dist\build\hello-world\hello-world"
hello from A
# Setup configure
+ "C:\projects\cabal\dist-newstyle\build\x86_64-windows\ghc-8.0.2\cabal-install-3.1.0.0\x\cabal\build\cabal\cabal.exe" "v1-configure" "-vverbose +markoutput +nowrap" "--distdir" "setup.cabal.dist\work\dist" "--global" "--with-ghc" "C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--with-haddock" "C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\haddock.exe" "--enable-deterministic" "--prefix=C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.cabal.dist\usr" "--package-db=clear" "--package-db=global" "--package-db=C:\SR\ghc-8.0.2\package.db" "--package-db=C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2"
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--numeric-version"
looking for tool ghc-pkg near compiler in C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin
found ghc-pkg in C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc-pkg.exe
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc-pkg.exe" "--version"
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--supported-languages"
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--info"
Warning: No remote package servers have been specified. Usually you would have one specified in the config file.
"C:\msys64\usr\bin\pkg-config.exe" "--version"
"C:\msys64\usr\bin\pkg-config.exe" "--list-all"
"C:\msys64\usr\bin\pkg-config.exe" "--modversion" "libalpm" "python2" "python3" "libsvn_fs_x" "libsvn_diff" "libsvn_fs" "libsvn_ra_serf" "python-3.7m" "libsvn_ra" "libsvn_client" "python-2.7" "bash-completion" "libsvn_fs_util" "libsvn_fs_fs" "python-3.7" "autoopts" "libsvn_ra_local" "libsvn_wc" "libsvn_subr" "libsvn_ra_svn" "libsvn_delta" "libsvn_repos"
-----BEGIN CABAL OUTPUT-----
Resolving dependencies...
-----END CABAL OUTPUT-----
creating setup.cabal.dist\work\dist\setup
copy .\Setup.hs to .\setup.cabal.dist\work\dist\setup\setup.hs
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--make" "-odir" ".\setup.cabal.dist\work\dist\setup" "-hidir" ".\setup.cabal.dist\work\dist\setup" "-i" "-i." "-package-db" "C:\SR\ghc-8.0.2\package.db" "-package-db" "C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "-package-id" "Cabal-3.1.0.0-inplace" ".\setup.cabal.dist\work\dist\setup\setup.hs" "-o" ".\setup.cabal.dist\work\dist\setup\setup.exe" "-threaded"
[1 of 1] Compiling Main             ( setup.cabal.dist\work\dist\setup\setup.hs, setup.cabal.dist\work\dist\setup\Main.o )
Linking .\setup.cabal.dist\work\dist\setup\setup.exe ...
C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.cabal.dist\work\dist\setup\setup.exe configure --verbose=verbose +nowrap +markoutput --builddir=setup.cabal.dist\work\dist --ghc --with-compiler=C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe --prefix=C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.cabal.dist\usr --global --package-db=clear --package-db=global --package-db=C:\SR\ghc-8.0.2\package.db --package-db=C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2 --enable-deterministic --ipid=internal-preprocessor-test-0.1.0.0 --dependency=base=base-4.9.1.0 --dependency=base=base-4.9.1.0 --dependency=base=base-4.9.1.0 --dependency=directory=directory-1.3.0.0 --disable-tests --exact-configuration --disable-benchmarks --with-ghc=C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe --with-haddock=C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\haddock.exe
Segmentation fault/access violation in generated code
stderr:
*** Exception: Command "C:\projects\cabal\dist-newstyle\build\x86_64-windows\ghc-8.0.2\cabal-install-3.1.0.0\x\cabal\build\cabal\cabal.exe" "v1-configure" "-vverbose +markoutput +nowrap" "--distdir" "setup.cabal.dist\work\dist" "--global" "--with-ghc" "C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--with-haddock" "C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\haddock.exe" "--enable-deterministic" "--prefix=C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.cabal.dist\usr" "--package-db=clear" "--package-db=global" "--package-db=C:\SR\ghc-8.0.2\package.db" "--package-db=C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" failed.
Output:
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--numeric-version"
looking for tool ghc-pkg near compiler in C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin
found ghc-pkg in C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc-pkg.exe
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc-pkg.exe" "--version"
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--supported-languages"
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--info"
Warning: No remote package servers have been specified. Usually you would have one specified in the config file.
"C:\msys64\usr\bin\pkg-config.exe" "--version"
"C:\msys64\usr\bin\pkg-config.exe" "--list-all"
"C:\msys64\usr\bin\pkg-config.exe" "--modversion" "libalpm" "python2" "python3" "libsvn_fs_x" "libsvn_diff" "libsvn_fs" "libsvn_ra_serf" "python-3.7m" "libsvn_ra" "libsvn_client" "python-2.7" "bash-completion" "libsvn_fs_util" "libsvn_fs_fs" "python-3.7" "autoopts" "libsvn_ra_local" "libsvn_wc" "libsvn_subr" "libsvn_ra_svn" "libsvn_delta" "libsvn_repos"
-----BEGIN CABAL OUTPUT-----
Resolving dependencies...
-----END CABAL OUTPUT-----
creating setup.cabal.dist\work\dist\setup
copy .\Setup.hs to .\setup.cabal.dist\work\dist\setup\setup.hs
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--make" "-odir" ".\setup.cabal.dist\work\dist\setup" "-hidir" ".\setup.cabal.dist\work\dist\setup" "-i" "-i." "-package-db" "C:\SR\ghc-8.0.2\package.db" "-package-db" "C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "-package-id" "Cabal-3.1.0.0-inplace" ".\setup.cabal.dist\work\dist\setup\setup.hs" "-o" ".\setup.cabal.dist\work\dist\setup\setup.exe" "-threaded"
[1 of 1] Compiling Main             ( setup.cabal.dist\work\dist\setup\setup.hs, setup.cabal.dist\work\dist\setup\Main.o )
Linking .\setup.cabal.dist\work\dist\setup\setup.exe ...
C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.cabal.dist\work\dist\setup\setup.exe configure --verbose=verbose +nowrap +markoutput --builddir=setup.cabal.dist\work\dist --ghc --with-compiler=C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe --prefix=C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.cabal.dist\usr --global --package-db=clear --package-db=global --package-db=C:\SR\ghc-8.0.2\package.db --package-db=C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2 --enable-deterministic --ipid=internal-preprocessor-test-0.1.0.0 --dependency=base=base-4.9.1.0 --dependency=base=base-4.9.1.0 --dependency=base=base-4.9.1.0 --dependency=directory=directory-1.3.0.0 --disable-tests --exact-configuration --disable-benchmarks --with-ghc=C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe --with-haddock=C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\haddock.exe
Segmentation fault/access violation in generated code
*** unexpected failure for cabal-testsuite\PackageTests\CustomPreProcess\setup.test.hs

As far as I can tell, this is #6219. This failure also happens on master, however, so I'm going to choose to ignore this failure unless someone tells me otherwise.

RyanGlScott · 2019-09-06T14:27:52Z

I think I've resolved all outstanding issues. Ready for a proper review now!

RyanGlScott · 2019-09-17T14:14:02Z

Would someone knowledgeable in the area be willing to give this a review?

RyanGlScott · 2019-10-08T11:03:57Z

A gentle ping on this.

cabal-install/Distribution/Client/CmdRepl.hs

cabal-install/Distribution/Client/CmdLegacy.hs

cabal-install/Distribution/Client/Install.hs

Cabal/Distribution/Simple/Setup.hs

cabal-install/Distribution/Client/ProjectConfig/Legacy.hs

DanielG

Ok, that looks good now.

Just one last order of buissness: adding a test. I checked and --test-option(s) has a test in cabal-testsuite/PackageTests/TestOptions/setup.test.hs so you should be able to use that as a template.

RyanGlScott · 2019-10-10T15:26:10Z

I checked and --test-option(s) has a test in cabal-testsuite/PackageTests/TestOptions/setup.test.hs so you should be able to use that as a template.

I saw that too, and in fact, there's already a corresponding test called cabal-testsuite/PackageTests/BenchmarkOptions/setup.test.hs. Upon closer inspection, I realized that this isn't testing what we care about. Namely, it's testing that ./Setup bench accepts --benchmark-options, not v2-bench. Do you happen to know if there are any test cases that more closely resemble the latter?

RyanGlScott · 2019-10-10T16:56:32Z

With enough persistence, I was able to cobble together a test case that exercises the cabal v2-bench --benchmark-options="..." code path. Let's see if CI likes it...

DanielG · 2019-10-11T06:48:16Z

Great! Just one more thing, can you make the test ensure that multiple --benchmark-option flags will append to the option set rather than overwrite it? A mixed invocation with --benchmark-options would also be nice. So something like --benchmark-option=1 --benchmark-options="2 3" -- benchmark-option=4 --benchmark-options="5 6" should result in 1 2 3 4 5 6.

Since we're relying a lot on genericlly derived Monoid instances and it's not clear at all from the code alone that these flags append I think we should test for that.

If you feel like it a test for v2-test for symmetry would also be good, in a separate commit please.

RyanGlScott · 2019-10-11T10:32:36Z

I've modified the --benchmark-options test case to test multiple uses of the flag, and I've added a corresponding test case for --test-options.

cabal-testsuite/PackageTests/NewBuild/CmdTest/OptionsFlag/cabal.test.hs

DanielG

Thanks for doing the v2-test test too :)

I think we're ready for merging as soon as you address the two remaining minor comments.

cabal-install/Distribution/Client/BuildReports/Anonymous.hs

This implements lots of plumbing to allow the `--benchmark-option(s)` flags to be used with `v2-bench`, analgous to `v2-test`'s `--test-option(s)` flag. Fixes #6209.

RyanGlScott · 2019-10-11T14:39:35Z

Hm, the CI at https://travis-ci.org/haskell-pushbot/cabal-binaries/builds/596608715 seems to have timed out. Moreover, I don't have the ability to restart it (likely due to lack of permissions for https://github.com/haskell-pushbot/cabal-binaries). What should I do?

RyanGlScott · 2019-10-11T14:50:35Z

Upon further investigation, it looks like the OS X jobs are known to time out frequently. Moreover, an upstream commit has explicitly allowed these jobs to fail (see #6278). In light of this, I'm going to just land this as-is.

DanielG · 2019-10-11T14:52:00Z

Yeah don't worry about the OSX jobs too much they're really flaky, but I see you merged it already :)

RyanGlScott commented Sep 6, 2019

View reviewed changes

cabal-install/Distribution/Client/ProjectConfig/Legacy.hs Outdated Show resolved Hide resolved

RyanGlScott mentioned this pull request Sep 6, 2019

Add --benchmark-options option #6209

Closed

RyanGlScott changed the title ~~WIP: Implement --benchmark-options for v2-bench~~ Implement --benchmark-options for v2-bench Sep 6, 2019

RyanGlScott requested review from phadej and 23Skidoo September 26, 2019 16:53

DanielG reviewed Oct 8, 2019

View reviewed changes

DanielG reviewed Oct 10, 2019

View reviewed changes

DanielG reviewed Oct 11, 2019

View reviewed changes

cabal-testsuite/PackageTests/NewBuild/CmdTest/OptionsFlag/cabal.test.hs Outdated Show resolved Hide resolved

DanielG reviewed Oct 11, 2019

View reviewed changes

cabal-install/Distribution/Client/BuildReports/Anonymous.hs Outdated Show resolved Hide resolved

RyanGlScott added 2 commits October 11, 2019 07:29

Implement --benchmark-options for v2-bench

0810c59

This implements lots of plumbing to allow the `--benchmark-option(s)` flags to be used with `v2-bench`, analgous to `v2-test`'s `--test-option(s)` flag. Fixes #6209.

Add test case for v2-test --test-option(s)

ad597e2

RyanGlScott merged commit 21762b7 into haskell:master Oct 11, 2019

DanielG mentioned this pull request Oct 11, 2019

Improve --test/benchmark-options test accuracy #6293

Merged

phadej added this to the 3.2.0.0 milestone Jul 14, 2020

Implement --benchmark-options for v2-bench #6224

Implement --benchmark-options for v2-bench #6224

Uh oh!

Conversation

RyanGlScott commented Sep 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fixes #6209.

Uh oh!

RyanGlScott commented Sep 6, 2019

Uh oh!

Uh oh!

sjakobi commented Sep 6, 2019

Uh oh!

RyanGlScott commented Sep 6, 2019

Uh oh!

RyanGlScott commented Sep 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sjakobi commented Sep 6, 2019

Uh oh!

RyanGlScott commented Sep 6, 2019

Uh oh!

RyanGlScott commented Sep 6, 2019

Uh oh!

RyanGlScott commented Sep 6, 2019

Uh oh!

RyanGlScott commented Sep 6, 2019

Uh oh!

RyanGlScott commented Sep 17, 2019

Uh oh!

RyanGlScott commented Oct 8, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DanielG left a comment

Choose a reason for hiding this comment

Uh oh!

RyanGlScott commented Oct 10, 2019

Uh oh!

RyanGlScott commented Oct 10, 2019

Uh oh!

DanielG commented Oct 11, 2019

Uh oh!

RyanGlScott commented Oct 11, 2019

Uh oh!

Uh oh!

DanielG left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

RyanGlScott commented Oct 11, 2019

Uh oh!

RyanGlScott commented Oct 11, 2019

Uh oh!

DanielG commented Oct 11, 2019

Uh oh!

Uh oh!

RyanGlScott commented Sep 6, 2019 •

edited

Loading

RyanGlScott commented Sep 6, 2019 •

edited

Loading