Skip to content

Implement --benchmark-options for v2-bench #6224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 11, 2019
Merged

Implement --benchmark-options for v2-bench #6224

merged 2 commits into from
Oct 11, 2019

Conversation

RyanGlScott
Copy link
Member

@RyanGlScott RyanGlScott commented Sep 6, 2019

This implements lots of plumbing to allow the --benchmark-option(s) flags to be used with v2-bench, analgous to v2-test's --test-option(s) flag.

Fixes #6209.

Please include the following checklist in your PR:

  • Patches conform to the coding conventions.
  • Any changes that could be relevant to users have been recorded in the changelog.
  • The documentation has been updated, if necessary.
  • If the change is docs-only, [ci skip] is used to avoid triggering the build bots.

Please also shortly describe how you tested your change. Bonus points for added tests!

@RyanGlScott
Copy link
Member Author

I have confirmed that I can run cabal v2-bench --benchmark-options="--help" locally, and it does what you would expect...

$ /home/rgscott/Documents/Hacking/Haskell/cabal/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cabal-install-3.1.0.0/build/cabal/cabal bench --enable-benchmarks --benchmark-options="--help"
Resolving dependencies...
Build profile: -w ghc-8.6.5 -O1
In order, the following will be built (use -v for more details):
 - text-show-3.8.2 (lib) (configuration changed)
 - text-show-3.8.2 (bench:bench) (configuration changed)
Configuring library for text-show-3.8.2..
Preprocessing library for text-show-3.8.2..
Building library for text-show-3.8.2..
Configuring benchmark 'bench' for text-show-3.8.2..
Warning: The package has an extraneous version range for a dependency on an
internal library: text-show -any && ==3.8.2, text-show -any && ==3.8.2. This
version range includes the current package but isn't needed as the current
package's library will always be used.
Preprocessing benchmark 'bench' for text-show-3.8.2..
Building benchmark 'bench' for text-show-3.8.2..
Running 1 benchmarks...
Benchmark bench: RUNNING...
Microbenchmark suite - built with criterion 1.5.5.0

Usage: bench [-I|--ci CI] [-L|--time-limit SECS] [--resamples COUNT]
             [--regress RESP:PRED..] [--raw FILE] [-o|--output FILE]
             [--csv FILE] [--json FILE] [--junit FILE] [-v|--verbosity LEVEL]
             [-t|--template FILE] [[-m|--match MATCH] [NAME...] |
             (-n|--iters ITERS) [-m|--match MATCH] [NAME...] | (-l|--list) |
             --version]

Available options:
  -h,--help                Show this help text
  -I,--ci CI               Confidence interval
  -L,--time-limit SECS     Time limit to run a benchmark
  --resamples COUNT        Number of bootstrap resamples to perform
  --regress RESP:PRED..    Regressions to perform
  --raw FILE               File to write raw data to
  -o,--output FILE         File to write report to
  --csv FILE               File to write CSV summary to
  --json FILE              File to write JSON summary to
  --junit FILE             File to write JUnit summary to
  -v,--verbosity LEVEL     Verbosity level
  -t,--template FILE       Template to use for report
  -m,--match MATCH         How to match benchmark names ("prefix", "glob",
                           "pattern", or "ipattern")
  -n,--iters ITERS         Run benchmarks, don't analyse
  -m,--match MATCH         How to match benchmark names ("prefix", "glob",
                           "pattern", or "ipattern")
  -l,--list                List benchmarks
  --version                Show version info

Regression metrics (for use with --regress):
  time                     wall-clock time
  cpuTime                  CPU time
  cycles                   CPU cycles
  iters                    loop iterations
  allocated                (+RTS -T) bytes allocated
  numGcs                   (+RTS -T) number of garbage collections
  bytesCopied              (+RTS -T) number of bytes copied during GC
  mutatorWallSeconds       (+RTS -T) wall-clock time for mutator threads
  mutatorCpuSeconds        (+RTS -T) CPU time spent running mutator threads
  gcWallSeconds            (+RTS -T) wall-clock time spent doing GC
  gcCpuSeconds             (+RTS -T) CPU time spent doing GC
Benchmark bench: FINISH

Well, mostly. What is up with this part?

Warning: The package has an extraneous version range for a dependency on an
internal library: text-show -any && ==3.8.2, text-show -any && ==3.8.2. This
version range includes the current package but isn't needed as the current
package's library will always be used.

The package whose benchmark I'm running (text-show) does not even have an internal library, so I have no idea why it's warning about that.

@sjakobi
Copy link
Member

sjakobi commented Sep 6, 2019

Well, mostly. What is up with this part?

Warning: The package has an extraneous version range for a dependency on an
internal library: text-show -any && ==3.8.2, text-show -any && ==3.8.2. This
version range includes the current package but isn't needed as the current
package's library will always be used.

The package whose benchmark I'm running (text-show) does not even have an internal library, so I have no idea why it's warning about that.

That looks like #5119.

@RyanGlScott
Copy link
Member Author

While it's possible that it's "just" #5119, it's worth noting that with this patch, the extraneous warning now appears any time you run cabal v2-bench (even without --benchmark-options). This is a borderline regression in behavior, which makes me rather nervous.

@RyanGlScott
Copy link
Member Author

RyanGlScott commented Sep 6, 2019

Actually, I retract my claim in #6224 (comment) — I don't think my patch caused this warning. That's because I get the same warning if I run cabal v2-bench using an unmodified cabal-install from commit 3042371. So while the reappearance of this warning is probably cause for concern, it seems doubtful that this patch caused it.

@sjakobi
Copy link
Member

sjakobi commented Sep 6, 2019

Thanks a lot for implementing this @RyanGlScott! The new options work very nicely for me! 👍

@RyanGlScott
Copy link
Member Author

On certain Travis configurations (e.g., GHCVER=8.2.2 SCRIPT=script USE_GOLD=YES), I'm getting test suite failures like this:

[17 of 34] Compiling UnitTests.Distribution.Client.ProjectConfig ( tests/UnitTests/Distribution/Client/ProjectConfig.hs, /home/travis/build/haskell/cabal/dist-newstyle/build/x86_64-linux/ghc-8.2.2/cabal-install-3.1.0.0/x/cabal/build/cabal/cabal-tmp/UnitTests/Distribution/Client/ProjectConfig.o )

tests/UnitTests/Distribution/Client/ProjectConfig.hs:561:7: error:

    • Couldn't match type ‘[PathTemplate] -> PackageConfig’

                     with ‘PackageConfig’

      Expected type: Gen PackageConfig

        Actual type: Gen ([PathTemplate] -> PackageConfig)

    • In the expression:

        PackageConfig

          <$>

            (MapLast . Map.fromList

               <$>

                 shortListOf

                   10 ((,) <$> arbitraryProgramName <*> arbitraryShortToken))

          <*>

            (MapMappend . Map.fromList

               <$>

                 shortListOf

                   10 ((,) <$> arbitraryProgramName <*> listOf arbitraryShortToken))

          <*> (toNubList <$> listOf arbitraryShortToken)

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> shortListOf 5 arbitraryShortToken

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> shortListOf 5 arbitraryShortToken

          <*> shortListOf 5 arbitraryShortToken

          <*> shortListOf 5 arbitraryShortToken

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitraryFlag arbitraryShortToken

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitraryFlag arbitraryShortToken

          <*> arbitrary

          <*> arbitrary

          <*> arbitraryFlag arbitraryShortToken

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitrary

          <*> arbitraryFlag arbitraryShortToken

          <*> arbitrary

          <*> shortListOf 5 arbitrary

      In an equation for ‘arbitrary’:

          arbitrary

            = PackageConfig

                <$>

                  (MapLast . Map.fromList

                     <$>

                       shortListOf

                         10 ((,) <$> arbitraryProgramName <*> arbitraryShortToken))

                <*>

                  (MapMappend . Map.fromList

                     <$>

                       shortListOf

                         10 ((,) <$> arbitraryProgramName <*> listOf arbitraryShortToken))

                <*> (toNubList <$> listOf arbitraryShortToken)

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> shortListOf 5 arbitraryShortToken

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> shortListOf 5 arbitraryShortToken

                <*> shortListOf 5 arbitraryShortToken

                <*> shortListOf 5 arbitraryShortToken

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitraryFlag arbitraryShortToken

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitraryFlag arbitraryShortToken

                <*> arbitrary

                <*> arbitrary

                <*> arbitraryFlag arbitraryShortToken

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitrary

                <*> arbitraryFlag arbitraryShortToken

                <*> arbitrary

                <*> shortListOf 5 arbitrary

            where

                arbitraryProgramName :: Gen String

                arbitraryProgramName

                  = elements

                      [programName prog | (prog, _) <- knownPrograms (defaultProgramDb)]

      In the instance declaration for ‘Arbitrary PackageConfig’

    |

561 |       PackageConfig

    |       ^^^^^^^^^^^^^...
The command "./travis-${SCRIPT}.sh -j2" exited with 1.

The thing is, I'm not actually sure how to reproduce this failure locally. I'm testing with cabal build all --enable-tests --enable-benchmarks. What should I run to build this file?

@RyanGlScott
Copy link
Member Author

With @hvr's help (thanks!), I believe I figured out how to fix the issue in #6224 (comment). Let's see if Travis agrees with me.

@RyanGlScott
Copy link
Member Author

The AppVeyor build is segfaulting:

cabal-testsuite\PackageTests\CustomPreProcess\setup.test.hs                                                                       FAIL (18.96s)
$ "C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\runghc.exe" "--" "--ghc-arg=-i" "--ghc-arg=-no-user-package-db" "--ghc-arg=-package-db" "--ghc-arg=C:\SR\ghc-8.0.2\package.db" "--ghc-arg=-package-db" "--ghc-arg=C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "--ghc-arg=-package-id" "--ghc-arg=Cabal-3.1.0.0-inplace" "--ghc-arg=-package-id" "--ghc-arg=async-2.2.2-1efad130fd1a9aa0bcbad23df9d6e3b7f4f52833" "--ghc-arg=-package-id" "--ghc-arg=base-4.9.1.0" "--ghc-arg=-package-id" "--ghc-arg=cabal-testsuite-3.1.0.0-inplace" "--ghc-arg=-package-id" "--ghc-arg=clock-0.7.2-088dcc63def76a4701b4ae9b8e5e4e0d49b5f553" "--ghc-arg=-package-id" "--ghc-arg=exceptions-0.10.3-ad54f44e07a565173d7d1654d483c10a94670bca" "--ghc-arg=-package-id" "--ghc-arg=filepath-1.4.1.1" "--ghc-arg=-package-id" "--ghc-arg=optparse-appl_-0.14.3.0-9b251e4813f894e07f1c2aac43d0da5797a64d59" "--ghc-arg=-package-id" "--ghc-arg=process-1.4.3.0" "--ghc-arg=-package-id" "--ghc-arg=transformers-0.5.2.0" "cabal-testsuite\PackageTests\CustomPreProcess\setup.test.hs" "--builddir" "C:\projects\cabal\dist-newstyle\build\x86_64-windows\ghc-8.0.2\cabal-testsuite-3.1.0.0" "cabal-testsuite\PackageTests\CustomPreProcess\setup.test.hs" "--with-cabal" "dist-newstyle\build\x86_64-windows\ghc-8.0.2\cabal-install-3.1.0.0\x\cabal\build\cabal\cabal.exe"
stdout:
# Setup configure
+ "C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\runghc.exe" "--" "--ghc-arg=-i" "--ghc-arg=-no-user-package-db" "--ghc-arg=-package-db" "--ghc-arg=C:\SR\ghc-8.0.2\package.db" "--ghc-arg=-package-db" "--ghc-arg=C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "--ghc-arg=-package-id" "--ghc-arg=Cabal-3.1.0.0-inplace" "--ghc-arg=-package-id" "--ghc-arg=async-2.2.2-1efad130fd1a9aa0bcbad23df9d6e3b7f4f52833" "--ghc-arg=-package-id" "--ghc-arg=base-4.9.1.0" "--ghc-arg=-package-id" "--ghc-arg=cabal-testsuite-3.1.0.0-inplace" "--ghc-arg=-package-id" "--ghc-arg=clock-0.7.2-088dcc63def76a4701b4ae9b8e5e4e0d49b5f553" "--ghc-arg=-package-id" "--ghc-arg=exceptions-0.10.3-ad54f44e07a565173d7d1654d483c10a94670bca" "--ghc-arg=-package-id" "--ghc-arg=filepath-1.4.1.1" "--ghc-arg=-package-id" "--ghc-arg=optparse-appl_-0.14.3.0-9b251e4813f894e07f1c2aac43d0da5797a64d59" "--ghc-arg=-package-id" "--ghc-arg=process-1.4.3.0" "--ghc-arg=-package-id" "--ghc-arg=transformers-0.5.2.0" "C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\.\.\Setup.hs" "configure" "-vverbose +markoutput +nowrap" "--distdir" "setup.dist\work\dist" "--global" "--with-ghc" "C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--with-haddock" "C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\haddock.exe" "--enable-deterministic" "--prefix=C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\usr" "--package-db=clear" "--package-db=global" "--package-db=C:\SR\ghc-8.0.2\package.db" "--package-db=C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2"
Using Parsec parser
-----BEGIN CABAL OUTPUT-----
Configuring internal-preprocessor-test-0.1.0.0...
-----END CABAL OUTPUT-----
Dependency base -any: using base-4.9.1.0
Dependency base -any: using base-4.9.1.0
Dependency directory -any: using directory-1.3.0.0
Dependency base -any: using base-4.9.1.0
Dependency internal-preprocessor-test -any: using internal-preprocessor-test-0.1.0.0
Source component graph:
    component exe:my-custom-preprocessor
    component lib dependency exe:my-custom-preprocessor
    component exe:hello-world dependency lib
Configured component graph:
    component internal-preprocessor-test-0.1.0.0-my-custom-preprocessor
        include base-4.9.1.0
        include directory-1.3.0.0
    component internal-preprocessor-test-0.1.0.0 include base-4.9.1.0
    component internal-preprocessor-test-0.1.0.0-hello-world
        include base-4.9.1.0
        include internal-preprocessor-test-0.1.0.0
Linked component graph:
    unit internal-preprocessor-test-0.1.0.0-my-custom-preprocessor
        include base-4.9.1.0
        include directory-1.3.0.0
    unit internal-preprocessor-test-0.1.0.0
        include base-4.9.1.0
        A=internal-preprocessor-test-0.1.0.0:A
    unit internal-preprocessor-test-0.1.0.0-hello-world
        include base-4.9.1.0
        include internal-preprocessor-test-0.1.0.0
Ready component graph:
    definite internal-preprocessor-test-0.1.0.0-my-custom-preprocessor
        depends base-4.9.1.0
        depends directory-1.3.0.0
    definite internal-preprocessor-test-0.1.0.0
        depends base-4.9.1.0
        depends internal-preprocessor-test-0.1.0.0-my-custom-preprocessor
    definite internal-preprocessor-test-0.1.0.0-hello-world
        depends base-4.9.1.0
        depends internal-preprocessor-test-0.1.0.0
Using Cabal-3.1.0.0 compiled by ghc-8.0
Using compiler: ghc-8.0.2
Using install prefix: C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\usr
Executables installed in: C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\usr\bin
Libraries installed in: C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\usr\x86_64-windows-ghc-8.0.2\internal-preprocessor-test-0.1.0.0
Dynamic Libraries installed in: C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\usr\x86_64-windows-ghc-8.0.2
Private executables installed in: C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\usr\internal-preprocessor-test-0.1.0.0\x86_64-windows-ghc-8.0.2\internal-preprocessor-test-0.1.0.0
Data files installed in: C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\usr\x86_64-windows-ghc-8.0.2\internal-preprocessor-test-0.1.0.0
Documentation installed in: C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\usr\doc\x86_64-windows-ghc-8.0.2\internal-preprocessor-test-0.1.0.0
Configuration files installed in: C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\usr\etc
Using alex version 3.2.4 found on system at: C:\Users\appveyor\AppData\Roaming\cabal\bin\alex.exe
Using ar found on system at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\mingw\bin\ar.exe
No c2hs found
No cpphs found
No doctest found
Using gcc version 5.2.0 found on system at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\mingw\bin\gcc.exe
Using ghc version 8.0.2 given by user at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe
Using ghc-pkg version 8.0.2 found on system at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc-pkg.exe
No ghcjs found
No ghcjs-pkg found
No greencard found
Using haddock version 2.17.3 given by user at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\haddock.exe
Using happy version 1.19.12 found on system at: C:\Users\appveyor\AppData\Roaming\cabal\bin\happy.exe
Using haskell-suite found on system at: haskell-suite-dummy-location
Using haskell-suite-pkg found on system at: haskell-suite-pkg-dummy-location
No hmake found
Using hpc version 0.67 found on system at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\hpc.exe
Using hsc2hs version 0.68.1 found on system at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\hsc2hs.exe
No hscolour found
No jhc found
Using ld found on system at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\mingw\bin\ld.exe
Using pkg-config version 0.29.2 found on system at: C:\msys64\usr\bin\pkg-config.exe
Using runghc version 8.0.2 found on system at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\runghc.exe
Using strip version 2.25 found on system at: C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\mingw\bin\strip.exe
Using tar found on system at: C:\msys64\usr\bin\tar.exe
No uhc found
# Setup build
+ "C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\runghc.exe" "--" "--ghc-arg=-i" "--ghc-arg=-no-user-package-db" "--ghc-arg=-package-db" "--ghc-arg=C:\SR\ghc-8.0.2\package.db" "--ghc-arg=-package-db" "--ghc-arg=C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "--ghc-arg=-package-id" "--ghc-arg=Cabal-3.1.0.0-inplace" "--ghc-arg=-package-id" "--ghc-arg=async-2.2.2-1efad130fd1a9aa0bcbad23df9d6e3b7f4f52833" "--ghc-arg=-package-id" "--ghc-arg=base-4.9.1.0" "--ghc-arg=-package-id" "--ghc-arg=cabal-testsuite-3.1.0.0-inplace" "--ghc-arg=-package-id" "--ghc-arg=clock-0.7.2-088dcc63def76a4701b4ae9b8e5e4e0d49b5f553" "--ghc-arg=-package-id" "--ghc-arg=exceptions-0.10.3-ad54f44e07a565173d7d1654d483c10a94670bca" "--ghc-arg=-package-id" "--ghc-arg=filepath-1.4.1.1" "--ghc-arg=-package-id" "--ghc-arg=optparse-appl_-0.14.3.0-9b251e4813f894e07f1c2aac43d0da5797a64d59" "--ghc-arg=-package-id" "--ghc-arg=process-1.4.3.0" "--ghc-arg=-package-id" "--ghc-arg=transformers-0.5.2.0" "C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\.\.\Setup.hs" "build" "-vverbose +markoutput +nowrap" "--distdir" "setup.dist\work\dist"
Component build order: executable 'my-custom-preprocessor', library, executable 'hello-world'
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc-pkg.exe" "init" "setup.dist\work\dist\package.conf.inplace"
creating setup.dist\work\dist\build\my-custom-preprocessor
creating setup.dist\work\dist\build\my-custom-preprocessor\autogen
creating setup.dist\work\dist\build\my-custom-preprocessor\autogen
-----BEGIN CABAL OUTPUT-----
Preprocessing executable 'my-custom-preprocessor' for internal-preprocessor-test-0.1.0.0..
-----END CABAL OUTPUT-----
-----BEGIN CABAL OUTPUT-----
Building executable 'my-custom-preprocessor' for internal-preprocessor-test-0.1.0.0..
-----END CABAL OUTPUT-----
creating setup.dist\work\dist\build\my-custom-preprocessor
creating setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--make" "-no-link" "-fbuilding-cabal-package" "-O" "-static" "-outputdir" "setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-odir" "setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-hidir" "setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-stubdir" "setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-i" "-isetup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-i." "-isetup.dist\work\dist\build\my-custom-preprocessor\autogen" "-isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build\my-custom-preprocessor\autogen" "-Isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-optP-include" "-optPsetup.dist\work\dist\build\my-custom-preprocessor\autogen\cabal_macros.h" "-hide-all-packages" "-no-user-package-db" "-package-db" "C:\SR\ghc-8.0.2\package.db" "-package-db" "C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "-package-db" "setup.dist\work\dist\package.conf.inplace" "-package-id" "base-4.9.1.0" "-package-id" "directory-1.3.0.0" "-XHaskell2010" ".\MyCustomPreprocessor.hs"
[1 of 1] Compiling Main             ( MyCustomPreprocessor.hs, setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp\Main.o )
Linking...
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--make" "-fbuilding-cabal-package" "-O" "-static" "-outputdir" "setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-odir" "setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-hidir" "setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-stubdir" "setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-i" "-isetup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-i." "-isetup.dist\work\dist\build\my-custom-preprocessor\autogen" "-isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build\my-custom-preprocessor\autogen" "-Isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor-tmp" "-optP-include" "-optPsetup.dist\work\dist\build\my-custom-preprocessor\autogen\cabal_macros.h" "-hide-all-packages" "-no-user-package-db" "-package-db" "C:\SR\ghc-8.0.2\package.db" "-package-db" "C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "-package-db" "setup.dist\work\dist\package.conf.inplace" "-package-id" "base-4.9.1.0" "-package-id" "directory-1.3.0.0" "-XHaskell2010" ".\MyCustomPreprocessor.hs" "-o" "setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor.exe"
Linking setup.dist\work\dist\build\my-custom-preprocessor\my-custom-preprocessor.exe ...
creating setup.dist\work\dist\build
creating setup.dist\work\dist\build\autogen
creating setup.dist\work\dist\build\autogen
-----BEGIN CABAL OUTPUT-----
Preprocessing library for internal-preprocessor-test-0.1.0.0..
-----END CABAL OUTPUT-----
creating setup.dist\work\dist\build
Preprocessing A.pre to setup.dist\work\dist\build\A.hs
-----BEGIN CABAL OUTPUT-----
Building library for internal-preprocessor-test-0.1.0.0..
-----END CABAL OUTPUT-----
creating setup.dist\work\dist\build
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--make" "-fbuilding-cabal-package" "-O" "-outputdir" "setup.dist\work\dist\build" "-odir" "setup.dist\work\dist\build" "-hidir" "setup.dist\work\dist\build" "-stubdir" "setup.dist\work\dist\build" "-i" "-isetup.dist\work\dist\build" "-i." "-isetup.dist\work\dist\build\autogen" "-isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build\autogen" "-Isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build" "-optP-include" "-optPsetup.dist\work\dist\build\autogen\cabal_macros.h" "-this-unit-id" "internal-preprocessor-test-0.1.0.0" "-hide-all-packages" "-no-user-package-db" "-package-db" "C:\SR\ghc-8.0.2\package.db" "-package-db" "C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "-package-db" "setup.dist\work\dist\package.conf.inplace" "-package-id" "base-4.9.1.0" "-XHaskell2010" "A"
[1 of 1] Compiling A                ( setup.dist\work\dist\build\A.hs, setup.dist\work\dist\build\A.o )
Linking...
[(DefiniteUnitId (DefUnitId {unDefUnitId = UnitId "base-4.9.1.0"}),DefaultRenaming)]
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\mingw\bin\ar.exe" "-r" "setup.dist\work\dist\build\objs-672\libHSinternal-preprocessor-test-0.1.0.0.a" "@setup.dist\work\dist\build\objs-672\ar2696224464.rsp"
C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\mingw\bin\ar.exe: creating setup.dist\work\dist\build\objs-672\libHSinternal-preprocessor-test-0.1.0.0.a
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\mingw\bin\ld.exe" "-x" "-r" "-o" "setup.dist\work\dist\build\HSinternal-preprocessor-test-0.1.0.0.o" "@setup.dist\work\dist\build\ld570528145.rsp"
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc-pkg.exe" "recache" "--package-db=setup.dist\work\dist\package.conf.inplace"
creating setup.dist\work\dist\build\hello-world
creating setup.dist\work\dist\build\hello-world\autogen
creating setup.dist\work\dist\build\hello-world\autogen
-----BEGIN CABAL OUTPUT-----
Preprocessing executable 'hello-world' for internal-preprocessor-test-0.1.0.0..
-----END CABAL OUTPUT-----
-----BEGIN CABAL OUTPUT-----
Building executable 'hello-world' for internal-preprocessor-test-0.1.0.0..
-----END CABAL OUTPUT-----
creating setup.dist\work\dist\build\hello-world
creating setup.dist\work\dist\build\hello-world\hello-world-tmp
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--make" "-no-link" "-fbuilding-cabal-package" "-O" "-static" "-outputdir" "setup.dist\work\dist\build\hello-world\hello-world-tmp" "-odir" "setup.dist\work\dist\build\hello-world\hello-world-tmp" "-hidir" "setup.dist\work\dist\build\hello-world\hello-world-tmp" "-stubdir" "setup.dist\work\dist\build\hello-world\hello-world-tmp" "-i" "-isetup.dist\work\dist\build\hello-world\hello-world-tmp" "-i." "-isetup.dist\work\dist\build\hello-world\autogen" "-isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build\hello-world\autogen" "-Isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build\hello-world\hello-world-tmp" "-optP-include" "-optPsetup.dist\work\dist\build\hello-world\autogen\cabal_macros.h" "-hide-all-packages" "-no-user-package-db" "-package-db" "C:\SR\ghc-8.0.2\package.db" "-package-db" "C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "-package-db" "setup.dist\work\dist\package.conf.inplace" "-package-id" "base-4.9.1.0" "-package-id" "internal-preprocessor-test-0.1.0.0" "-XHaskell2010" ".\Hello.hs"
[1 of 1] Compiling Main             ( Hello.hs, setup.dist\work\dist\build\hello-world\hello-world-tmp\Main.o )
Linking...
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--make" "-fbuilding-cabal-package" "-O" "-static" "-outputdir" "setup.dist\work\dist\build\hello-world\hello-world-tmp" "-odir" "setup.dist\work\dist\build\hello-world\hello-world-tmp" "-hidir" "setup.dist\work\dist\build\hello-world\hello-world-tmp" "-stubdir" "setup.dist\work\dist\build\hello-world\hello-world-tmp" "-i" "-isetup.dist\work\dist\build\hello-world\hello-world-tmp" "-i." "-isetup.dist\work\dist\build\hello-world\autogen" "-isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build\hello-world\autogen" "-Isetup.dist\work\dist\build\global-autogen" "-Isetup.dist\work\dist\build\hello-world\hello-world-tmp" "-optP-include" "-optPsetup.dist\work\dist\build\hello-world\autogen\cabal_macros.h" "-hide-all-packages" "-no-user-package-db" "-package-db" "C:\SR\ghc-8.0.2\package.db" "-package-db" "C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "-package-db" "setup.dist\work\dist\package.conf.inplace" "-package-id" "base-4.9.1.0" "-package-id" "internal-preprocessor-test-0.1.0.0" "-XHaskell2010" ".\Hello.hs" "-o" "setup.dist\work\dist\build\hello-world\hello-world.exe"
Linking setup.dist\work\dist\build\hello-world\hello-world.exe ...
# hello-world
+ "C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.dist\work\.\dist\build\hello-world\hello-world"
hello from A
# Setup configure
+ "C:\projects\cabal\dist-newstyle\build\x86_64-windows\ghc-8.0.2\cabal-install-3.1.0.0\x\cabal\build\cabal\cabal.exe" "v1-configure" "-vverbose +markoutput +nowrap" "--distdir" "setup.cabal.dist\work\dist" "--global" "--with-ghc" "C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--with-haddock" "C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\haddock.exe" "--enable-deterministic" "--prefix=C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.cabal.dist\usr" "--package-db=clear" "--package-db=global" "--package-db=C:\SR\ghc-8.0.2\package.db" "--package-db=C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2"
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--numeric-version"
looking for tool ghc-pkg near compiler in C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin
found ghc-pkg in C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc-pkg.exe
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc-pkg.exe" "--version"
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--supported-languages"
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--info"
Warning: No remote package servers have been specified. Usually you would have one specified in the config file.
"C:\msys64\usr\bin\pkg-config.exe" "--version"
"C:\msys64\usr\bin\pkg-config.exe" "--list-all"
"C:\msys64\usr\bin\pkg-config.exe" "--modversion" "libalpm" "python2" "python3" "libsvn_fs_x" "libsvn_diff" "libsvn_fs" "libsvn_ra_serf" "python-3.7m" "libsvn_ra" "libsvn_client" "python-2.7" "bash-completion" "libsvn_fs_util" "libsvn_fs_fs" "python-3.7" "autoopts" "libsvn_ra_local" "libsvn_wc" "libsvn_subr" "libsvn_ra_svn" "libsvn_delta" "libsvn_repos"
-----BEGIN CABAL OUTPUT-----
Resolving dependencies...
-----END CABAL OUTPUT-----
creating setup.cabal.dist\work\dist\setup
copy .\Setup.hs to .\setup.cabal.dist\work\dist\setup\setup.hs
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--make" "-odir" ".\setup.cabal.dist\work\dist\setup" "-hidir" ".\setup.cabal.dist\work\dist\setup" "-i" "-i." "-package-db" "C:\SR\ghc-8.0.2\package.db" "-package-db" "C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "-package-id" "Cabal-3.1.0.0-inplace" ".\setup.cabal.dist\work\dist\setup\setup.hs" "-o" ".\setup.cabal.dist\work\dist\setup\setup.exe" "-threaded"
[1 of 1] Compiling Main             ( setup.cabal.dist\work\dist\setup\setup.hs, setup.cabal.dist\work\dist\setup\Main.o )
Linking .\setup.cabal.dist\work\dist\setup\setup.exe ...
C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.cabal.dist\work\dist\setup\setup.exe configure --verbose=verbose +nowrap +markoutput --builddir=setup.cabal.dist\work\dist --ghc --with-compiler=C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe --prefix=C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.cabal.dist\usr --global --package-db=clear --package-db=global --package-db=C:\SR\ghc-8.0.2\package.db --package-db=C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2 --enable-deterministic --ipid=internal-preprocessor-test-0.1.0.0 --dependency=base=base-4.9.1.0 --dependency=base=base-4.9.1.0 --dependency=base=base-4.9.1.0 --dependency=directory=directory-1.3.0.0 --disable-tests --exact-configuration --disable-benchmarks --with-ghc=C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe --with-haddock=C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\haddock.exe
Segmentation fault/access violation in generated code
stderr:
*** Exception: Command "C:\projects\cabal\dist-newstyle\build\x86_64-windows\ghc-8.0.2\cabal-install-3.1.0.0\x\cabal\build\cabal\cabal.exe" "v1-configure" "-vverbose +markoutput +nowrap" "--distdir" "setup.cabal.dist\work\dist" "--global" "--with-ghc" "C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--with-haddock" "C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\haddock.exe" "--enable-deterministic" "--prefix=C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.cabal.dist\usr" "--package-db=clear" "--package-db=global" "--package-db=C:\SR\ghc-8.0.2\package.db" "--package-db=C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" failed.
Output:
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--numeric-version"
looking for tool ghc-pkg near compiler in C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin
found ghc-pkg in C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc-pkg.exe
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc-pkg.exe" "--version"
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--supported-languages"
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--info"
Warning: No remote package servers have been specified. Usually you would have one specified in the config file.
"C:\msys64\usr\bin\pkg-config.exe" "--version"
"C:\msys64\usr\bin\pkg-config.exe" "--list-all"
"C:\msys64\usr\bin\pkg-config.exe" "--modversion" "libalpm" "python2" "python3" "libsvn_fs_x" "libsvn_diff" "libsvn_fs" "libsvn_ra_serf" "python-3.7m" "libsvn_ra" "libsvn_client" "python-2.7" "bash-completion" "libsvn_fs_util" "libsvn_fs_fs" "python-3.7" "autoopts" "libsvn_ra_local" "libsvn_wc" "libsvn_subr" "libsvn_ra_svn" "libsvn_delta" "libsvn_repos"
-----BEGIN CABAL OUTPUT-----
Resolving dependencies...
-----END CABAL OUTPUT-----
creating setup.cabal.dist\work\dist\setup
copy .\Setup.hs to .\setup.cabal.dist\work\dist\setup\setup.hs
"C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe" "--make" "-odir" ".\setup.cabal.dist\work\dist\setup" "-hidir" ".\setup.cabal.dist\work\dist\setup" "-i" "-i." "-package-db" "C:\SR\ghc-8.0.2\package.db" "-package-db" "C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2" "-package-id" "Cabal-3.1.0.0-inplace" ".\setup.cabal.dist\work\dist\setup\setup.hs" "-o" ".\setup.cabal.dist\work\dist\setup\setup.exe" "-threaded"
[1 of 1] Compiling Main             ( setup.cabal.dist\work\dist\setup\setup.hs, setup.cabal.dist\work\dist\setup\Main.o )
Linking .\setup.cabal.dist\work\dist\setup\setup.exe ...
C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.cabal.dist\work\dist\setup\setup.exe configure --verbose=verbose +nowrap +markoutput --builddir=setup.cabal.dist\work\dist --ghc --with-compiler=C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe --prefix=C:\projects\cabal\cabal-testsuite\PackageTests\CustomPreProcess\setup.cabal.dist\usr --global --package-db=clear --package-db=global --package-db=C:\SR\ghc-8.0.2\package.db --package-db=C:\projects\cabal\dist-newstyle\packagedb\ghc-8.0.2 --enable-deterministic --ipid=internal-preprocessor-test-0.1.0.0 --dependency=base=base-4.9.1.0 --dependency=base=base-4.9.1.0 --dependency=base=base-4.9.1.0 --dependency=directory=directory-1.3.0.0 --disable-tests --exact-configuration --disable-benchmarks --with-ghc=C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\ghc.exe --with-haddock=C:\ProgramData\chocolatey\lib\ghc\tools\ghc-8.0.2\bin\haddock.exe
Segmentation fault/access violation in generated code
*** unexpected failure for cabal-testsuite\PackageTests\CustomPreProcess\setup.test.hs

As far as I can tell, this is #6219. This failure also happens on master, however, so I'm going to choose to ignore this failure unless someone tells me otherwise.

@RyanGlScott RyanGlScott changed the title WIP: Implement --benchmark-options for v2-bench Implement --benchmark-options for v2-bench Sep 6, 2019
@RyanGlScott
Copy link
Member Author

I think I've resolved all outstanding issues. Ready for a proper review now!

@RyanGlScott
Copy link
Member Author

Would someone knowledgeable in the area be willing to give this a review?

@RyanGlScott
Copy link
Member Author

A gentle ping on this.

Copy link
Collaborator

@DanielG DanielG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, that looks good now.

Just one last order of buissness: adding a test. I checked and --test-option(s) has a test in cabal-testsuite/PackageTests/TestOptions/setup.test.hs so you should be able to use that as a template.

@RyanGlScott
Copy link
Member Author

I checked and --test-option(s) has a test in cabal-testsuite/PackageTests/TestOptions/setup.test.hs so you should be able to use that as a template.

I saw that too, and in fact, there's already a corresponding test called cabal-testsuite/PackageTests/BenchmarkOptions/setup.test.hs. Upon closer inspection, I realized that this isn't testing what we care about. Namely, it's testing that ./Setup bench accepts --benchmark-options, not v2-bench. Do you happen to know if there are any test cases that more closely resemble the latter?

@RyanGlScott
Copy link
Member Author

With enough persistence, I was able to cobble together a test case that exercises the cabal v2-bench --benchmark-options="..." code path. Let's see if CI likes it...

@DanielG
Copy link
Collaborator

DanielG commented Oct 11, 2019

Great! Just one more thing, can you make the test ensure that multiple --benchmark-option flags will append to the option set rather than overwrite it? A mixed invocation with --benchmark-options would also be nice. So something like --benchmark-option=1 --benchmark-options="2 3" -- benchmark-option=4 --benchmark-options="5 6" should result in 1 2 3 4 5 6.

Since we're relying a lot on genericlly derived Monoid instances and it's not clear at all from the code alone that these flags append I think we should test for that.

If you feel like it a test for v2-test for symmetry would also be good, in a separate commit please.

@RyanGlScott
Copy link
Member Author

I've modified the --benchmark-options test case to test multiple uses of the flag, and I've added a corresponding test case for --test-options.

Copy link
Collaborator

@DanielG DanielG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing the v2-test test too :)

I think we're ready for merging as soon as you address the two remaining minor comments.

This implements lots of plumbing to allow the `--benchmark-option(s)`
flags to be used with `v2-bench`, analgous to `v2-test`'s
`--test-option(s)` flag.

Fixes #6209.
@RyanGlScott
Copy link
Member Author

Hm, the CI at https://travis-ci.org/haskell-pushbot/cabal-binaries/builds/596608715 seems to have timed out. Moreover, I don't have the ability to restart it (likely due to lack of permissions for https://github.com/haskell-pushbot/cabal-binaries). What should I do?

@RyanGlScott
Copy link
Member Author

Upon further investigation, it looks like the OS X jobs are known to time out frequently. Moreover, an upstream commit has explicitly allowed these jobs to fail (see #6278). In light of this, I'm going to just land this as-is.

@RyanGlScott RyanGlScott merged commit 21762b7 into haskell:master Oct 11, 2019
@DanielG
Copy link
Collaborator

DanielG commented Oct 11, 2019

Yeah don't worry about the OSX jobs too much they're really flaky, but I see you merged it already :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add --benchmark-options option
4 participants