`extra-deps: Cabal` is ignored when `.cabal` file does not have `Cabal` dependency

[andreasabel]

I am using cabal-version: 2.4 in my .cabal file because I am using ** globbing patterns in the data-files section.

It seems that Cabal-2.4.0.1 shipped with lts-14.27 does not correctly implement these yet, at least stack build fails:

.stack-work/install/x86_64-osx/2d2cad02bc2492d52c57d57f66cc81f62e153722b88c98941d8195159454b5ac/8.6.5/share/x86_64-osx-ghc-8.6.5/MiniAgda-0.2022.4.6/test/should-fail: copyFile: does not exist (No such file or directory)
Completed 10 action(s).
'cabal copy' failed.  Error message:

--  While building package MiniAgda-0.2022.4.6 (scroll up to its section to see the error) using:
      /Users/abel/.stack/setup-exe-cache/x86_64-osx/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.5 --builddir=.stack-work/dist/x86_64-osx/Cabal-2.4.0.1 copy
    Process exited with code: ExitFailure 1

Indeed the should-fail directory does not exist, but ** exists instead (looks like a bug handling globbing pattern **):

$ ls .stack-work/install/x86_64-osx/2d2cad02bc2492d52c57d57f66cc81f62e153722b88c98941d8195159454b5ac/8.6.5/share/x86_64-osx-ghc-8.6.5/MiniAgda-0.2022.4.6/test
**      fail    succeed

However, working around this problem by bumping the Cabal version in the stack.yaml files fails:

extra-deps:
- Cabal-2.4.1.0

is seemingly ignored, and also any newer Cabal version.

Note that my .cabal file does not declare a dependency on Cabal directly, the dependency is more indirect via cabal-version: 2.4.

Maybe related:

• extra-dep is ignored when other version already in snapshot #5679

[mpilgrem]

I understand that the cabal-version field in a .cabal file just specifies the minimum version of the Cabal library needed to interpret the .cabal file - it does not affect the version of the Cabal library that the stack executable was actually compiled with. What version of the stack executable are you using? stack version 2.7.5 was compiled using lts-17.15 (GHC 8.10.4) and uses Cabal-3.2.1.0.

[mpilgrem]

I tested **-type wildcards with stack (version >= 2.7.5) (on Windows 11), and they worked as expected.

Extract from test.cabal file:

cabal-version: 2.4

data-files: data\**\*.txt

Test program:

module Main where

import Paths_test (getDataFileName)

main :: IO ()
main = do
  fp <- getDataFileName "data\\sub\\data.txt"
  contents <- readFile fp
  putStrLn contents

With test data file data\sub\data.txt.

[andreasabel]

@mpilgrem, thanks for looking into this.

I am using stack-2.7.5 (on macOS):

$ stack --version
Version 2.7.5, Git revision ba147e6f59b2da75b1beb98b1888cce97f7032b1 x86_64 hpack-0.34.4

Can you reproduce the problem as follows?

$ stack build --fast --resolver lts-14.27 MiniAgda-0.2022.3.11
...
MiniAgda> [28 of 28] Compiling Main
MiniAgda> Linking .stack-work/dist/x86_64-osx/Cabal-2.4.0.1/build/miniagda/miniagda ...
MiniAgda> copy/register
MiniAgda> /Users/abel/.stack/snapshots/x86_64-osx/d3e970dab78034f004c567b01d27995921e3c08bf0479ca586d6b82c323ccee4/8.6.5/share/x86_64-osx-ghc-8.6.5/MiniAgda-0.2022.3.11/test/should-fail: copyFile: does not exist (No such file or directory)
'cabal copy' failed.  Error message: ...

[mpilgrem]

My experience of trying to build MiniAgda (on Windows 11) is below (extracts only of output). The dependencies build but the compiler cannot parse all of the source code. I will investigate further:

> stack unpack MiniAgda
> cd MiniAgda-0.2022.3.11
> stack init
Looking for .cabal or package.yaml files to use to init the project.
Using cabal packages:
- .\

Selecting the best among 21 snapshots...

* Partially matches https://github.com/raw/commercialhaskell/stackage-snapshots/master/lts/15/16.yaml
    haskell-src-exts version 1.22.0 found
        - MiniAgda requires >=1.21 && <1.22

* Matches https://github.com/raw/commercialhaskell/stackage-snapshots/master/lts/14/27.yaml

Selected resolver: https://github.com/raw/commercialhaskell/stackage-snapshots/master/lts/14/27.yaml
Initialising configuration using resolver: https://github.com/raw/commercialhaskell/stackage-snapshots/master/lts/14/27.yaml
Total number of user packages considered: 1
Writing configuration to file: stack.yaml
All done.

> stack build
Preparing to install GHC to an isolated location.
GHC installed to C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\ghc-8.6.5\

Building all executables for `MiniAgda' once. After a successful build of all of them, only specified executables will be rebuilt.
MiniAgda        > configure (exe)
Configuring MiniAgda-0.2022.3.11...
MiniAgda        > build (exe)
Preprocessing executable 'miniagda' for MiniAgda-0.2022.3.11..
unused terminals: 3
shift/reduce conflicts:  30
reduce/reduce conflicts: 55
Building executable 'miniagda' for MiniAgda-0.2022.3.11..
[ 1 of 28] Compiling Abstract[boot]
[ 2 of 28] Compiling Collection
[ 3 of 28] Compiling Lexer
[ 4 of 28] Compiling Semiring
[ 5 of 28] Compiling SparseMatrix
[ 6 of 28] Compiling Tokens
[ 7 of 28] Compiling Util
[ 8 of 28] Compiling TreeShapedOrder
[ 9 of 28] Compiling TraceError
[10 of 28] Compiling TCM[boot]
[11 of 28] Compiling Value[boot]
[12 of 28] Compiling Warshall
[13 of 28] Compiling Polarity
[14 of 28] Compiling Abstract

src\Abstract.hs:352:5: error: parse error on input ‘deriving’
    |
352 |     deriving (Eq, Ord, Show)
    |     ^^^^^^^^

Completed 4 action(s).

--  While building package MiniAgda-0.2022.3.11 (scroll up to its section to see the error) using:
      C:\sr\setup-exe-cache\x86_64-windows\Cabal-simple_Z6RU0evB_2.4.0.1_ghc-8.6.5.exe --builddir=.stack-work\dist\e626a42b build exe:miniagda --ghc-options " -fdiagnostics-color=always"
    Process exited with code: ExitFailure 1

[mpilgrem]

The problem with Abstract.hs is (for some reason) a Haskell comment on the previous line (-- | Type -- no longer used) that is interpreted as a Haddock comment:

data Class
  = Tm      -- sort of terms, only needed for erasure
--  | Ty    -- use Set 0!  -- sort of type(constructor)s, only needed for erasure
--  | Ki      -- sort of kinds  -- use Set 0 ... for mor precision
  | Size    -- sort of sizes
  | TSize   -- sort of Size
  -- | Type    -- no longer used
    deriving (Eq, Ord, Show)

if that comment is deleted, then compilation continues to the next parse error - which is similar (a Haddock-like comment). If all of those problematic comments are deleted, the compilation then fails with:

[26 of 28] Compiling TypeChecker
[27 of 28] Compiling ToHaskell
[28 of 28] Compiling Main
Linking .stack-work\\dist\\e626a42b\\build\\miniagda\\miniagda.exe ...
MiniAgda> copy/register
C:\Users\mikep\Documents\Code\Haskell\MiniAgda-0.2022.3.11\.stack-work\install\205f41b2\share\x86_64-windows-ghc-8.6.5\MiniAgda-0.2022.3.11\test\**: CreateDirectory "\\\\?\\C:\\Users\\mikep\\Documents\\Code\\Haskell\\MiniAgda-0.2022.3.11\\.stack-work\\install\\205f41b2\\share\\x86_64-windows-ghc-8.6.5\\MiniAgda-0.2022.3.11\\test\\**": invalid argument (The filename, directory name, or volume label syntax is incorrect.)
'cabal copy' failed.  Error message:

--  While building package MiniAgda-0.2022.3.11 (scroll up to its section to see the error) using:
      C:\sr\setup-exe-cache\x86_64-windows\Cabal-simple_Z6RU0evB_2.4.0.1_ghc-8.6.5.exe --builddir=.stack-work\dist\e626a42b copy
    Process exited with code: ExitFailure 1

Possible causes of this issue:
* No module named "Main". The 'main-is' source file should usually have a header indicating that it's a 'Main' module.
* A cabal file that refers to nonexistent other files (e.g. a license-file that doesn't exist). Running 'cabal check' may point out these issues.

[mpilgrem]

I took a different approach to building MiniAgda using stack, which was successful. MiniAgda.cabal indicates that the package has been tested with GHC 9.0.2. So, I tried this stack.yaml:

resolver: lts-19.4
packages:
. -

stack build then suggests the required addition to extra-deps:, which I adopted, using:

resolver: lts-19.4
packages:
- .
extra-deps:
- haskell-src-exts-1.21.1@sha256:11d18ec3f463185f81b7819376b532e3087f8192cffc629aac5c9eec88897b35,4541

Then, stack build has no difficulty in building the package yielding (extracts only):

Building all executables for `MiniAgda' once. After a successful build of all of them, only specified executables will be rebuilt.
MiniAgda> configure (exe)
Configuring MiniAgda-0.2022.3.11...
MiniAgda> build (exe)
Preprocessing executable 'miniagda' for MiniAgda-0.2022.3.11..
unused terminals: 3
shift/reduce conflicts:  30
reduce/reduce conflicts: 55
Building executable 'miniagda' for MiniAgda-0.2022.3.11..
[ 1 of 28] Compiling Abstract[boot]
...
[28 of 28] Compiling Main
Linking .stack-work\dist\d53b6a14\build\miniagda\miniagda.exe ...
MiniAgda> copy/register
Installing executable miniagda in C:\Users\mikep\Documents\Code\Haskell\MiniAgda-0.2022.3.11\.stack-work\install\8f09109b\bin

[mpilgrem]

After some further investigation:

First, the change log of the Cabal package explains that it was not until Cabal-3.0.0.0 that the ** wildcard was supported for Setup.hs copy:

Setup.hs copy and install now work in the presence of data-files that use ** syntax (#6125).

Second, GHC did not come with Cabal >= 3.0.0.0 until GHC 8.8.1.

Third, the first LTS resolver based on GHC >= 8.8.1 was lts-15.1 (GHC 8.8.2).

Fourth, a now-deleted documentation file architecture.md (see #5056) used to explain (some of this may no longer be true):

The version of the Cabal library used to build packages

Stack builds a Setup.hs file against a version of the Cabal library, in order to build packages.

The boot version of Cabal used by GHC, which is globally available to stack, is used to compile the build-type: Simple setup executable. All packages using the same compiler and Cabal version are built with the same executable. These executables are cached in the setup-exe-cache configuration directory.

Build artefacts are placed in the corresponding .stack-work/dist/Cabal-xxxxx directory.

For packages with build-type: Custom, Stack compiles Setup.hs against the version of the Cabal library present in the snapshot (which may be overridden using extra-deps), and uses that setup executable to perform builds. This treats Cabal as any other dependency package.

The process is as follows:

1. Stack uses the boot version of Cabal to build the required version of Cabal, which is treated as though built with build-type: Simple. This will take a short while — typically a few minutes.

2. Stack uses this version of Cabal to build the setup executable from the Setup.hs file. The resulting executable is placed in the .stack-work/dist/Cabal-xxxxx directory corresponding to the boot version of Cabal — even though it was built with the snapshot version.

3. This setup executable builds the package. Again, build artefacts are placed in the .stack-work/dist/Cabal-xxxxx directory — even though it was built with the snapshot version. A resulting executable is copied to the .stack-work/install/$compiler-variant/$snapshot/$compiler-version/bin directory.

MiniAgda.cabal specifies build-type: Simple. So, resolver lts-14.27 (GHC 8.6.5, which shipped with Cabal-2.4.0.1) explains Cabal-simple_Z6RU0evB_2.4.0.1_ghc-8.6.5.exe and why ** wildcards do not work.

[mpilgrem]

So, stack can build MiniAgda with stack.yaml:

# GHC 8.8.2
resolver: lts-15.1
packages:
- .
extra-deps:
- haskell-src-exts-1.21.1@sha256:11d18ec3f463185f81b7819376b532e3087f8192cffc629aac5c9eec88897b35,4541

but subject to the strange problem of the Haddock-type comments that will not parse.

[mpilgrem]

The malformed Haddock comments giving rise to parsing problems above are likely because I have ghc-options: "$everything": -haddock set in my config.yaml file. I think this was fixed in GHC 9.0.1.

[mpilgrem]

On further investigation, stack can build MiniAgda with resolver lts-14.27, but a few changes are required. The MiniAgda.cabal needs to specify build-type: Custom and have a added custom-setup stanza:

custom-setup
  setup-depends:       base
                     , Cabal

and a simple Setup.hs needs to added in the root folder of the package (it does not have one on Hackage):

import Distribution.Simple
main = defaultMain

The stack.yaml file needs to specify that Cabal-3.0.0.0 is an extra dependency:

# GHC 8.6.5
resolver: lts-14.27
packages:
- .
extra-deps:
- Cabal-3.0.0.0
- haskell-src-exts-1.21.1@sha256:11d18ec3f463185f81b7819376b532e3087f8192cffc629aac5c9eec88897b35,4541

stack will then use Cabal-3.0.0.0 to build the setup executable from the Setup.hs and then use that setup to build MiniAgda.

[mpilgrem]

Given my investigation above and the updating of the documentation, I am going to close this issue.