cabal-install uses a lot of memory during `stack init --solver`

[sjakobi]

Steps to reproduce:

$ git clone https://github.com/hauxir/haskell-tetris
$ cd haskell-tetris
$ time stack -v init --solver --resolver=lts-4.2

Debug output:

Version 1.0.3, Git revision dca69c661115f95fb3bb85130ec61c346277620c (dirty) (3086 commits) x86_64
2016-01-19 18:12:12.990009: [debug] Checking for project config at: /home/simon/src/haskell-tetris/stack.yaml @(stack_2YYTYNMgaWYDECOGrPBMuy:Stack.Config src/Stack/Config.hs:660:9)
2016-01-19 18:12:12.990380: [debug] Checking for project config at: /home/simon/src/stack.yaml @(stack_2YYTYNMgaWYDECOGrPBMuy:Stack.Config src/Stack/Config.hs:660:9)
2016-01-19 18:12:12.991358: [debug] Checking for project config at: /home/simon/stack.yaml @(stack_2YYTYNMgaWYDECOGrPBMuy:Stack.Config src/Stack/Config.hs:660:9)
2016-01-19 18:12:12.991507: [debug] Checking for project config at: /home/stack.yaml @(stack_2YYTYNMgaWYDECOGrPBMuy:Stack.Config src/Stack/Config.hs:660:9)
2016-01-19 18:12:12.992094: [debug] Checking for project config at: /stack.yaml @(stack_2YYTYNMgaWYDECOGrPBMuy:Stack.Config src/Stack/Config.hs:660:9)
2016-01-19 18:12:12.992243: [debug] No project config file found, using defaults. @(stack_2YYTYNMgaWYDECOGrPBMuy:Stack.Config src/Stack/Config.hs:687:13)
2016-01-19 18:12:12.993178: [debug] Run process: ldd /home/simon/.local/bin/stack @(stack_2YYTYNMgaWYDECOGrPBMuy:System.Process.Read src/System/Process/Read.hs:269:3)
2016-01-19 18:12:13.013674: [info] Using cabal packages: @(stack_2YYTYNMgaWYDECOGrPBMuy:Stack.Solver src/Stack/Solver.hs:446:5)
2016-01-19 18:12:13.013937: [info] - htetris.cabal
 @(stack_2YYTYNMgaWYDECOGrPBMuy:Stack.Solver src/Stack/Solver.hs:447:5)
2016-01-19 18:12:13.014728: [debug] Trying to decode /home/simon/.stack/build-plan-cache/x86_64-linux/lts-4.2.cache @(stack_2YYTYNMgaWYDECOGrPBMuy:Data.Binary.VersionTagged src/Data/Binary/VersionTagged.hs:55:5)
2016-01-19 18:12:13.040505: [debug] Success decoding /home/simon/.stack/build-plan-cache/x86_64-linux/lts-4.2.cache @(stack_2YYTYNMgaWYDECOGrPBMuy:Data.Binary.VersionTagged src/Data/Binary/VersionTagged.hs:64:13)
2016-01-19 18:12:13.041418: [info] Using resolver: lts-4.2 @(stack_2YYTYNMgaWYDECOGrPBMuy:Stack.Solver src/Stack/Solver.hs:316:5)
2016-01-19 18:12:13.041565: [debug] Trying to decode /home/simon/.stack/build-plan-cache/x86_64-linux/lts-4.2.cache @(stack_2YYTYNMgaWYDECOGrPBMuy:Data.Binary.VersionTagged src/Data/Binary/VersionTagged.hs:55:5)
2016-01-19 18:12:13.062709: [debug] Success decoding /home/simon/.stack/build-plan-cache/x86_64-linux/lts-4.2.cache @(stack_2YYTYNMgaWYDECOGrPBMuy:Data.Binary.VersionTagged src/Data/Binary/VersionTagged.hs:64:13)
2016-01-19 18:12:13.065219: [debug] Run process: ghc --info @(stack_2YYTYNMgaWYDECOGrPBMuy:System.Process.Read src/System/Process/Read.hs:269:3)
2016-01-19 18:12:13.684337: [debug] Run process: ghc --info @(stack_2YYTYNMgaWYDECOGrPBMuy:System.Process.Read src/System/Process/Read.hs:269:3)
2016-01-19 18:12:15.159662: [info] Using compiler: ghc-7.10.3 @(stack_2YYTYNMgaWYDECOGrPBMuy:Stack.Solver src/Stack/Solver.hs:272:13)
2016-01-19 18:12:15.159852: [info] Asking cabal to calculate a build plan... @(stack_2YYTYNMgaWYDECOGrPBMuy:Stack.Solver src/Stack/Solver.hs:339:5)
2016-01-19 18:12:15.160173: [info] Trying with packages from lts-4.2 as hard constraints... @(stack_2YYTYNMgaWYDECOGrPBMuy:Stack.Solver src/Stack/Solver.hs:341:10)
2016-01-19 18:12:20.712686: [debug] Run process: cabal --config-file=/tmp/cabal-solver30141/cabal.config install --enable-tests --enable-benchmarks --dry-run --only-dependencies --reorder-goals --max-backjumps=-1 --package-db=clear --package-db=global -v "--constraint=HsOpenSSL -fast-bignum" "--constraint=NineP -bytestring-in-base" "--constraint=brick +demos" "--constraint=cabal-rpm -old-locale" "--constraint=curl +new-base" "--constraint=hxt +network-uri" "--constraint=hxt-http +network-uri" "--constraint=hxt-relaxng +network-uri" "--constraint=jose-jwt -doctest" "--constraint=mersenne-random-pure64 -small_base" "--constraint=nix-paths +allow-relative-paths" "--constraint=pandoc +https" "--constraint=pandoc -old-locale" "--constraint=persistent-sqlite -systemlib" "--constraint=tar -old-time" "--constraint=text -integer-simple" "--constraint=time-locale-compat -old-locale" "--constraint=yaml -system-libyaml" /home/simon/src/haskell-tetris/ @(stack_2YYTYNMgaWYDECOGrPBMuy:System.Process.Read src/System/Process/Read.hs:269:3)
2016-01-19 18:12:34.367635: [info] Successfully determined a build plan with 2 external dependencies. @(stack_2YYTYNMgaWYDECOGrPBMuy:Stack.Solver src/Stack/Solver.hs:366:13)
2016-01-19 18:12:34.372972: [info] Initialising configuration using resolver: lts-4.2 @(stack_2YYTYNMgaWYDECOGrPBMuy:Stack.Init src/Stack/Init.hs:106:5)
2016-01-19 18:12:34.373081: [info] Writing configuration to file: stack.yaml @(stack_2YYTYNMgaWYDECOGrPBMuy:Stack.Init src/Stack/Init.hs:107:5)
2016-01-19 18:12:34.374329: [info] All done. @(stack_2YYTYNMgaWYDECOGrPBMuy:Stack.Init src/Stack/Init.hs:112:5)
11.70user 1.90system 0:21.45elapsed 63%CPU (0avgtext+0avgdata 1444500maxresident)k
1090984inputs+709656outputs (1308major+394172minor)pagefaults 0swaps

Using a maximum of 1.35GB of RAM is a bit too much IMO. Could be a problem with cabal of course, but I don't have time to investigate right now.

[mgsloan]

It seems highly likely that this is due to cabal-install, as time also takes into account sub-processes.

Marking this as support / upstream on a hunch, but leaving it open for further investigation.

[harendra-kumar]

Nope! Its not cabal-install and its not solver. The culprit seems to be readProcessStdout.

cabal-install is forked as a separate process so it cannot contribute to stack's memory footprint. I don't think time adds up the stats of all the child processes. anyway I ran cabal-install independently with the same command and the memory consumption was pretty low.

Then I replaced readProcessStdout with createProcess and the leak went away. So, it is somewhere in readProcessStdout. I wonder if it shows up in all consumers of readProcessStdout.

[snthibaud]

I seem to run into this issue as well. In my case 'stack init --solver' takes so much memory (>3GB) that my hard drive starts thrashing and I cannot finish solving. Is there a workaround until this issue has been fixed upstream?

[mgsloan]

For this usecase I do see cabal pegging the CPU for a few seconds in htop. time does add up usage of child processes.

Cabal-install versions with this patch might have a bit better memory use. Tweaking the dependency solver behavior with "--max-backjumps N" where N is low or perhaps with "--reorder-goals" might work, but there isn't currently a way to pass extra options to the solve.r

[harendra-kumar]

Yeah indeed it seems to be cabal. I ran solver on the lens package and saw cabal's RSS going upto 1.4GB in top output. I might have screwed up somewhere in my earlier experiment.

We can explore if we can commit a change tweaking the cabal options for a better cabal memeory footprint.

[snthibaud]

Thanks for you replies. In the end I limited the version ranges in my .cabal file and then I saw that there were no solutions to my selection of packages. A patch to one of the packages that made that package compatible with ghc 7.10 solved my issue.
I think the cabal's exhaustive search was a bit too exhaustive and something like '--max-backjumps N' might help, but I think it would be nice to have defaults that make sense as suggested by harenda-kumar (perhaps depending on available memory).

[mgsloan]

There's nothing for us to do here, since it's an upstream issue. Closing this. If you think this is in error, please comment with more info.