Skip to content

Memory leaks in cabal install #2546

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gibiansky opened this issue Apr 20, 2015 · 7 comments
Closed

Memory leaks in cabal install #2546

gibiansky opened this issue Apr 20, 2015 · 7 comments

Comments

@gibiansky
Copy link

I have a Travis CI setup that installs Ihaskell along with all of its dependencies. Travis boxes have fairly limited memory, and so the cabal install regularly fails at least 1-2 times due to out of memory errors (I am 90% sure that's what they are), before finally succeeding on the 3rd try (or sometimes even failing then).

This sounds like a memory leak to me: if by restarting the program I use less memory, then it seems like the memory could be recycled anyways. I've encountered this many times when working with cabal, and usually it's not a big deal – just rerun the cabal install and it'll work.

However it does seem a bit strange. Has anyone investigated this? Is this a well-known issue (is this a dupe? I searched in the issues list but couldn't find anything)? Is there any way I can contribute to fixing it without really knowing cabal source?

@ttuegel
Copy link
Member

ttuegel commented Apr 21, 2015

@gibiansky Can you capture one of the error messages? It would also help if you could reproduce with a profiled build. If you aren't sure how to do that, I can provide instructions. I will try to reproduce here.

BTW, what GHC version are you using? I have heard reports of increased memory usage with GHC 7.10. In any case, the profiled build will confirm that it's cabal and not GHC leaking memory.

@gibiansky
Copy link
Author

@ttuegel

I see this at least on GHC 7.8 with Cabal 1.18. Error messages are uninformative and look like this:

Failed to install ghc-parser-0.1.6.0
Last 10 lines of the build log ( /home/travis/.cabal/logs/ghc-parser-0.1.6.0.log ):
Building ghc-parser-0.1.6.0...
Preprocessing library ghc-parser-0.1.6.0...
[1 of 2] Compiling Language.Haskell.GHC.HappyParser ( src-7.8.3/Language/Haskell/GHC/HappyParser.hs, dist/build/Language/Haskell/GHC/HappyParser.o )

Because the OS just kills the cabal process for using too much memory (and when it's restarted and has to build less packages, it works fine).

Another time I've seen it on 7.10 with Cabal 1.22, this time with a better error message:

cabal: Error: some packages failed to install:
haskell-src-exts-1.16.0.1 failed during the building phase. The exception was:
ExitFailure (-9)
This may be due to an out-of-memory condition.

So I don't think this is a regression.

How do I use a profiled build? (I assume a profiled build of cabal-install and Cabal.)

If you want to see the details of the builds that occasionally cause this, look at my Travis CI configuration which installs GHC, cabal, and all the packages I need. The step that fails is travis_retry ./build.sh all which installs a whole ton of packages and dependencies (275 of them)

@23Skidoo
Copy link
Member

Failed to install ghc-parser-0.1.6.0
Last 10 lines of the build log ( /home/travis/.cabal/logs/ghc-parser-0.1.6.0.log ):
Building ghc-parser-0.1.6.0...
Preprocessing library ghc-parser-0.1.6.0...
[1 of 2] Compiling Language.Haskell.GHC.HappyParser ( src-7.8.3/Language/Haskell/GHC/HappyParser.hs, dist/build/Language/Haskell/GHC/HappyParser.o )

This could be GHC itself using too much memory and not Cabal.

@23Skidoo
Copy link
Member

One hypothesis is that you're running too many parallel GHC processes. Try limiting parallelism with install -j N.

@gibiansky
Copy link
Author

Trying it with cabal install -j 1. We'll see if it makes a difference – Travis build bots only have 2 cores anyways, so if I understand correctly cabal install -j would by default use 2 jobs instead of 1.

@ttuegel
Copy link
Member

ttuegel commented Apr 21, 2015

How do I use a profiled build? (I assume a profiled build of cabal-install and Cabal.)

Yes. It should be sufficient to run, with cabal-install < 1.22,

cabal install --enable-library-profiling --enable-executable-profiling Cabal cabal-install

or, with cabal-install >= 1.22,

cabal install --enable-profiling Cabal cabal-install

If you want to avoid modifying the user package database, you can do

mkdir cabal-prof && cd cabal-prof
cabal sandbox init

prior to running one of the above commands. This will install cabal-install in the sandbox, so you will need to run .cabal-sandbox/bin/cabal instead of just cabal. When you run the failing command, put +RTS -p -RTS before all the other arguments to cabal. That will produce a file cabal.prof that has an overview of where cabal is allocating memory; please post that file here.

@ttuegel
Copy link
Member

ttuegel commented Apr 23, 2015

I tried to replicate this myself with a profiling build of cabal-install. The maximum residency of cabal-install is 95 MB, which is reasonable for the number of packages being installed. On the other hand, GHC uses significantly more memory, upwards of 1 GB, probably more than Travis allows. I am closing this issue, as it appears to be GHC's problem.

@ttuegel ttuegel closed this as completed Apr 23, 2015
facebook-github-bot pushed a commit to facebook/duckling that referenced this issue Aug 11, 2017
Summary:
The Travis build fails.
Trying haskell/cabal#2546 to see if that helps.

Reviewed By: niteria

Differential Revision: D5612089

fbshipit-source-id: d4df127
stevana added a commit to advancedtelematic/quickcheck-state-machine that referenced this issue Sep 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants