[RFC] Add option to find best install plan before backjump limit

[grayjay]

I've started implementing install plan scoring, as described in #2860. cabal install now prints the score of each install plan and allows the user to find better plans with the option --max-score, where zero is perfect. cabal should find the same install plans as before when no maximum is specified. This pull request isn't ready to be merged yet. It needs documentation and tests, and I have a few questions in TODO comments. Also, this is the most basic way to expose install plan scoring, and it might be better to automate determining when to continue searching before this is merged.

I rebased onto #2916 because this change made the existing space leak worse. Then I compared performance between #2916 and the latest commit. Memory usage is similar, but the latest commit is noticeably slower when the solver runs for longer. I compiled cabal with GHC 7.10.2 and ran cabal --ignore-sandbox install --dry-run --max-backjumps -1 with the following packages and compilers:

Package	Compiler	Measurement	c1189c3 (#2916)	8689097 (#2917)
aeson	GHC 7.10.2	time	0.884 s	0.876 s
hackage-server-0.4	GHC 7.10.2	time	6.01 s	6.66 s
hackage-server-0.4	GHC 7.6.3	time	122.6 s	140.5 s
hackage-server-0.4	GHC 7.6.3	total memory (+RTS -s)	256MB	258MB

These comments that I made on the issue are still relevant:
#2860 (comment)
#2860 (comment)

[grayjay]

I rebased this to deal with conflicts and made a few minor changes that shouldn't affect the behavior.

[23Skidoo]

@kosmikus, would be nice if you could provide @grayjay with some feedback on this.

[BardurArantsson]

@grayjay Note that there are now new conflicts, probably because my dead code removal patch was merged.

[grayjay]

I resolved the conflicts. The dead code removal actually shortened the diff by about 80 lines.

[grayjay]

I tried exposing install plan scoring in a more user-friendly way. The latest commit adds a new option, --find-best-solution. The solver searches until the backjump limit and then uses the best-scoring install plan found. If there is no backjump limit, the search is exhaustive. The only benefit of --max-score now is that it can allow the solver to run faster by pruning low-scoring nodes during the whole run. --find-best-solution works by decreasing the maximum allowed score as the solver finds better and better solutions.

There are a few things I'd like to improve if this feature is added to cabal:

• The solver is much slower than on master. This branch ran for 43% longer in one long-running test.
• On master, explore creates a log from the search tree, and then logToProgress processes the log and applies the backjump limit. I think it would be easier to implement the interaction between --max-backjumps and --find-best-solution if I first moved application of the backjump limit into explore. That would take significant refactoring, though. This branch currently just stores the best install plans in the log entries, which is awkward.
• I think --max-score is unnecessary if we have --find-best-solution, but I left it in for testing.
• Currently none of the solver's log is shown without -v3. I think it would make sense to show some things at lower verbosity, such as the scores of solutions that are skipped, and whether the install plan that is used is the best or just the best before the backjump limit.

[kosmikus]

@grayjay Thanks for rebasing this. I'm hoping that I can finally take a look at this soon.

[grayjay]

@kosmikus Thanks. I just rebased on top of the module reorganization and added some comments and quickcheck tests. I also added a commit that does dynamic goal reordering based on conflict sets. It sometimes finds the best solution faster, especially when combined with --reorder-goals.

EDIT: --dynamic-goal-reordering can also prevent the solver from finding a solution quickly, so I don't know if it's any better on average.

[kosmikus]

@grayjay Thanks! I had just been experimenting with the same/a similar thing. My "dynamic-goal-reordering" branch is at https://github.com/kosmikus/cabal/tree/count-conflicts

My initial experiments with this have been very promising. There are only very few cases where it seems to be slower, and many where it seems to perform vastly better. So I'd actually be optimistic that something like this can become the new default.

I will have a look at how our approaches differ, if at all.

[grayjay]

@kosmikus I started rebasing this onto master, because I think it will be easier to split it apart when it is up to date.

[grayjay]

I'm done rebasing.

[grayjay]

I rebased onto #3594.

[grayjay]

This PR needs more work.