Add solver input to make enabling test/benchmark stanzas automatic rather than just manual

[dcoutts]

Currently in the solver we have flags and stanzas which have significant similarities.

Flags guard arbitrary additional dependencies and can either be specified up front manually or they can be variables that are solved for. Stanzas guard additional dependencies (for tests and benchmarks) and whether they are enabled or disabled is specified up front.

The task is to add an automatic mode to the stanzas case. That is, just like flags, whether a stanzas is enabled or not could be specified up front or it could be solved for.

This shouldn't be too hard since stanzas and flags are already treated very similarly in the solver, and flags already have this feature, so it should be a matter of some copying to add the feature to stanzas too.

The motivation is that some new code wants to run the solver in a way to enable testsuites and benchmarks if possible, but turn them off if not possible (and sometimes manually force enable/disable).

[kosmikus]

This is probably relatively easy, so it might be interesting for someone who wants to gain more experience in hacking on the solver.

[dcoutts]

See PR #2907

[grayjay]

Was this fixed in #3092?

[ezyang]

@grayjay I don't think so, at the very least you need to add a command line to toggle behavior.

[grayjay]

@dcoutts Do you know what else needs to be done for this issue?

[kosmikus]

I'm confused. I thought that the purpose of #3092 was indeed to enable this, but that it was never the intention to enable tests/benchmarks completely automatically. Instead, I think the idea is to run the solver in such a way that it tries to find an install plan that would be compatible with enabling tests and benchmarks. If you then later decide to enable them (which still has to be done manually), then ideally, you'll just end up installing a few additional packages, rather than a completely different set of packages.

So the motivation, as far as I understood dcoutts, was to gain a certain amount of stability between enabling and disabling tests/benchs and the respective install plans, not to actually enable them fully automatically.

[dcoutts]

Was this fixed in #3092?

Yes it was.