Warnnings on assumptions that lead to vacuous proofs

[feliperodri]

CBMC version: develop

Operating system: macOS Mojave 10.14.6

What behaviour did you expect: Is there a way to detect false assumptions and throw a warning whenever they appear in the formula? This could help users detect possible vacuous satisfaction.

What happened instead: No warrings. Verification succeeds.

Exact command line resulting in the issue: cbmc main.c
Test case:

// main.c
#include <assert.h>
#include <stdbool.h>

int main() {
  bool cond;
  __CPROVER_assume(cond);
  assert(cond);
}

[feliperodri]

cc. @markrtuttle @SaswatPadhi

[tautschnig]

We currently print such warnings when simplification/constant propagation results in assume(false) (we print aborting path on assume(false) at <location>). Some assumptions, however, will only be found to be unsatisfiable when invoking the solver back-end. It shouldn't be too hard to also test these for satisfiability. Is that what you're asking for?

[tautschnig]

As a side-note: I don't really understand how the provide code example relates to the question.

[SaswatPadhi]

Hi Michael,

Some assumptions, however, will only be found to be unsatisfiable when invoking the solver back-end. It shouldn't be too hard to also test these for satisfiability. Is that what you're asking for?

Yes, precisely.

I don't think the posted code example highlights the real problem -- we just check \forall cond . cond => cond, and it should pass without any warnings.

A __CPROVER_assume essentially prunes the state space for the remainder of the program. We want to show a warning when this state space becomes "empty" after an assumption, because then all assertions would vacuously pass. As you said, aasume(false) prints aborting ... but if the check is more sophisticated we don't see any output:

#include <assert.h>

int main() {
  int x;
  __CPROVER_assume(x > 0);
  __CPROVER_assume(x < 0); // we should get a warning for this
  assert(0 == 1);
}

As you said, this would have to be checked with the solver back-end. Also, it would be nice if the messages we show (aborting ... and whatever we decide for the case above) are warnings.

[SaswatPadhi]

BTW, the fix for this issue could also subsume #5955, so we may not need a separate warning for usage of __CPROVER_r_ok / __CPROVER_w_ok.

[martin-cs]

I don't know if it would help but goto-instrument has an option for converting asserts to assumes. I wonder if something similar would help. Converting assume(X) to assert(X) or maybe assert(!X) would allow all of the existing tools to be used to check whether the assumes were triggered / vacuous. There might be options for --cover as well, for example --cover-failed-assertions. Plus I am sure there used to be an option to insert a assert(0) at the end of main to check for whether things were passing because they are vacuous.

[SaswatPadhi]

Converting assume(X) to assert(X) or maybe assert(!X) would allow all of the existing tools to be used to check whether the assumes were triggered / vacuous.

Yes, this is a nice idea. We could insert an assert(!X) right before assume(X) -- if the assertion turns out to be valid, then the assumption would render the subsequent assertions vacuously true.

Plus I am sure there used to be an option to insert a assert(0) at the end of main to check for whether things were passing because they are vacuous.

We sometimes do that manually in our proofs, but this doesn't help us find the root cause. If we could show a warning for the exact assumption after which everything was vacuously true, that would be super helpful in debugging bad proofs.

[martin-cs]

So it seems there are three possible bits of functionality here:

1. Finding reachable but vacuous / redundant assumptions.

2. Finding unreachable assumptions.

3. Finding assumptions that have no passing traces.

4. Is probably best as an option to goto-check, as you say, inserting assert(!X) before each assumption. 2. is an easy clone of --cover assertion and could be done in the coverage code. 3. Feels could be implemented either way by adding an coverage assertion after each assumption (--cover post-assumption maybe?) or adding assert(!X) after the assumption instead of before.

I see value in all of these but I think they should be instrumentation / cover modes rather than separate analyses or warnings.

[TGWDB]

Adding to the discussion, but also want to direct to a concrete plan for what a suitable solution would be.

So it seems there are three possible bits of functionality here:

1. Finding reachable but vacuous / redundant assumptions.

I agree this seems reasonable and nice, although I'm not sure it addresses the motivation/cause that was mentioned by @SaswatPadhi :

If we could show a warning for the exact assumption after which everything was vacuously true, that would be super helpful in debugging bad proofs.

My interpretation of the above is the desire to see the point at which an unreachable block/path is created (perhaps regardless of the existence of assumptions in this path?). Clarity on this interpretation appreciated.

2. Finding unreachable assumptions.

Kind of like the above, except also subtly different. Could be nice to have, but may not address the root motivation.

3. Finding assumptions that have no passing traces.

I think, this is closest to the motivation. I would perhaps rephrase this as something like: "raise a warning/error when an assumption causes traces to no longer continue/become unreachable". I think my rephrase is different in that this is to find the assumption that causes the traces to no longer pass, where as the quoted approach would (also) warn on assumptions that are unreachable due to earlier assumptions causing them to be unreachable. (Note that either of these may be the best approach, but I want to discuss the options.)

4. Is probably best as an option to goto-check, as you say, inserting assert(!X) before each assumption. 2. is an easy clone of --cover assertion and could be done in the coverage code. 3. Feels could be implemented either way by adding an coverage assertion after each assumption (--cover post-assumption maybe?) or adding assert(!X) after the assumption instead of before.

I see value in all of these but I think they should be instrumentation / cover modes rather than separate analyses or warnings.

I agree they all have value, I'd like to know which one(s) are the best to solve the issue so a fix can be developed.