-
Notifications
You must be signed in to change notification settings - Fork 273
Fix-up to nondet-static option [TG-4365] #2643
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Test-gen pointer bump: https://github.com/diffblue/test-gen/pull/2112 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please review whether adding a flag (and in particular one that doesn't seem to be widely used) to symbolt
is the right design choice. If yes, more changes are required as indicated below.
src/util/symbol.h
Outdated
@@ -68,6 +68,10 @@ class symbolt | |||
bool is_lvalue, is_file_local, is_extern, is_volatile, | |||
is_parameter, is_auxiliary, is_weak; | |||
|
|||
// Can a static variable be nondet-initialized? User static variables can be | |||
// but proprietary ones no (e.g., inflight_exception, clinit_already_run etc) | |||
bool is_static_nondet_init = false; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This new field needs to be serialised and de-serialiased when writing/reading goto binaries. Also text output needs to be updated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not just is_nondet_init
(or some more grammatically correct name)? (The static
information is already in is_static_lifetime
if needed).
@@ -31,7 +31,7 @@ void nondet_static( | |||
|
|||
Forall_goto_program_instructions(i_it, init) | |||
{ | |||
const goto_programt::instructiont &instruction=*i_it; | |||
const goto_programt::instructiont instruction=*i_it; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why take a copy?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In case the instruction will be overriden, I need some info from the original instruction as it's being overriden (and without making a copy it gets into a loop). I could also make a copy of it only at point when I actually need it.
@@ -54,7 +54,7 @@ void nondet_static( | |||
if(is_constant_or_has_constant_components(sym.type(), ns)) | |||
continue; | |||
|
|||
i_it=init.insert_before(++i_it); | |||
//i_it=init.insert_before(++i_it); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This line should just be removed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it definitely should have been.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PR seems incomplete at the moment.
@@ -635,6 +635,7 @@ void java_bytecode_convert_classt::convert( | |||
symbolt new_symbol; | |||
|
|||
new_symbol.is_static_lifetime=true; | |||
new_symbol.is_static_nondet_init=true; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The commit message could be more informative (e.g. Add is_static_nondet_init to symbol)
@@ -46,6 +46,10 @@ void nondet_static( | |||
if(!ns.lookup(sym.get_identifier()).is_static_lifetime) | |||
continue; | |||
|
|||
// static that can be nondet initialized? | |||
if(!ns.lookup(sym.get_identifier()).is_static_nondet_init) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could this replace the __CPROVER prefix check above? (These variables should all have static lifetime anyway?)
src/util/symbol.h
Outdated
@@ -68,6 +68,10 @@ class symbolt | |||
bool is_lvalue, is_file_local, is_extern, is_volatile, | |||
is_parameter, is_auxiliary, is_weak; | |||
|
|||
// Can a static variable be nondet-initialized? User static variables can be | |||
// but proprietary ones no (e.g., inflight_exception, clinit_already_run etc) | |||
bool is_static_nondet_init = false; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not just is_nondet_init
(or some more grammatically correct name)? (The static
information is already in is_static_lifetime
if needed).
From design point of view - we discussed this design with Peter, and it sounded like the cleanest solution. Although CI shows that this breaks how |
@tautschnig @peterschrammel Here is my branch with option (2) implemented: develop...svorenova:fixup-nondet-static-2 |
svorenova:fixup-nondet-static-2 is not an option. |
Looking at it again, I think using an ID_C comment is more appropriate. This is done similarly in comparable situations such as constant and initialization_required. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd say almost certainly rather than make a new symbol flag, either maintain the information you need (presumably a set of symbol IDs to reinit) elsewhere, or perhaps tag the symbol using a comment on its type, as e.g. already done to record a method's defining class, here: https://github.com/diffblue/cbmc/blob/develop/jbmc/src/java_bytecode/java_bytecode_convert_method.cpp#L448)
That strategy isn't clean by any stretch of the imagination, but it does at least preserve the serialized (GBF) format and thus only impacts code that uses the new information.
@@ -635,6 +635,7 @@ void java_bytecode_convert_classt::convert( | |||
symbolt new_symbol; | |||
|
|||
new_symbol.is_static_lifetime=true; | |||
new_symbol.is_static_nondet_init=true; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wouldn't, this will break GBF compatibility across this change, for example
(The long-term plan is for symbols to have an additional |
1ea7e9e
to
bd46eee
Compare
I've re-implemented this using a new comment. Let me know what you think. |
// should not be nondet-initialized, e.g., proprietary variables? | ||
if( | ||
ns.lookup(sym.get_identifier()) | ||
.type.get_bool(ID_C_no_nondet_initialization_allowed)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Presumably also check for ID_C_constant?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Constants are checked below, using is_constant_or_has_constant_components
Better! I liked your suggestion of replacing clinit methods though rather than returning all this stuff to __CPROVER_initialize, did that plan not go anywhere? |
@smowton Yes, that change will follow in a separate PR. This PR only cleans up nondet-static a bit but we would still call static initializer. The following PR will make sure that even in the static initializer wrapper we only nondet initialize variables. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@peterschrammel @tautschnig maybe you can think of a better name for "this is internal state, nondet-static shouldn't trash it"?
// static lifetime? | ||
if(!ns.lookup(sym.get_identifier()).is_static_lifetime) | ||
continue; | ||
|
||
// should not be nondet-initialized, e.g., proprietary variables? | ||
if( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Braces around multi-line if
@@ -24,6 +24,7 @@ void java_internal_additions(symbol_table_baset &dest) | |||
symbol.base_name="__CPROVER_rounding_mode"; | |||
symbol.name=CPROVER_PREFIX "rounding_mode"; | |||
symbol.type=signed_int_type(); | |||
symbol.type.set(ID_C_no_nondet_initialization_allowed, true); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe ID_C_is_internal_state
? Contrasting user static variables? No strong opinion though.
bd46eee
to
ec4d5cd
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This PR failed Diffblue compatibility checks (cbmc commit: ec4d5cd).
Build URL: https://travis-ci.com/diffblue/test-gen/builds/80742567
Status will be re-evaluated on next push.
Please contact @peterschrammel, @thk123, or @allredj for support.
Common spurious failures:
- the cbmc commit has disappeared in the mean time (e.g. in a force-push)
- the author is not in the list of contributors (e.g. first-time contributors).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Passed Diffblue compatibility checks (cbmc commit: f0dc854).
Build URL: https://travis-ci.com/diffblue/test-gen/builds/80752951
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't want to say #751 BUT ... what is the correct way of marking that a variable is "internal only" and what should that mean for different transformations?
@@ -38,10 +38,6 @@ void nondet_static( | |||
const symbol_exprt &sym=to_symbol_expr( | |||
to_code_assign(instruction.code).lhs()); | |||
|
|||
// is it a __CPROVER_* variable? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we are going to switch from marking internal / CPROVER generated variables by a CPROVER_PREFIX and instead use this new comment attribute, please could we do it across the whole code-base as I doubt that nondet-static is the only place that makes this distinction.
Also, "change how we mark internal variables" is a bigger change that should be in a PR marked "Fix-up nondet-static".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While I agree with what Martin said (that we should reconsider various __CPROVER_
prefixed symbols): 1) for several of them the main purpose of the prefix is disambiguation (avoid conflicts with user-provided symbols; 2) let's not ask for huge extra work going well beyond the purpose of the PR (this is happening in several PRs and often stalls progress) - create an issue so that it isn't forgotten instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK; I'm just want to avoid a proliferation of undocumented, slightly disjoint conventions for what are "internal" variables.
// static lifetime? | ||
if(!ns.lookup(sym.get_identifier()).is_static_lifetime) | ||
continue; | ||
|
||
// should not be nondet-initialized, e.g., proprietary variables? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Terminological nit-picking but ... what is a proprietary variable in an open source system?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like this approach. I think there are various details that can (and likely should) be improved as noted by the reviewers, but they should all be reasonably easy to address.
src/util/irep_ids.def
Outdated
@@ -665,6 +665,7 @@ IREP_ID_TWO(java_lambda_method_handles, lambda_method_handles) | |||
IREP_ID_ONE(havoc_object) | |||
IREP_ID_TWO(overflow_shl, overflow-shl) | |||
IREP_ID_TWO(C_no_initialization_required, #no_initialization_required) | |||
IREP_ID_TWO(C_no_nondet_initialization_allowed, #no_nondet_initialization_allowed) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A bit of nit-picking: "allowed" - who sets those rules? Do I get jailed if I do a nondet-init? :-) More seriously: The commit message, which is the only documentation here, provides examples, but doesn't really set out the rules. Effectively the commit message will need to set out the semantics.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do I get jailed if I do a nondet-init?
You get punished with false alarms.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll leave out the allowed
to make it clear - just no nondet-init, period, don't even try :) I will also amend the commit message in lines with my last message - marking internal variables that are not CPROVER variables, language specific internal variables.
@@ -38,10 +38,6 @@ void nondet_static( | |||
const symbol_exprt &sym=to_symbol_expr( | |||
to_code_assign(instruction.code).lhs()); | |||
|
|||
// is it a __CPROVER_* variable? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While I agree with what Martin said (that we should reconsider various __CPROVER_
prefixed symbols): 1) for several of them the main purpose of the prefix is disambiguation (avoid conflicts with user-provided symbols; 2) let's not ask for huge extra work going well beyond the purpose of the PR (this is happening in several PRs and often stalls progress) - create an issue so that it isn't forgotten instead.
The naming is more explicit on purpose because there might not only be 'internal' variables that must not be nondet-initialised. But I'm also fine with calling it differently. (since when does Github post when you press Enter... grrrrr) |
As Peter mentioned the new comment is supposed to identify variables beyond _CPROVER variables that we do not want to nondet-initialize - for example |
f0dc854
to
1ea8624
Compare
I now pushed the changes along the lines of my previous comment. These should be reflecting all the comments. I also changed how the last commit works as I learned about |
9d038ef
to
dd44837
Compare
…ized This will be used by nondet-static option and will mark internal variables that are not CPROVER variables such as language specific internal variables - in Java, for example, @inflight_exception and clinit_already_run
This sets the flag to be true for java::@inflight_exception and static initializer internal variables (including clinit_already_run)
This will also make sure that nondet-static option does not nondet- initialize these
49cd54d
to
42e3acf
Compare
On Fri, 2018-08-03 at 00:28 -0700, svorenova wrote:
As Peter mentioned the new comment is supposed to identify variables
beyond _CPROVER variables that we do not want to nondet-initialize -
for example ***@***.***_excpetion` is not a CPROVER variable and it
probably shouldn't be (it's only used in Java?).
Rounding mode is already a language specific CPROVER variable. I don't
know what the inflight exception does but if it's part of the model
that CPROVER constructs of how the program works, then maybe it should
be.
Now that I think about it, it makes sense to leave the check for
CPROVER variables in nondet-static and really consider the new
comment as a mean to mark things in the grey zone between core
CPROVER variables and user defined variables such as language
specific internal variables. Would you agree?
That's part of why I asked the question. I think the behaviour for
CPROVER variables should stay unchanged. I can see a case for adding
an extra if statement checking for an ID_C_block_nondet_static if you
want additional options of blocking which global variables get non-
det'd (although, is it better as an argument to nondet static?). This
would be making clear that this is just a flag for that one piece of
functionality and not about a general treatment of "internal"
variables.
Ofcourse, if you want something that marks a global variable as
"internal" (apart from CPROVER prefixes) and affects multiple
subsystems, then that is a different issue and should be addressed
differently. But I don't think that is the case here?
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the changes; LGTM now.
Instead of keeping both the original line that sets the value of a variable and the new line that nondet initializes it, we only keep the new one
42e3acf
to
daff1d1
Compare
I had to revert the last commit, |
@martin-cs By language specific variable I meant a variable that only exists for particular languages, not necessarily all that we support, such as inflight_exception which in Java determines whether there is an uncaught exception at any given time. I didn't mean that their value is dependent on the chosen language. |
On Fri, 2018-08-03 at 03:48 -0700, svorenova wrote:
@martin-cs By language specific variable I meant a variable that only
exists for particular languages, not necessarily all that we support,
such as inflight_exception which in Java determines whether there is
an uncaught exception at any given time. I didn't mean that their
value is dependent on the chosen language.
Yes; language specific variables can be and do have CPROVER prefixes.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Passed Diffblue compatibility checks (cbmc commit: daff1d1).
Build URL: https://travis-ci.com/diffblue/test-gen/builds/80862172
5d3ea03 Merge pull request diffblue#2693 from zhixing-xu/fix_rw_range_upper 2b40338 Update test.desc c6d0427 Merge pull request diffblue#2674 from diffblue/msvc-link 318474f Merge pull request diffblue#2678 from romainbrenguier/feature/extend-builtin-functions-part3 c63030f Regression test for goto-link personality 64fedd0 Microsoft LINK personality d9f9dd9 Merge pull request diffblue#2688 from tautschnig/concat-dir-file 99946fc Merge pull request diffblue#2675 from diffblue/goto-cl-echo-file 252d857 Merge pull request diffblue#2687 from tautschnig/fo-directory 0d7ebd5 Fixed a problem where the rw_set range's upper bound not set correctly 29eab32 goto-cl: Fail invocation of trying to compile multiple files to non-directory 7f587c4 Fix concat_dir_file for Windows and unit-test it a44468b Document string_builtin_function::eval ecc0e43 Document eval_is_upper_case a345d30 Document eval_string 58307a0 Tests for String.toUpperCase 71de2e3 Tests for String.toLowerCase 579d00c Remove redundant function application ID check e2961b5 Add builtin class for string_to_upper_case fe9071b Make to_upper_case work for Latin-1 supplement 53ccd3f Refactor string_to_upper_case be477c9 Remove assumptions that input char are < 0x100 836cbad Extract an is_upper_case function 1137ffd Improve documentation of add_axioms_for_to_lower_case a71f2c0 Implement builtin string_to_lower_case function fae6a48 Merge pull request diffblue#2467 from tautschnig/vs-except 24de513 Merge pull request diffblue#2474 from tautschnig/vs-identifier e14f2f2 Merge pull request diffblue#2685 from diffblue/clcache-again 08698cc Merge pull request diffblue#2681 from diffblue/remove-aig d42020b remove --aig option cd4a163 AWS codebuild windows: set clcache base directory 1ba928c cleanup unnecessary path from configuration file 8c4801b Refactor add_axioms_for_to_lower_case 4291232 Merge pull request diffblue#2686 from diffblue/buildspec-apt-cache 3adb717 AWS codebuild: cache apt lists and packages 0efb169 remove AIGs 3a9c825 Merge pull request diffblue#2682 from diffblue/fix-clcache b9b5660 Merge pull request diffblue#2483 from tautschnig/vs-java-parameters a66ab1e CL prints the name of the file that's compiled onto stdout 0698a5f Merge pull request diffblue#2673 from diffblue/goto-cl-Fo 75855bf Java front-end: remove unused parameters 4df2187 debugging output to resolve seg fault 5bc7456 goto-cl: /Fo can set an output directory a43e4fa add is_directory to file_util.h 4ad91fb Codebuild for windows: set up cache path properly effb01b Merge pull request diffblue#2641 from diffblue/typedef-type 5ef2802 Merge pull request diffblue#2679 from tautschnig/version-string 2efea52 Refine test patterns to avoid spurious matches aa7ebbc Merge pull request diffblue#2672 from diffblue/goto-cc-multiple-source-files aaea781 Merge pull request diffblue#2671 from diffblue/fix_get_base_name 8b51faf fix get_base_name 694daaf gcc mode: error in case multiple files are given with -c and -o 09fdca3 Merge pull request diffblue#2643 from svorenova/fixup-nondet-static d42054a Merge pull request diffblue#2669 from diffblue/spurious-cover-test b391c9e introduce typedef_type in the C frontend 114030b Merge pull request diffblue#2664 from romainbrenguier/feature/extend-builtin-functions-part2 46f6231 cbmc test no longer uses --cover daff1d1 Make nondet-static replace lines in CPROVER_init 1bca129 Merge pull request diffblue#2665 from tautschnig/gcc-conditional-stmt 31366ad Tests for StringBuilder.setCharAt f4285e7 Add builtin support for string_set_char 2a8ea0f Better specify out-of-bounds case for set_char 9415a24 Refactor add_axioms_for_set_char 9762886 Refactor string_concat_char builtin function b62bf01 Make nondet-static check for ID_C_no_nondet_initialization a50562e Mark java.lang.String.Literals with ID_C_constant b3c08d3 Mark internal Java variables with ID_C_no_nondet_initialization f2dc978 Add a new comment to mark variables that should not be nondet initialized da7966c Merge pull request diffblue#2645 from mmuesly/feature_test_posix_memalign cb4a340 Merge pull request diffblue#2647 from diffblue/cleanout-jar-filet 2b27a2d Merge pull request diffblue#2622 from martin-cs/feature/context-sensitive-ait-merge-2 00b26a8 Adds a test for posix_memalign in stdlib.c 0f5a057 Merge pull request diffblue#2667 from tautschnig/slicer-cleanup 16e6462 Remove no-longer-used ifdef 96d345b Clean GCC conditional expressions in right-hand sides of declarations 086c266 Merge pull request diffblue#2651 from smowton/smowton/feature/unroll-enum-clone-loops f69b244 Merge pull request diffblue#2625 from smowton/smowton/feature/value-set-accuracy 730b3e2 Merge pull request diffblue#2655 from romainbrenguier/feature/extend-builtin-functions c4569dd remove java_class_loader_limitt from jar_filet e0f954d Merge pull request diffblue#2659 from smowton/smowton/fix/cmake-third-time-is-charm 7f547b1 Merge pull request diffblue#2656 from smowton/smowton/fix/testsuite-name 3a1593b Tests for String.valueOf(int) 7414d76 Add builtin support for string_of_int 0b44c89 Add version of make_string accepting iterators f81f082 Rename function to add_axioms_for_string_of_int 9a2a7c2 Switch version.cpp from a rule product to a byproduct 943d60c Value set: handle with, array and struct expressions more accurately 4f158a3 Mark regression tests as expecting failure for symex driven loading c9a53f9 Add regression tests for changes to JBMC enumeration support a985eae Interpreter: deal with member-of-constant-struct expressions 361469b Change source location of jump target in {table|lookup}switch 6270968 java-unwind-enum-static: also unwind clone loop in Enum.values() 714de0d Symex: expose call stack to unwinding decision making fb239ef Fix jbmc-generic-symex-driven-lazy-loading test name 28ba192 Strengthen the invariant on what are acceptable function calls. 773bc86 Convert various comments, asserts and throws into invariants. e65f027 Add comments to the abstract interpreter interface. 1fe0796 Convert various older domains to use the more recent ait API. afe32b7 Refactor the methods that access "the abstract domain at a location". aa743b3 Remove unused exception name from catch statement 5703504 Remove unused parameter identifier git-subtree-dir: cbmc git-subtree-split: 5d3ea03
This PR makes
nondet-static
option behave in a bit cleaner way: