Incremental VM build fails with "Can't load Kernel binary: Invalid SDK hash."

[alexmarkov]

Repro:

1. Create a local branch and build runtime

git checkout -b tmp master
tools/build.py -a x64 -m release runtime

2. Modify VM sources, for example

diff --git a/runtime/vm/object.cc b/runtime/vm/object.cc
index 6c402b8287..1e9daeed81 100644
--- a/runtime/vm/object.cc
+++ b/runtime/vm/object.cc
@@ -6426,7 +6426,7 @@ void TypeArguments::EnumerateURIs(URIs* uris) const {
 
 const char* TypeArguments::ToCString() const {
   if (IsNull()) {
-    return "TypeArguments: null";
+    return "TypeArguments: null1";
   }
   Zone* zone = Thread::Current()->zone();
   const char* prev_cstr = OS::SCreate(zone, "TypeArguments: (H%" Px ")",

3. Commit the change into a local branch

git commit -a -m foo

4. Build runtime once again

tools/build.py -a x64 -m release runtime

Build fails with the following error

ninja -C out/ReleaseX64 -j1000 -l64 runtime
ninja: Entering directory `out/ReleaseX64'
[12/51] ACTION //runtime/bin:generate_snapshot_bin(//build/toolchain/linux:clang_x64)
FAILED: gen/runtime/bin/vm_snapshot_data.bin gen/runtime/bin/vm_snapshot_instructions.bin gen/runtime/bin/isolate_snapshot_data.bin gen/runtime/bin/isolate_snapshot_instructions.bin 
python ../../build/gn_run_binary.py compiled_action gen_snapshot --enable-experiment=non-nullable --null-safety --deterministic --snapshot_kind=core --vm_snapshot_data=gen/runtime/bin/vm_snapshot_data.bin --vm_snapshot_instructions=gen/runtime/bin/vm_snapshot_instructions.bin --isolate_snapshot_data=gen/runtime/bin/isolate_snapshot_data.bin --isolate_snapshot_instructions=gen/runtime/bin/isolate_snapshot_instructions.bin /usr/local/google/home/alexmarkov/work/dart/sdk/out/ReleaseX64/vm_platform_strong_stripped.dill
Command failed: ./gen_snapshot --enable-experiment=non-nullable --null-safety --deterministic --snapshot_kind=core --vm_snapshot_data=gen/runtime/bin/vm_snapshot_data.bin --vm_snapshot_instructions=gen/runtime/bin/vm_snapshot_instructions.bin --isolate_snapshot_data=gen/runtime/bin/isolate_snapshot_data.bin --isolate_snapshot_instructions=gen/runtime/bin/isolate_snapshot_instructions.bin /usr/local/google/home/alexmarkov/work/dart/sdk/out/ReleaseX64/vm_platform_strong_stripped.dill
output: Can't load Kernel binary: Invalid SDK hash.

[13/51] CXX obj/runtime/vm/libdart_vm_jit.object.o
ninja: build stopped: subcommand failed.
ninja -C out/ReleaseX64 -j1000 -l64 runtime done.

This is a basic development workflow which is now broken. Same error happens when switching between commits and building the VM.

The workaround is to remove out directory and make a clean build, but it is much slower.

/cc @cskau-g @mkustermann

Hi Alex,
I'm sorry you ran into this.
I believe this is the same issue Tess ran into yesterday as well. The short version is that GN bakes the sdk_hash into the generated Ninja build files, and build.py will happily reuse these even after the sdk_hash has changed (your commit) when the build files need to be updated.

I'd say this is a bit of a short-coming of our build system.
I have a CL for build.py which should address this, but there's a bit of discussion going on about the future of gn.py, so it hasn't landed yet.

In the meantime, you have two options for work-arounds:

tools/gn.py [...]  &&  tools/build.py [...]

To will make sure GN runs and regenerates the Ninja build files every time, which is a lot quicker that rebuilding the whole thing.

Or:

tools/gn.py --no-verify-sdk-hash [...]

To configure your build to disable the check all together.

Do let me know if this doesn't do it for you.

I'll see if I can't land a fix for build.py.

[mkustermann]

The workaround is to remove out directory and make a clean build, but it is much slower.

As @cskau-g says a simple tools/gn.py --no-verify-sdk-hash -ax64 -mrelease should disable the verification for all future builds - even switching between commits or anything else.

(@alexmarkov This is effectively the same problem as with tools/gn.py --bytecode - with the difference that sdk hash verification fails whereas with bytecode one sometimes just silently builds the wrong configuration)

[mkustermann]

We'll add the same flag to tools/build.py and possibly make it correctly re-run GN if needed - to avoid confusion

[alexmarkov]

(@alexmarkov This is effectively the same problem as with tools/gn.py --bytecode - with the difference that sdk hash verification fails whereas with bytecode one sometimes just silently builds the wrong configuration)

This looks different to me: bytecode is a different build mode, and the only way to select build mode with our current build scripts is to clean out and run gn.py with corresponding options. In the scenario above build mode doesn't change at all.

I don't think --no-verify-sdk-hash is a sustainable solution, as it builds Dart SDK in a mode which is different from CQ/CI bots, so building and testing your change locally doesn't mean it would build and run on CQ/CI.

[aam]

I'd say this is a bit of a short-coming of our build system.

I hope we can have the build(ninja) to rely only on the sources. Otherwise, builds mechanism of confirming whether output is up-to-date or not is useless.

[alexmarkov]

Thank you very much, @rmacnak-google !
Your change 48c2181 fixed the build errors after committing a change or switching between commits.