Skip to content

cmake: build stage3 by default #12508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Aug 29, 2022
Merged

cmake: build stage3 by default #12508

merged 4 commits into from
Aug 29, 2022

Conversation

andrewrk
Copy link
Member

This is a simplification of the cmake build script which introduces a
new "stage3" target that is built by default, which builds and installs
a stage3 zig.

It greatly simplifies the build instructions for Zig, making it conform
to the regular cmake routine, while still producing a stage3 artifact.

@andrewrk andrewrk force-pushed the cmake-stage3 branch 3 times, most recently from 66e374c to 232472f Compare August 20, 2022 05:55
@andrewrk
Copy link
Member Author

This is tripping over a stage3 bug, presumably having to do with -fstack-protector:

[nix-shell:~/dev/zig/build]$ gdb stage3/bin/zig -ex 'run test ../test/behavior.zig -I../test'
GNU gdb (GDB) 11.1
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from stage3/bin/zig...
Starting program: /home/andy/dev/zig/build/stage3/bin/zig test ../test/behavior.zig -I../test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libthread_db.so.1".
[New Thread 0x7ffff7c65640 (LWP 2995828)]
[New Thread 0x7ffff6c64640 (LWP 2995829)]
[New Thread 0x7ffff5c63640 (LWP 2995830)]
[New Thread 0x7ffff4c62640 (LWP 2995831)]
[New Thread 0x7ffff3c61640 (LWP 2995832)]
[New Thread 0x7ffff2c60640 (LWP 2995833)]
[New Thread 0x7ffff1c5f640 (LWP 2995834)]
[New Thread 0x7ffff0c5e640 (LWP 2995835)]
[New Thread 0x7fffefc5d640 (LWP 2995836)]
[New Thread 0x7fffeec5c640 (LWP 2995837)]
[New Thread 0x7fffedc5b640 (LWP 2995838)]
[New Thread 0x7fffecc5a640 (LWP 2995839)]
[New Thread 0x7fffebc59640 (LWP 2995840)]
[New Thread 0x7fffeac58640 (LWP 2995841)]
[New Thread 0x7fffe9c57640 (LWP 2995842)]
[New Thread 0x7fffe8c56640 (LWP 2995843)]

Thread 2 "zig" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7c65640 (LWP 2995828)]
0x0000000006fda4a9 in io.reader.Reader(fs.file.File,error{AccessDenied,BrokenPipe,ConnectionResetByPeer,ConnectionTimedOut,InputOutput,IsDir,NotOpenForReading,OperationAborted,SystemResources,Unexpected,WouldBlock},(function 'read')).readStruct__anon_105903 () at /home/andy/dev/zig/lib/std/io/reader.zig:312
312	            return matches;
(gdb) bt
#0  0x0000000006fda4a9 in io.reader.Reader(fs.file.File,error{AccessDenied,BrokenPipe,ConnectionResetByPeer,ConnectionTimedOut,InputOutput,IsDir,NotOpenForReading,OperationAborted,SystemResources,Unexpected,WouldBlock},(function 'read')).readStruct__anon_105903 () at /home/andy/dev/zig/lib/std/io/reader.zig:312
#1  0x0000000006fddb68 in Module.astGenFile () at Module.zig:3517
#2  0x0000000006fe33cb in Compilation.workerAstGenFile () at Compilation.zig:3281
#3  0x0000000006ded036 in ThreadPool.spawn__anon_80998.Closure.runFn () at ThreadPool.zig:83
#4  0x0000000006dc35f5 in ThreadPool.worker () at ThreadPool.zig:129
#5  0x0000000006fb90ec in Thread.callFn__anon_103802 () at /home/andy/dev/zig/lib/std/Thread.zig:393
#6  0x0000000006dc3e8c in Thread.PosixThreadImpl.spawn__anon_79394.Instance.entryFn () at /home/andy/dev/zig/lib/std/Thread.zig:663
#7  0x00007ffff7e4dd40 in start_thread () from /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libpthread.so.0
#8  0x00007ffff7d6c03f in clone () from /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libc.so.6
(gdb) 

If I use zig2 instead of stage3/bin/zig then it passes.

@BratishkaErik
Copy link
Contributor

imo this will be VERY useful for "zig" package maintainers

@BratishkaErik
Copy link
Contributor

Looks like it losts information if LLVm is linked dynamically or statically?

[195/195] Building stage3
FAILED: CMakeFiles/stage3 /home/bratishkaerik/zig/build/CMakeFiles/stage3 
cd /home/bratishkaerik/zig && /home/bratishkaerik/zig/build/zig2 build --zig-lib-dir /home/bratishkaerik/zig/lib --prefix /home/bratishkaerik/zig/build/stage3 -Dconfig_h=/home/bratishkaerik/zig/build/config.h -Denable-llvm -Denable-stage1 -Dskip-install-lib-files=false -Dtarget=native -Dcpu=baseline
error(compilation): clang failed with stderr: In file included from /home/bratishkaerik/zig/src/stage1/util.cpp:9:
In file included from /home/bratishkaerik/zig/src/stage1/stage2.h:17:
In file included from /home/bratishkaerik/zig/src/stage1/stage1.h:13:
/home/bratishkaerik/zig/src/zig_llvm.h:13:10: fatal error: 'llvm-c/Core.h' file not found

error(compilation): clang failed with stderr: In file included from /home/bratishkaerik/zig/src/stage1/target.cpp:9:
In file included from /home/bratishkaerik/zig/src/stage1/error.hpp:11:
In file included from /home/bratishkaerik/zig/src/stage1/stage2.h:17:
In file included from /home/bratishkaerik/zig/src/stage1/stage1.h:13:
/home/bratishkaerik/zig/src/zig_llvm.h:13:10: fatal error: 'llvm-c/Core.h' file not found

error(compilation): clang failed with stderr: In file included from /home/bratishkaerik/zig/src/stage1/bigfloat.cpp:8:
In file included from /home/bratishkaerik/zig/src/stage1/bigfloat.hpp:12:
In file included from /home/bratishkaerik/zig/src/stage1/error.hpp:11:
In file included from /home/bratishkaerik/zig/src/stage1/stage2.h:17:
In file included from /home/bratishkaerik/zig/src/stage1/stage1.h:13:
/home/bratishkaerik/zig/src/zig_llvm.h:13:10: fatal error: 'llvm-c/Core.h' file not found

error(compilation): clang failed with stderr: In file included from /home/bratishkaerik/zig/src/stage1/stage1.cpp:8:
In file included from /home/bratishkaerik/zig/src/stage1/stage1.h:13:
/home/bratishkaerik/zig/src/zig_llvm.h:13:10: fatal error: 'llvm-c/Core.h' file not found

error(compilation): clang failed with stderr: In file included from /home/bratishkaerik/zig/src/stage1/range_set.cpp:1:
In file included from /home/bratishkaerik/zig/src/stage1/range_set.hpp:11:
In file included from /home/bratishkaerik/zig/src/stage1/all_types.hpp:13:
/home/bratishkaerik/zig/src/zig_llvm.h:13:10: fatal error: 'llvm-c/Core.h' file not found

error(compilation): clang failed with stderr: In file included from /home/bratishkaerik/zig/src/stage1/os.cpp:8:
In file included from /home/bratishkaerik/zig/src/stage1/os.hpp:13:
In file included from /home/bratishkaerik/zig/src/stage1/error.hpp:11:
In file included from /home/bratishkaerik/zig/src/stage1/stage2.h:17:
In file included from /home/bratishkaerik/zig/src/stage1/stage1.h:13:
/home/bratishkaerik/zig/src/zig_llvm.h:13:10: fatal error: 'llvm-c/Core.h' file not found

error(compilation): clang failed with stderr: In file included from /home/bratishkaerik/zig/src/stage1/analyze.cpp:8:
In file included from /home/bratishkaerik/zig/src/stage1/analyze.hpp:11:
In file included from /home/bratishkaerik/zig/src/stage1/all_types.hpp:13:
/home/bratishkaerik/zig/src/zig_llvm.h:13:10: fatal error: 'llvm-c/Core.h' file not found

error(compilation): clang failed with stderr: In file included from /home/bratishkaerik/zig/src/stage1/parser.cpp:8:
In file included from /home/bratishkaerik/zig/src/stage1/parser.hpp:11:
In file included from /home/bratishkaerik/zig/src/stage1/all_types.hpp:13:
/home/bratishkaerik/zig/src/zig_llvm.h:13:10: fatal error: 'llvm-c/Core.h' file not found

error(compilation): clang failed with stderr: In file included from /home/bratishkaerik/zig/src/stage1/error.cpp:8:
In file included from /home/bratishkaerik/zig/src/stage1/error.hpp:11:
In file included from /home/bratishkaerik/zig/src/stage1/stage2.h:17:
In file included from /home/bratishkaerik/zig/src/stage1/stage1.h:13:
/home/bratishkaerik/zig/src/zig_llvm.h:13:10: fatal error: 'llvm-c/Core.h' file not found

error(compilation): clang failed with stderr: In file included from /home/bratishkaerik/zig/src/stage1/bigint.cpp:8:
In file included from /home/bratishkaerik/zig/src/stage1/bigfloat.hpp:12:
In file included from /home/bratishkaerik/zig/src/stage1/error.hpp:11:
In file included from /home/bratishkaerik/zig/src/stage1/stage2.h:17:
In file included from /home/bratishkaerik/zig/src/stage1/stage1.h:13:
/home/bratishkaerik/zig/src/zig_llvm.h:13:10: fatal error: 'llvm-c/Core.h' file not found

error(compilation): clang failed with stderr: In file included from /home/bratishkaerik/zig/src/stage1/ir_print.cpp:8:
In file included from /home/bratishkaerik/zig/src/stage1/all_types.hpp:13:
/home/bratishkaerik/zig/src/zig_llvm.h:13:10: fatal error: 'llvm-c/Core.h' file not found

error(compilation): clang failed with stderr: In file included from /home/bratishkaerik/zig/src/stage1/errmsg.cpp:8:
In file included from /home/bratishkaerik/zig/src/stage1/errmsg.hpp:13:
In file included from /home/bratishkaerik/zig/src/stage1/stage1.h:13:
/home/bratishkaerik/zig/src/zig_llvm.h:13:10: fatal error: 'llvm-c/Core.h' file not found

error(compilation): clang failed with stderr: In file included from /home/bratishkaerik/zig/src/stage1/astgen.cpp:8:
In file included from /home/bratishkaerik/zig/src/stage1/astgen.hpp:11:
In file included from /home/bratishkaerik/zig/src/stage1/all_types.hpp:13:
/home/bratishkaerik/zig/src/zig_llvm.h:13:10: fatal error: 'llvm-c/Core.h' file not found

error(compilation): clang failed with stderr: In file included from /home/bratishkaerik/zig/src/stage1/tokenizer.cpp:8:
In file included from /home/bratishkaerik/zig/src/stage1/tokenizer.hpp:13:
In file included from /home/bratishkaerik/zig/src/stage1/bigfloat.hpp:12:
In file included from /home/bratishkaerik/zig/src/stage1/error.hpp:11:
In file included from /home/bratishkaerik/zig/src/stage1/stage2.h:17:
In file included from /home/bratishkaerik/zig/src/stage1/stage1.h:13:
/home/bratishkaerik/zig/src/zig_llvm.h:13:10: fatal error: 'llvm-c/Core.h' file not found

error(compilation): clang failed with stderr: In file included from /home/bratishkaerik/zig/src/stage1/codegen.cpp:8:
In file included from /home/bratishkaerik/zig/src/stage1/analyze.hpp:11:
In file included from /home/bratishkaerik/zig/src/stage1/all_types.hpp:13:
/home/bratishkaerik/zig/src/zig_llvm.h:13:10: fatal error: 'llvm-c/Core.h' file not found

error(compilation): clang failed with stderr: In file included from /home/bratishkaerik/zig/src/stage1/ir.cpp:8:
In file included from /home/bratishkaerik/zig/src/stage1/astgen.hpp:11:
In file included from /home/bratishkaerik/zig/src/stage1/all_types.hpp:13:
/home/bratishkaerik/zig/src/zig_llvm.h:13:10: fatal error: 'llvm-c/Core.h' file not found

/home/bratishkaerik/zig/src/stage1/util.cpp:1:1: error: unable to build C object: clang exited with code 1
/home/bratishkaerik/zig/src/stage1/target.cpp:1:1: error: unable to build C object: clang exited with code 1
/home/bratishkaerik/zig/src/stage1/bigfloat.cpp:1:1: error: unable to build C object: clang exited with code 1
/home/bratishkaerik/zig/src/stage1/stage1.cpp:1:1: error: unable to build C object: clang exited with code 1
/home/bratishkaerik/zig/src/stage1/range_set.cpp:1:1: error: unable to build C object: clang exited with code 1
/home/bratishkaerik/zig/src/stage1/os.cpp:1:1: error: unable to build C object: clang exited with code 1
/home/bratishkaerik/zig/src/stage1/analyze.cpp:1:1: error: unable to build C object: clang exited with code 1
/home/bratishkaerik/zig/src/stage1/parser.cpp:1:1: error: unable to build C object: clang exited with code 1
/home/bratishkaerik/zig/src/stage1/error.cpp:1:1: error: unable to build C object: clang exited with code 1
/home/bratishkaerik/zig/src/stage1/bigint.cpp:1:1: error: unable to build C object: clang exited with code 1
/home/bratishkaerik/zig/src/stage1/ir_print.cpp:1:1: error: unable to build C object: clang exited with code 1
/home/bratishkaerik/zig/src/stage1/errmsg.cpp:1:1: error: unable to build C object: clang exited with code 1
/home/bratishkaerik/zig/src/stage1/astgen.cpp:1:1: error: unable to build C object: clang exited with code 1
/home/bratishkaerik/zig/src/stage1/tokenizer.cpp:1:1: error: unable to build C object: clang exited with code 1
/home/bratishkaerik/zig/src/stage1/codegen.cpp:1:1: error: unable to build C object: clang exited with code 1
/home/bratishkaerik/zig/src/stage1/ir.cpp:1:1: error: unable to build C object: clang exited with code 1
error: zig...
error: The following command exited with error code 1:
/home/bratishkaerik/zig/build/zig2 build-exe --stack 33554432 /home/bratishkaerik/zig/src/stage1.zig -lc -cflags -std=c++14 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -D_GNU_SOURCE -fvisibility-inlines-hidden -fno-exceptions -fno-rtti -Werror=type-limits -Wno-missing-braces -Wno-comment -- /home/bratishkaerik/zig/src/stage1/analyze.cpp /home/bratishkaerik/zig/src/stage1/astgen.cpp /home/bratishkaerik/zig/src/stage1/bigfloat.cpp /home/bratishkaerik/zig/src/stage1/bigint.cpp /home/bratishkaerik/zig/src/stage1/buffer.cpp /home/bratishkaerik/zig/src/stage1/codegen.cpp /home/bratishkaerik/zig/src/stage1/errmsg.cpp /home/bratishkaerik/zig/src/stage1/error.cpp /home/bratishkaerik/zig/src/stage1/heap.cpp /home/bratishkaerik/zig/src/stage1/ir.cpp /home/bratishkaerik/zig/src/stage1/ir_print.cpp /home/bratishkaerik/zig/src/stage1/mem.cpp /home/bratishkaerik/zig/src/stage1/os.cpp /home/bratishkaerik/zig/src/stage1/parser.cpp /home/bratishkaerik/zig/src/stage1/range_set.cpp /home/bratishkaerik/zig/src/stage1/stage1.cpp /home/bratishkaerik/zig/src/stage1/target.cpp /home/bratishkaerik/zig/src/stage1/tokenizer.cpp /home/bratishkaerik/zig/src/stage1/util.cpp /home/bratishkaerik/zig/src/stage1/softfloat_ext.cpp -cflags -std=c99 -O3 -- /home/bratishkaerik/zig/src/stage1/parse_f128.c /home/bratishkaerik/zig/zig-cache/o/5cb161ec87b5a777b7018cb7f6137d85/libsoftfloat.a -lc++ /home/bratishkaerik/zig/build/zigcpp/libzigcpp.a /usr/lib/llvm/14/lib64/libclang-cpp.so.14 /usr/lib64/liblldMinGW.so /usr/lib64/liblldELF.so /usr/lib64/liblldCOFF.so /usr/lib64/liblldWasm.so /usr/lib64/liblldMachO.so /usr/lib64/liblldCommon.so /usr/lib/llvm/14/lib64/libLLVM-14.so /usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0/libstdc++.so -lunwind -lc_nonshared --cache-dir /home/bratishkaerik/zig/zig-cache --global-cache-dir /home/bratishkaerik/.cache/zig --name zig -target native-native -mcpu x86_64 --pkg-begin build_options /home/bratishkaerik/zig/zig-cache/options/7qIbWK_19aGD2pQphLAZE8dY4hPHx0js-jqawt_YP01TWR7C7je-esozZ5R2qKdI --pkg-end --pkg-begin compiler_rt /home/bratishkaerik/zig/src/empty.zig --pkg-end -I /home/bratishkaerik/zig/src -I /home/bratishkaerik/zig/deps/SoftFloat-3e/source/include -I /home/bratishkaerik/zig/deps/SoftFloat-3e-prebuilt -I /usr/include -D ZIG_LINK_MODE=Static -fno-build-id --zig-lib-dir /home/bratishkaerik/zig/lib --enable-cache 
error: the following build command failed with exit code 1:
/home/bratishkaerik/zig/zig-cache/o/7ef886c3d6f9477262255b91f9618cc6/build /home/bratishkaerik/zig/build/zig2 /home/bratishkaerik/zig /home/bratishkaerik/zig/zig-cache /home/bratishkaerik/.cache/zig --zig-lib-dir /home/bratishkaerik/zig/lib --prefix /home/bratishkaerik/zig/build/stage3 -Dconfig_h=/home/bratishkaerik/zig/build/config.h -Denable-llvm -Denable-stage1 -Dskip-install-lib-files=false -Dtarget=native -Dcpu=baseline
ninja: build stopped: subcommand failed.

@BratishkaErik
Copy link
Contributor

Hm. this crashes with "-Dstatic-llvm=false" too

@BratishkaErik
Copy link
Contributor

BratishkaErik commented Aug 22, 2022

"-Denable-stage1" causes this (on master too)

@nektro
Copy link
Contributor

nektro commented Aug 22, 2022

does this change include an option to disable this behavior and have cmake build stage1 only?

@andrewrk
Copy link
Member Author

You can use e.g. make zig1 or make zig2

@andrewrk
Copy link
Member Author

stage3 has bad debug info in this branch:

[nix-shell:~/dev/zig/build]$ valgrind stage3/bin/zig build-exe ../test/standalone/hello_world/hello.zig 
==3206771== Memcheck, a memory error detector
==3206771== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==3206771== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==3206771== Command: stage3/bin/zig build-exe ../test/standalone/hello_world/hello.zig
==3206771== 
### unhandled dwarf2 abbrev form code 0x25
### unhandled dwarf2 abbrev form code 0x25
### unhandled dwarf2 abbrev form code 0x25
### unhandled dwarf2 abbrev form code 0x23
==3206771== Valgrind: debuginfo reader: ensure_valid failed:
==3206771== Valgrind:   during call to ML_(img_get)
==3206771== Valgrind:   request for range [390293489, +4) exceeds
==3206771== Valgrind:   valid image size of 273249688 for image:
==3206771== Valgrind:   "/home/andy/dev/zig/build/stage3/bin/zig"
==3206771== 
==3206771== Valgrind: debuginfo reader: Possibly corrupted debuginfo file.
==3206771== Valgrind: I can't recover.  Giving up.  Sorry.
==3206771== 

@andrewrk
Copy link
Member Author

stage3 has bad debug info in this branch:

Never mind, it's just that valgrind does not yet support DWARF v5

@andrewrk
Copy link
Member Author

Next blocker is #12625

andrewrk added a commit that referenced this pull request Aug 25, 2022
Previously, when lowering AIR instructions `wrap_errunion_payload`,
`wrap_errunion_err`, and `wrap_optional`, the LLVM backend would create
an alloca instruction to store the result, but did not set the alignment
on it. This caused UB which went undetected for a long time until we
started enabling the stack protector.

Closes #12594
Unblocks #12508
Inspires #12634

Tests passed locally:
 * test-behavior
 * test-cases
@andrewrk andrewrk force-pushed the cmake-stage3 branch 4 times, most recently from 36057ee to 0961350 Compare August 27, 2022 01:39
This is a simplification of the cmake build script which introduces a
new "stage3" target that is built by default, which builds and installs
a stage3 zig.

It greatly simplifies the build instructions for Zig, making it conform
to the regular cmake routine, while still producing a stage3 artifact.
Adds a `unused: u32 = 0` field to `Zir.Header`.

We could leave this as padding, however it triggers a Valgrind warning because
we read and write undefined bytes to the file system. This is harmless, but
it's essentially free to have a zero field here and makes the warning go away,
making it more likely that following Valgrind warnings will be taken seriously.
This makes it so that `ninja install` will work as expected.
 * CMakeLists: pass `-Dstrip` for release zig builds
 * pass -target and -mcpu to zig1. works around llvm on freebsd
   incorrectly detecting "freestanding" instead of "freebsd" for the
   native OS.
 * ci.ziglang.org is now responsible for creating aarch64-macos tarballs
   rather than Azure.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants