FreeBSD CI tests disabled due to using more memory than sr.ht provides

[andrewrk]

This is one of the blocking issues for #1759

[andrewrk]

See related discussion and statement from Drew DeVault here: #3197

[andrewrk]

Here's some output from manually running commands on sr.ht:

[build@build /usr/home/build/zig/build]$ release/bin/zig build test-std
Code Generation [315/16844] fs.file.File.stat...The following command terminated unexpectedly:
/usr/home/build/zig/build/release/bin/zig test /usr/home/build/zig/lib/std/std.zig --test-name-prefix std-native-Debug--multi  --cache-dir /usr/home/build/zig/zig-cache --name test --override-lib-dir /usr/home/build/zig/lib 

[build@build /usr/home/build/zig/build]$ release/bin/zig build test-compare-output
./zig-cache/o/ePRh5kNcnWwdgJLiKdDdbtOorcnkvNkcNLT-8oMnd52r9s3wN4FAgdueMeCuooZj/source.zig:1:11: error: C import failed: out of memory


[build@build /usr/home/build/zig/build]$ release/bin/zig build test-standalone
PASS build test/standalone/hello_world/hello.zig (Debug)
PASS build test/standalone/hello_world/hello.zig (ReleaseSafe)
PASS build test/standalone/hello_world/hello.zig (ReleaseFast)
PASS build test/standalone/hello_world/hello.zig (ReleaseSmall)
/usr/home/build/zig/test/standalone/hello_world/hello_libc.zig:1:11: error: C import failed: out of memory


[build@build /usr/home/build/zig/build]$ release/bin/zig build test-translate-c
unable to parse C file: out of memory


Test 327/556 compile-error @cImport with bogus include...
=========== Expected compile error: ============
.h:1:10: note: 'bogus.h' file not found
================= Full output: =================
./zig-cache/o/7JsRoo4WcAjfi5v-IMGMSRDmnGwkzySji6QOFlKe1LRkDoARYndwL2uSjdjLfQYn/tmp.zig:1:11: error: C import failed: out of memory
const c = @cImport(@cInclude("bogus.h"));

docgen example code 255/263..../docgen_tmp/test.zig:1:11: error: C import failed: out of memory
const c = @cImport({
          ^

It appears that a simple "hello world" @cImport is causing "out of memory". This is the example from the language reference:

const c = @cImport({
    // See https://github.com/ziglang/zig/issues/515
    @cDefine("_NO_CRT_STDIO_INLINE", "1");
    @cInclude("stdio.h");
});
pub fn main() void {
    _ = c.printf("hello\n");
}

[build@build /usr/home/build/zig/build]$ ./zig build-exe hello.zig -lc
Semantic Analysis [377/616] ./hello.zig:1:11: error: C import failed: out of memory
const c = @cImport({
          ^

I'm not sure how to measure peak RSS on this freebsd system (time -v gives me "bash: -v: command not found"). But this should be a pretty low memory task, so more investigation is needed.

[mikdusan]

unable to reproduce ./zig build-exe hello.zig -lc

• tried a myriad of local build debug/release/ZIG_STATIC=ON variations
• tried ziglang.org package zig-freebsd-x86_64-0.5.0+a090a5e3b
• /usr/bin/time -l reports max RSS in the ~200MB to ~230MB range depending on zig binary type

I'm on: FreeBSD freebsd121 12.1-RELEASE-p2 FreeBSD 12.1-RELEASE-p2 GENERIC amd64

could it be environmental? what does limit or ulimit -a report?

freebsd121▸ ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          33554432
-s: stack size (kbytes)             524288
-c: core file size (blocks)         unlimited
-m: resident set size (kbytes)      unlimited
-l: locked-in-memory size (kbytes)  64
-u: processes                       10328
-n: file descriptors                175473
-b: socket buffer size (bytes)      unlimited
-v: virtual memory size (kbytes)    unlimited
-p: pseudo-terminals                unlimited
-w: swap size (kbytes)              unlimited
-k: kqueues                         unlimited
-N 14:                              unlimited

[andrewrk]

Let me see if I can get you access to the sr.ht build servers. Then you could click "resubmit build", wait for it to fail, then it will let you ssh in and mess around.

[andrewrk]

This may not be related to out of memory. (Thanks to @mikdusan for diagnosing)

I believe we are hitting this code path:

zig/src/zig_clang.cpp

Lines 2102 to 2105 in fb6b94f

	// Early failures in LoadFromCommandLine may return with ErrUnit unset.
	if (!ast_unit && !err_unit) {
	return nullptr;
	}

I believe the diags object there has errors and this code is failing to access them. Once we access them, I believe we would see something like this:

error: unknown target CPU 'athlon-xp'
note: valid target CPU values are: nocona, core2, penryn, bonnell, atom, silvermont, slm, goldmont, goldmont-plus, tremont,
      nehalem, corei7, westmere, sandybridge, corei7-avx, ivybridge, core-avx-i, haswell, core-avx2, broadwell, skylake,
      skylake-avx512, skx, cascadelake, cannonlake, icelake-client, icelake-server, knl, knm, k8, athlon64, athlon-fx,
      opteron, k8-sse3, athlon64-sse3, opteron-sse3, amdfam10, barcelona, btver1, btver2, bdver1, bdver2, bdver3, bdver4,
      znver1, x86-64

This is not actually zig's fault. The freebsd environment on sr.ht is broken; reporting the wrong CPU. I filed https://todo.sr.ht/~sircmpwn/builds.sr.ht/263

However the incorrect "out of memory" error message is zig's fault for not using libclang API correctly; translate-c needs to be improved to properly report these error messages.

[mikdusan]

zig/src/zig_clang.cpp

Lines 2102 to 2105 in fb6b94f

	// Early failures in LoadFromCommandLine may return with ErrUnit unset.
	if (!ast_unit && !err_unit) {
	return nullptr;
	}

we actually get past that part -- err_unit is valid. The following begin/end loop establishes range of private clang::ASTUnit.StoredDiagnostics which is empty. However, private clang::ASTUnit.FailedParseDiagnostics contains the warning we see clang driver emit.

...and looking into clang, they may have a bug and frankly I don't think clang driver proper even uses this interface. It looks like the primary user is CIndex and their diagnostic printing uses same logic as ours. So this answers the question of how clang driver is able to show the glorious error and not us -- because apparently it doesn't use clang::ASTUnit::LoadFromCommandLine. Possible bug; this line can be cut?

https://github.com/llvm/llvm-project/blob/llvmorg-9.0.1/clang/lib/Frontend/ASTUnit.cpp#L1812

Right now I've imported ASTUnit.hpp and changed to public just to get further into a solve. There's some other issues here. I'll drop a link to my branch shortly.

update: so getting past the private issue, we end up with 3 diags that do not have source file location information:

INDEX	LEVEL	MESSSAGE
0	Warning	c: 'linker' input unused
1	Error	unknown target CPU 'bogus'
2	Note	valid target CPU values are: nocona, core2, penryn, bonnell, atom, silvermont, slm, goldmont, goldmont-plus, tremont, nehalem, corei7, westmere, sandybridge, corei7-avx, ivybridge, core-avx-i, haswell, core-avx2, broadwell, skylake, skylake-avx512, skx, cascadelake, cooperlake, cannonlake, icelake-client, icelake-server, knl, knm, k8, athlon64, athlon-fx, opteron, k8-sse3, athlon64-sse3, opteron-sse3, amdfam10, barcelona, btver1, btver2, bdver1, bdver2, bdver3, bdver4, znver1, znver2, x86-64

So with this hack in place:
mikdusan@dded8bb

I get the following output:

[LemonBoy]

This is not actually zig's fault. The freebsd environment on sr.ht is broken; reporting the wrong CPU. I filed https://todo.sr.ht/~sircmpwn/builds.sr.ht/263

The wrong cpu problem is qemu's fault, see this bug report.

https://github.com/llvm/llvm-project/blob/llvmorg-9.0.1/clang/lib/Frontend/ASTUnit.cpp#L1812

That line makes sense to me, you're moving FailedParseDiagnostics into StoredDiagnostics to be able to iterate over those using the same machinery.

Edit: Oh your wording confused me, StoredDiagnostics is non-empty and gets overwritten by an empty FailedParseDiagnostics ... This is definitely a bug!

However, private clang::ASTUnit.FailedParseDiagnostics contains the warning we see clang driver emit.

Shooting a mail to the LLVM folks to ask if this is a bug in the API or a problem with the way they're used is the way to go IMO.

[mikdusan]

yes, the swap takes what we could access with machinery and stashes it away to a private and we're left with nothing. I verified this with lldb.

[andrewrk]

1/2 issues resolved:

Through some testing, it looks like FreeBSD boots up with -cpu host now, so the workaround is no longer necessary. I'll get the change deployed. Cheers!

I still want to get this other problem fixed. We need access to those other kind of diagnostics. Opening new issue to track: #4455

[andrewrk]

Thanks to @aveao this is no longer an issue. We still have stage1 resource utilization issues but we are well underway with self-hosted to solve that. The FreeBSD CI runner is now doing a full zig build test!