Make macOS builds optional?

[mbrukman]

Due to Apple licensing restrictions, macOS can only be run on Apple hardware. That means that CI services such as Travis CI are limited in the number of concurrent macOS builds based on the amount of actual hardware they have on hand (or can rent remotely, if that's even possible), whereas Linux builds can be parallelized using bare-metal hardware or VMs, including cloud providers, so they can easily scale elastically to meet demand.

As a result, Travis CI very quickly runs the Linux builds, but macOS builds are queued, and make take quite a bit longer to process. In other projects, I have run into issues where macOS builds were queued for a very long time, leading to blocked code reviews or finding out much later that the build had a basic issue, which is not visible if just looking at GitHub, the Travis status is "pending". I also noticed a similar situation

I would like to propose that the Jsonnet project consider making the following changes:

• macOS builds are marked as "optional", so their delays or failures don't block builds
• a build where Linux build succeeds is marked as "passing"

This can be done easily using .travis.yml config.

This is especially the case if we add Bazel-based builds in addition to Make-based builds in issue #137.

Thoughts?

[ant31]

a build where Linux build succeeds is marked as "passing"

Would you merge a PR that fail on MacOS ? if No, both should be marked as passing

macOS builds are marked as "optional", so their delays or failures don't block builds

They don't block builds, they delay the reports. On a working open PR, a dev could open travis and see the result for Linux, if it's green it can except a green one for MacOS too.

In the worst case, linux is green / macos is red, then there's nothing to speed up iteration (unless answer to the first question is 'yes we can merge with macos failing')

leading to blocked code reviews

True, but there are 4 open PR, around the same amount merged per month, IMO at that cadence a 2 hours delay is acceptable.

An alternative could be to build macos only on the master branch(if having master red time-to-time is ok)

[mbrukman]

At the very least, we should remove one of the two macOS builds, since as I mentioned in this comment, GCC and Clang are the same thing on macOS, so testing with both is just wasting cycles, which slows down the number of builds this repo (or the Google GitHub org) can do in parallel on Travis.

The current config runs a cross-product of { Linux, macOS } × { Clang, GCC } by virtue of specifying OSes compilers as separate lists. To implement this change, we'll have to switch to an explicit matrix config and specify the variants we want:

• Linux + Clang
• Linux + GCC
• macOS + Clang

[sbarzowski]

Are you sure they are actually the same thing on Travis? It's not impossible to install real gcc on a Mac :-). It's just by default gcc is an alias for clang.

I'd still be ok with dropping macOS + GCC configuration - the likelihood of having a problem which is both os-dependent and compiler-dependent is very low in our case. And probably people on Mac prefer Clang anyway.

[mbrukman]

Are you sure they are actually the same thing on Travis? It's not impossible to install real gcc on a Mac :-). It's just by default gcc is an alias for clang.

A bit of research tells me that GCC and Clang are, indeed, the same on Travis, and after folks complained about it, Travis installed GCC separately, but with a different name so as to not conflict with the /usr/bin/gcc already present in macOS (see travis-ci/travis-ci#2423, travis-ci/travis-ci#4587).

That means that to get The Real GCC™, you need to specify it as follows:

CC=gcc-4.8
CXX=gcc-4.8

but you have to choose a specific version, because each XCode version comes with a different version of GCC (e.g., see the second issue above for getting GCC 4.9). So it appears that Travis CI is likely burning a lot of extra macOS cycles running tests on Clang twice, because I presume most people aren't aware that GCC == Clang on macOS.

All in all, it sounds like it's simpler and easier to just drop the macOS + GCC build, and just test with GCC on Linux to get multi-compiler testing.

---

Also, on a separate note, it turns out that GCC has been a symlink to Clang on macOS since OS X Lion in 2011 (source).

Given that both /usr/bin/gcc and /usr/bin/g++ exist on macOS, I'm pretty sure no regular developer (such as who is using Jsonnet) will typically build & install GCC manually on macOS, unless they're:

• developing GCC itself
• want a specific version of GCC itself
• cross-compiling for a target that Clang/LLVM does not support, but GCC does

[sbarzowski]

Recently it doesn't seem to be that much of a problem and there is a looot of other stuff to do. Closing.