x/build/env/darwin-arm64: run buildlets in a per-build VM

[toothrot]

Currently, the darwin-arm64 buildlets do not run in a clean VM for each test run. Now that QEMU has progressed a bit, we should try running them in a VM again.

[toothrot]

The state of support for virtualization of darwin/arm64 is poor. There is no solution in the near future for hosting a Mac VM on ARM64 in our current environment (VMWare on MacStadium) or in our qemu-hvm environments. This is due to a number of issues, not limited to the fact that darwin/arm64 uses an iOS like boot-process instead of a typical PC boot process.

However, the AWS buildlets may work well for us. The workflow of creating a macOS instance on AWS is roughly:

• purchase 24-hr leases of "Dedicated Hosts", marked as auto-assignable
• Create mac1.metal VMs that get auto-assigned to the host pool

This comes with some caveats. After an instance is stopped, the Dedicated Host is apparently re-imaged by AWS (in a "Pending" state for about an hour). This means, an instance per-buildlet is prohibitively expensive at our volume, as a downtime of an hour between builds would necessitate a very large pool of Dedicated Hosts for our hundreds of darwin builds per day that we do.

We could avoid some of the re-image downtime issue by re-using instances between builds. This has a bit of an issue with keeping environments pristine, but we may be able to automate some of that away with APFS filesystem snapshots, and retiring the instances on some interval.

Finally, for releaselets, we could ensure that we always use a fresh VM that hasn't been tainted by previous builds running on it.

This is less featureful than our always-fresh image approach that we have on MacStadium / VMWare, but it would allow for a somewhat-clean environment for our darwin/ARM64 builders without too much overhead.

Finally, this approach could be re-used for all of our mac builders. This would have a massive benefit of reducing the effort needed to build and maintain new macOS images for each release, which has now doubled with two processor architectures. Our existing approach for AWS AMIs using Packer should work nicely in this scenario.

[toothrot]

Documentation on automatic Dedicated Host provisioning and Releasing: https://docs.aws.amazon.com/license-manager/latest/userguide/host-resource-groups.html

[prattmic]

I'm starting prototyping with this testing out AWS Mac instances, first by creating some standard reverse builders.

[prattmic]

https://go.dev/cl/430696 added an AWS darwin-amd64 reverse builder (darwin-amd64-12-aws), which I've set up manually with a launchd service.

It actually almost "just works". All of the x/ repos seem to be passing. The main Go repo fails on a specific test, os/signal.TestDetectNohup and TestNohup: https://build.golang.org/log/bee053b56d2f0d612725e4427f5640cdad5cad34

This failure is oddly specific: the system /usr/bin/nohup binary is unhappy. We've had this issue before in #5135 inside of tmux. This is also a common problem online, though frustratingly I have yet to find a concrete description of the problem, just workarounds.

One mention is that sshd must have PAM enabled. The AWS sshd config does indeed disable PAM, so that may be related. (Though my launchd service isn't run through ssh, so it isn't clear how that is related).

I'm going to investigate running QEMU on these instances, so I'll pause investigation into os/signal for now, since it may just not be an issue in QEMU guests.

[gopherbot]

Change https://go.dev/cl/432115 mentions this issue: cmd/buildet: allow halt of macOS QEMU VMs

[gopherbot]

Change https://go.dev/cl/432396 mentions this issue: dashboard: increase hast-darwin-amd64-12-aws count

[gopherbot]

Change https://go.dev/cl/432395 mentions this issue: env/darwin: AWS darwin instances

[prattmic]

We now have three hosts running six reverse builder VMs fully set up and (almost) ready.

There is one failing test (https://build.golang.org/log/c40b5c45d0dc28318fd9ad0149efddfe39ff27d7) because an extra deprecation warning printed by bash.

Once the builds are working next steps (short and long term) will be:

• Create instances for older amd64 macOS releases.
• Create instances for arm64 macOS releases (if desired).
• Investigating putting instances behind NAT. Currently they have public IPs, but all inbound connections blocked.
• Investigate smarter scheduling like makemac. Right now we just create guests in a loop with a fixed guest OS version.