Broken release build `osx1015-release-tar`

[targos]

All runs have failed for the last seven days:

07:08:12 SIGN="Developer ID Application: Node.js Foundation" PKGDIR="node-v18.0.0-nightly20211128265a47d680-darwin-x64" sh tools/osx-codesign.sh
07:08:12 + set -e
07:08:12 + '[' -z 'Developer ID Application: Node.js Foundation' ']'
07:08:12 + codesign --sign 'Developer ID Application: Node.js Foundation' --entitlements tools/osx-entitlements.plist --options runtime --timestamp node-v18.0.0-nightly20211128265a47d680-darwin-x64/bin/node
07:08:12 node-v18.0.0-nightly20211128265a47d680-darwin-x64/bin/node: timestamps differ by 186 seconds - check your system clock
07:08:14 make: *** [node-v18.0.0-nightly20211128265a47d680-darwin-x64.tar] Error 1

@nodejs/jenkins-release-admins

[targos]

Example: https://ci-release.nodejs.org/job/iojs+release/8106/nodes=osx1015-release-pkg/console

[richardlau]

I poked around in the command history on release-orka-macos10.15-x64-1 and found a sudo sntp -sS time.apple.com command. I ran it and it appears to have set the time back a few minutes.

administrator@release-orka-macos10 ~ % date
Tue Nov 30 06:16:32 PST 2021
administrator@release-orka-macos10 ~ % sudo sntp -sS time.apple.com
sntp 4.2.8p10@1.3728-o Tue Mar 21 14:36:42 UTC 2017 (139~6322)
kod_init_kod_db(): Cannot open KoD db file /var/db/ntp-kod: No such file or directory
2021-11-30 06:16:39.359527 (+0800) -187.79954 +/- 125.199938 time.apple.com 17.253.6.125 s1 no-leap
administrator@release-orka-macos10 ~ % date
Tue Nov 30 06:13:37 PST 2021
administrator@release-orka-macos10 ~ %

Rerunning a failed macOS build 🤞: https://ci-release.nodejs.org/job/iojs+release/8108/nodes=osx1015-release-pkg/console

[targos]

Thanks, the incident is resolved, but I think there are two things that could improve this for the future:

• Could we enable automatic time sync on the macOS hosts?
• This issue was only noticed when we actually tried to do a manual release. Should we setup some sort of monitoring on the release job so we know as soon as possible when there is an problem?

[mhdawson]

@sxa do you know if we have automatic time sync setup on the macOS hosts for OpenJDK? If so maybe we can pull over what's done there.

[targos]

Happened againg today: https://ci-release.nodejs.org/job/iojs+release/8681/nodes=osx1015-release-tar/console

Please fix ASAP, I have a release to promote today.

[richardlau]

Synced with Apple's time server again:

administrator@release-orka-macos10 ~ % date
Tue Aug 16 01:33:03 PDT 2022
administrator@release-orka-macos10 ~ % sudo sntp -sS time.apple.com
sntp 4.2.8p10@1.3728-o Tue Mar 21 14:36:42 UTC 2017 (139~6322)
kod_init_kod_db(): Cannot open KoD db file /var/db/ntp-kod: No such file or directory
2022-08-16 01:33:09.323310 (+0700) -209.268495 +/- 139.512559 time.apple.com 17.253.6.253 s1 no-leap
administrator@release-orka-macos10 ~ % date
Tue Aug 16 01:29:46 PDT 2022
administrator@release-orka-macos10 ~ %

[targos]

Thank you. Rebuild: https://ci-release.nodejs.org/job/iojs+release/8683/

[Trott]

Optimistically closing this, but if that's not the right thing to do, please re-open.