utils/update-checkout error from git checkout icu maint/maint-69

[vanvoorden]

Description

https://github.com/apple/swift/blob/main/docs/HowToGuides/GettingStarted.md#cloning-the-project

Hi! I just tried following the cloning-the-project steps on a new (vanilla) swift checkout (from latest main). I'm seeing some errors on console:

Traceback (most recent call last):
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/Users/rick/Developer/swift-project/swift/utils/update_checkout/update_checkout/update_checkout.py", line 376, in obtain_additional_swift_sources
    shell.run(['git', '--git-dir',
  File "/Users/rick/Developer/swift-project/swift/utils/swift_build_support/swift_build_support/shell.py", line 257, in run
    raise eout
Exception: ['git', '--git-dir', '/Users/rick/Developer/swift-project/icu/.git', '--work-tree', '/Users/rick/Developer/swift-project/icu', 'checkout', 'maint/maint-69']

I tried running cloning-the-project again and see similar errors:

[icu]                                   + git checkout maint/maint-69
[icu]                                   error: Your local changes to the following files would be overwritten by checkout:
[icu]                                   	.ci-builds/.azure-exhaustive-tests.yml
[icu]                                   	.ci-builds/.azure-pipelines.yml
[icu]                                   	.ci-builds/data-filter.json
...
[icu]                                   error: The following untracked working tree files would be overwritten by checkout:
[icu]                                   	.appveyor.yml
[icu]                                   	.ci-builds/.azure-valgrind.yml
[icu]                                   	.github/workflows/icu_ci.yml
...
[icu]                                   	ic

Anyone seen these errors before or know the steps to work around and test building swift locally? I'm on Apple Silicon BTW. Thanks!

Reproduction

mkdir swift-project
cd swift-project
git clone https://github.com/apple/swift.git swift
cd swift
utils/update-checkout --clone-with-ssh

Expected behavior

utils/update-checkout succeeds and installs deps with no errors.

Environment

uname -a
Darwin C97RX2X127.local 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:55:06 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6020 arm64

sw_vers
ProductName:		macOS
ProductVersion:		14.2.1
BuildVersion:		23C71

git --version
git version 2.39.3 (Apple Git-145)

git remote -v
origin	[email protected]:apple/swift.git (fetch)
origin	[email protected]:apple/swift.git (push)
git branch
* main
git rev-parse HEAD
e7475c89edb66a26d762c6e9a24c9bdb1a80b87a

Additional information

No response

[vanvoorden]

mkdir swift-project
cd swift-project
git clone [email protected]:apple/swift.git swift --branch swift-5.9.2-RELEASE
cd swift
utils/update-checkout --clone-with-ssh

Hmm… I just tried cloning swift-5.9.2-RELEASE and I did not see an error from update-checkout…

[vanvoorden]

git diff
diff --git a/utils/update_checkout/update-checkout-config.json b/utils/update_checkout/update-checkout-config.json
index 3cd73079ace..0e284599338 100644
--- a/utils/update_checkout/update-checkout-config.json
+++ b/utils/update_checkout/update-checkout-config.json
@@ -190,7 +190,7 @@
                 "swift-nio-ssl": "2.15.0",
                 "swift-experimental-string-processing": "swift/release/5.10",
                 "curl": "curl-8_4_0",
-                "icu": "maint/maint-69",
+                "icu": "release-65-1",
                 "libxml2": "v2.11.5",
                 "zlib": "v1.3"
             }
utils/update-checkout --clone-with-ssh

Hacking maint/maint-69 back to release-65-1 on main leads to the same error for me…

[AnthonyLatsis]

Exception: ['git', '--git-dir', '/Users/rick/Developer/swift-project/icu/.git', '--work-tree', '/Users/rick/Developer/swift-project/icu', 'checkout', 'maint/maint-69']

Is this all it prints—no further details on the exception?

---

Try utils/update-checkout --clone-with-ssh --skip-repository icu as a workaround. I think the ICU checkout is for non-Darwin platforms.

[vanvoorden]

Try utils/update-checkout --clone-with-ssh --skip-repository icu as a workaround. I think the ICU checkout is for non-Darwin platforms.

mkdir swift-project
cd swift-project
git clone https://github.com/apple/swift.git swift
cd swift
utils/update-checkout --clone-with-ssh --skip-repository icu

@AnthonyLatsis That worked for me from latest main. Thanks!

[vanvoorden]

Is this all it prints—no further details on the exception?

https://gist.github.com/vanvoorden/b6e9240e1d236b0202635eecc8a867fd

@AnthonyLatsis Here is what I saw. The 2024-01-17-1 log was my first attempt.

[AnthonyLatsis]

Hm. I guess we’re not getting any output from Git on why the checkout subcommand failed because of the echo=False here:

https://github.com/apple/swift/blob/1da24ece36a799ce23bf7c2f2e8f24db5a0fa62a/utils/update_checkout/update_checkout/update_checkout.py#L376-L381

Cloning dependencies from scratch with echo=True might help us identify the problem it you’re up for it.

[vanvoorden]

git-lfs filter-process: git-lfs: command not found
fatal: the remote end hung up unexpectedly

@AnthonyLatsis here is what I see after hacking echo=True.

[vanvoorden]

https://github.com/unicode-org/icu/blob/main/docs/devsetup/source/index.md?plain=1#L75-L94

@AnthonyLatsis Here is the git-lfs requirement from icu. Has this error never thrown on CI before?

I'm also on Apple Silicon BTW… it looks like the CI jobs are running from Intel?

[AnthonyLatsis]

Thanks for checking!

it looks like the CI jobs are running from Intel?

Yes.

Has this error never thrown on CI before?

No idea. This was irrelevant to macOS until recently because icu had been specified as a Linux-only dependency. macOS CI nodes are running fine, so they must have git-lfs installed, purposefully or not.

@compnerd Was making icu part of the macOS checkout intentional? Also, I have never seen anyone run into this on Linux either, despite git-lfs not being immediately among the Linux build dependencies listed in the Docker files. Is it a transitive dependency?

[compnerd]

@AnthonyLatsis I think that it is better to have the uniform repository checkouts. I think it was less of a concern when Windows was less reliant on update-checkout, but since there was a push to homogenise the CI environment, I think that uniformly checking out the repositories is a reasonable expectation. It is technically possible to build for Windows from macOS, and that would require that the sources are checked out.

[vanvoorden]

It is technically possible to build for Windows from macOS, and that would require that the sources are checked out.

@AnthonyLatsis @compnerd Hmm… if update-checkout now requires (by default) dependencies needed to build for a platform the engineer might not be interested in… do we have the option to either opt out of that dependency (or ask for an explicit opt-in) with a parameter passed to update-checkout? I do have the option to pass --skip-repository icu as a workaround… but the trend here is that update-checkout might (in the future) require additional deps only for building windows? It seems like there could be more deps in the future engineers focusing just on macOS for local development want to skip out on. Giving those engineers the option to skip out through just one flag (something like skip-windows-deps?) could improve the DevX… what do you think about that?

If we do decide to leave update-checkout as it is (and require engineers on macOS to either have lfs installed or explicitly opt out of icu) would you be willing to help update the GettingStarted.md documentation to show engineers what to do about this error? Thanks!

[AnthonyLatsis]

I think that it is better to have the uniform repository checkouts. I think it was less of a concern when Windows was less reliant on update-checkout, but since there was a push to homogenise the CI environment, I think that uniformly checking out the repositories is a reasonable expectation. It is technically possible to build for Windows from macOS, and that would require that the sources are checked out.

I’m happy to accept whatever the platform support folks think is best, but we need to be careful not to forget to update project dependencies.

Giving those engineers the option to skip out through just one flag (something like skip-windows-deps?) could improve the DevX… what do you think about that?

Yes, having tailored checkouts for specific workflows would be nice. This has been on my radar for a while. An average compiler development workflow needs only half, if not less, of what update-checkout clones.

[compnerd]

ICU is not windows specific, it is needed for Linux, Android, PS4, Windows, and WASI off the top of my head. Potentially for Haiku as well. It is not needed very specifically for Darwin because it is required for Foundation, which you do not build on Darwin and instead use the system provided copy. I think that introducing a --do-not-clone-foundation-dependencies would be a better name for skipping ICU, CURL, and libxml2.

[compnerd]

BTW, lfs shouldn't be required; it will fail the initial sync but should succeed subsequently (in fact, I don't have git-lfs installed on Windows).

[vanvoorden]

it will fail the initial sync but should succeed subsequently

@compnerd Here is what I saw. The second attempt also failed:

error: The following untracked working tree files would be overwritten by checkout

[keiabc]

swift 5.10 same error

swift/utils/update-checkout --tag swift-5.10-RELEASE --clone

[AnthonyLatsis]

Workaround: --skip-repository icu

[AnthonyLatsis]

FYI: The need to re-clone rarely arises, so this problem should not bother you on subsequent updates after the initial cloning even without the workaround.