ci: rework tests to break out ios/android tests

[henryiii]

Fail fast disabled, and trying to split out the android/ios runs.

The total time shouldn't be that much more than before, need to verify (a minute or two on sample builds is fine).

[joerick]

What do you think about also introducing flaky on the ios tests?

[henryiii]

We could, or there's an action that reruns on failure; actually I see at least two:

• https://github.com/marketplace/actions/retry-step
• https://github.com/marketplace/actions/retry-action

I've used one of those before, not sure which one.

[Copilot]

Pull Request Overview

This PR restructures the CI test workflow to separate iOS and Android tests into dedicated test runs, while maintaining existing coverage for other platforms. The changes enable platform-specific testing by introducing a platform matrix parameter and updating test configuration to handle different build frontends.

• Split test matrix to explicitly separate iOS/Android from native platform tests
• Updated test fixtures to intercept build calls for Android and iOS platforms
• Modified CI workflow to conditionally run certain steps based on platform type

Reviewed Changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 3 comments.

File	Description
unit_test/main_tests/conftest.py	Added Android and iOS platform imports and build interception for testing
.github/workflows/test.yml	Restructured test matrix to separate platform-specific runs and added conditional logic for different platforms

[henryiii]

Weird, android is getting stuck in the unit_tests and hanging, while that should be identical to native. And macOS android is failing to download. Not sure what is causing this yet, though.

[joerick]

The android unit test is probably failing due to something related to Docker - the docker tests are selected on that runner but binfmt isn't installed. I think my suggestion would be to not run the docker unit tests when testing android.

I haven't been able to figure out why the macOS android build is failing on Azure though.

[joerick]

Given we already have sys.platform and CIBW_PLATFORM changing the behaviour in this script, having another thing called "platform" is confusing me! Maybe we call this --test-select instead?

[joerick]

@mhsmith there's a test failure here that you might be interested in? On Azure pipelines, macOS x86_64, the android runner seems to fail while installing the android ndk. The error looks like this:

2025-08-11T09:49:21.8926460Z > assert "Android requires the build frontend to be 'build'" in capfd.readouterr().err 2025-08-11T09:49:21.8942480Z E assert "Android requires the build frontend to be 'build'" in 'Installing NDK - this may take several minutes\nWarning: An error occurred while preparing SDK package NDK (Side by side) 27.2.12479018: /Users/runner/Library/Android/sdk/.temp/PackageOperation01/unzip/android-ndk-r27c/sources/third_party/shaderc/third_party/spirv-tools/source/fuzz/fuzzer_pass_push_ids_through_variables.cpp.\ncibuildwheel: \x1b[91merror\x1b[0m: Command [\'/private/var/folders/hg/d8rl25qs4nl00s45_rtwkjmr0000gn/T/cibw-run-ho0e8esi/cp313-android_x86_64/python/android.py\', \'env\'] failed with code 1. Loading package information... \nLoading local repository... \n[ ] 3% Loading local repository... \n[ ] 3% Fetch remote repository... \n[= ] 3% Fetch remote repository... \n[= ] 4% Fetch remote repository... \n[= ] 5% Fetch remote repository... \n[== ] 5% Fetch remote repository... \n[== ] 6% Fetch remote repository... \n[== ] 7% Fetch remote repository... \n[== ] 7% Computing updates... \n[=== ] 8% Computing updates... \n[=== ] 10% Computing updates... \n[=== ] 10% Installing NDK (Side by side) 27.2\n[=== ] 10% Verifying previous download... \n[=== ] 10% Checking existing file... \n[=== ] 10% Downloading android-ndk-r27c-darwi\n[==== ] 10% Downloading android-ndk-r27c-darwi\n[==== ] 11% Downloading android-ndk-r27c-darwi\n[==== ] 12% Downloading android-ndk-r27c-darwi\n[===== ] 13% Downloading android-ndk-r27c-darwi\n[===== ] 14% Downloading android-ndk-r27c-darwi\n[===== ] 15% Downloading android-ndk-r27c-darwi\n[====== ] 15% Downloading android-ndk-r27c-darwi\n[====== ] 16% Downloading android-ndk-r27c-darwi\n[====== ] 17% Downloading android-ndk-r27c-darwi\n[======= ] 18% Downloading android-ndk-r27c-darwi\n[======= ] 19% Downloading android-ndk-r27c-darwi\n[======= ] 20% Downloading android-ndk-r27c-darwi\n[======== ] 20% Downloading android-ndk-r27c-darwi\n[======== ] 21% Downloading android-ndk-r27c-darwi\n[======== ] 22% Downloading android-ndk-r27c-darwi\n[========= ] 23% Downloading android-ndk-r27c-darwi\n[========= ] 24% Downloading android-ndk-r27c-darwi\n[========= ] 25% Downloading android-ndk-r27c-darwi\n[========== ] 25% Downloading android-ndk-r27c-darwi\n[========== ] 26% Downloading android-ndk-r27c-darwi\n[========== ] 27% Downloading android-ndk-r27c-darwi\n[=========== ] 28% Downloading android-ndk-r27c-darwi\n[=========== ] 29% Downloading android-ndk-r27c-darwi\n[=========== ] 30% Downloading android-ndk-r27c-darwi\n[============ ] 30% Downloading android-ndk-r27c-darwi\n[============ ] 31% Downloading android-ndk-r27c-darwi\n[============ ] 32% Downloading android-ndk-r27c-darwi\n[============ ] 33% Downloading android-ndk-r27c-darwi\n[============ ] 33% Unzipping... \n[============ ] 33% Unzipping... android-ndk-r27c/ \n[============ ] 33% Unzipping... android-ndk-r27c/simp\n[============= ] 33% Unzipping... android-ndk-r27c/simp\n[============= ] 33% Unzipping... android-ndk-r27c/ndk-\n[============= ] 33% Unzipping... android-ndk-r27c/wrap\n[============= ] 33% Unzipping... android-ndk-r27c/meta\n[============= ] 33% Unzipping... android-ndk-r27c/pyth\n[============= ] 33% Unzipping... android-ndk-r27c/sour\n[============= ] 33% Unzipping... android-ndk-r27c/preb\n[============= ] 33% Unzipping... android-ndk-r27c/READ\n[============= ] 33% Unzipping... android-ndk-r27c/sour\n[===================== ] 55% Unzipping... android-ndk-r27c/sour\n \n[===================== ] 55% Unzipping... android-ndk-r27c/sour\nCommand "set -eu; HOST=x86_64-linux-android; PREFIX=/private/var/folders/hg/d8rl25qs4nl00s45_rtwkjmr0000gn/T/cibw-run-ho0e8esi/cp313-android_x86_64/python/prefix; . /private/var/folders/hg/d8rl25qs4nl00s45_rtwkjmr0000gn/T/cibw-run-ho0e8esi/cp313-android_x86_64/python/android-env.sh; export" returned exit status 1\n\n\n'

It seems to just stop unzipping the archive and error out. The output's a bit cluttered because pytest is capturing it, I'll try to make it easier to read...

[mhsmith]

Could the runner be running out of disk space while unzipping? I hit that problem on GHA, though I think the symptoms were different. I worked around it here:

cibuildwheel/.github/workflows/test.yml

Lines 74 to 75 in 3ebb6bd

	- name: Free up disk space
	if: runner.os == 'Linux'

[joerick]

Ah, it looks like I might have accidentally worked around it! I moved the first android test into the serial phase of the tests and it disappeared. So I'm thinking that maybe the issue was that the sdkmanager call was being performed concurrently, and that was failing in a weird way. Maybe the android-env.sh script could lock around this, as it is system global state. I'm actually a bit surprised that sdkmanager doesn't do that itself.