ENH: support setuptools-style cross compilation on macOS #226
Conversation
dnicolodi
force-pushed
the
macos-platform-tag
branch
from
aad81df to
4806e74
Compare
|
From what I understand, this won't make cross-compiled binaries yet, it will just be tagged cross compiled? |
|
It does if you put the right arguments in |
|
Great, |
|
Also, simply setting |
This can be done, and should make things work for some cases. But is it really desirable to add this code complexity internally? Teaching the end user how to do it the right way with a cross file themselves seem cleaner, and much more educational because it scales to cross-compilation on any platform. I am also wondering how many packages would even use this at all. If you use The tweak made in this PR right now is fairly minimal, so if that works and nothing else breaks then it should be fine to merge. But it seems like it doesn't, and auto-generating a cross file is quite a bit of complexity and tech debt for probably very limited number of users and the wrong design. |
|
AFAIK, cross compilation is supported by the Python ecosystem only in one case: on macOS when building binaries "for the other architecture" (for arm64 when building on x86_64 or for x86_64 when building on arm64) and when not linking to any dependency that does not come in also with the object files for the other architecture. This is enabled by the fact that the macOS toolchain allows to cross-build This is currently supported by a number of tools in an ad-hoc way involving the The Meson way to setup cross-compilation is via a cross-file. However, this alone does not result in usable extension modules and package wheels: Meson uses the Python I think it is possible to get meson-python to generate correct cross-built wheels using the just introduced capability of specifying options for Meson on the command line to specify a proper cross-file and setting the This would not however integrate well with cibuildwheel and similar tools that expect the magic environment variables to work. As a downstream python extension author I would very much like to be able to delegate the nitty gritty details of preparing redistributable wheels to cibuildwheel and not having to find the proper incantations myself. Therefore, I see value in supporting the cibuildwheel way of doing things, even if it looks ugly. For doing so we need to check the What concerns me, however, is that this PR highlights the fact that there may be something wrong in how we determine the wheel tags. I modeled the code in meson-python after what the packaging module does, but apparently, on macOS packaging a wheel disagree on what is the correct platform tag: the first derives it from |
|
Just to point out, hundreds of packages today use cibuildwheel cross-compiling, and are very happy with it. The only downside is it's tricker to handle dependencies, but if you are handling them right (usually downloading a compatible build or building them yourself with the proper flags), then that works too. It's also important to check a produced wheel on Apple Silicon, but that's up to the author, we can't automate that. I believe it's important to make the simple cases easy, and work at least as well as setuptools. Setuptools does support cross compiling on macOS, also on Windows, even if it is not completely official. Having a crossfile for tricker cases is fine. But it will be more instructive to a user if they first understand why they need it by trying the simple way first. And if simple cases work "out of the box", many users won't have to learn something they don't want to learn (very few people actually want to learn about packaging & building, they just want it to work). If you avoid all compiled dependencies, for example, you shouldn't have to do anything. MacOS version is something I think you should compute yourself (along with the arch, which I also think should be computed on macOS). If |
It is not "fine", it is a requirement to cross-compile with Meson. Either it needs to be provided by the user or generated. I feel I'm start to repeat myself a bit too much.
Please define "compute yourself". Do you mean go and poke around the system to get the information or rely on the Python standard library? Everything else you write is already how things work AFAIK so, unless you have evidence of the contrary, keeping repeating some abstract requirement that we already agreed upon and implemented only adds to the confusion. |
|
Sorry, I just meant having a user provide a crossfile is fine for tricker cases, but should not be required for simple to moderate cases if the flags setuptools, scikit-build-core, and Maturin all use are set (ie, generated if it doesn't exist and the flags are set). I was responding to @rgommers's suggestion that this rarely or maybe never works - for hundreds of small to moderate projects, it works very well. By compute yourself, I was responding to "we are producing wheels where the wheel filename tags and the extension modules filename suffix do not agree" - I think both need to be set following the procedure rather than relying on anything else. The value of macOS in |
Okay, fair enough. I think longer term we'll all use native M1 CI and this will become a bit of technical debt, but you say it has value for you and you're putting in the work, so I'll pause my concerns here. Also, let me volunteer to help write docs for cross-compiling. I'll need docs for that at some point anyway for SciPy. |
Not quite yet, my priority currently is shipping the fist non-beta scikit-build-core, which is already on day 5 of of my 0-4 day estimate. And it sounds like I'll have to figure out how to generate a cross-compile file for Meson to do this, so I'd like to wait till after or during the docs being written, so I know what to do. :) But I can work on it if no one else gets to it first. |
This is a PR already, I meant @dnicolodi. But thanks - looks like we're all volunteering to do something here! |
|
Quick thought, and something I did for the scripts dir in scikit-build-core: what about throwing an error if |
|
There are two things here: producing the target wheel tag, and producing binaries for the target arch. The PR as I understand it does the first without doing the second. My suggestion above would handle the second one by producing an error for now, and eventually I think it would be nice to generate a crossfile if the envvar(s) are set. On the flip side, if a user does supply a crossfile, I'm assuming they still also need to set |
Which procedure? Where is "the right thing" documented? For now I've seen |
I hear you. This journey for me started more than a month ago trying to publish an uber simple package and so far it involved patching pip, meson, meson-python, and we are not there yet. Generating the cross file seems straightforward. I'll give it a try later today. |
dnicolodi
force-pushed
the
macos-platform-tag
branch
8 times, most recently
from
526e8cb to
06840fb
Compare
dnicolodi
changed the title
|
@henryiii I added a first draft of support for |
dnicolodi
force-pushed
the
macos-platform-tag
branch
from
06840fb to
aa331d9
Compare
dnicolodi
force-pushed
the
macos-platform-tag
branch
3 times, most recently
from
0f306c6 to
03149fd
Compare
|
Tests are passing. Turns out that blanket exceptions suppression does not really aid debugging... To keep the tradition with setuptools this is carefully undocumented, but it should work. @FFY00 This builds on the support for passing options to meson that you recently added. I would appreciate if you could have a close look. Thank you! |
dnicolodi
force-pushed
the
macos-platform-tag
branch
2 times, most recently
from
d2c3f5b to
96ec63c
Compare
|
I tested the cross-compiled wheels on macOS arm64 via Cirrus CI and they work. I think this can be merged. |
|
Options are applied in this order: meson-python defaults, |
|
If there are no comments I'm going to merge this soon. I've tested this with one project of mines and it works. |
|
@FFY00 can you please have a look? I would like your opinion on the |
Use sysconfig.get_platform() instead of platform.mac_ver() to base the platform tag computation. The ability to overwrite the value returned by former via the _PYTHON_HOST_PLATFORM environment variable is used in macOS extension modules cross-compilation. See mesonbuild#222 for background.
dnicolodi
force-pushed
the
macos-platform-tag
branch
from
96ec63c to
553ef60
Compare
| if sysconfig.get_platform().startswith('macosx-'): | ||
| archflags = self._env.get('ARCHFLAGS') | ||
| if archflags is not None: | ||
| arch, *other = filter(None, (x.strip() for x in archflags.split('-arch'))) | ||
| if other: | ||
| raise ConfigError(f'multi-architecture builds are not supported but $ARCHFLAGS={archflags!r}') | ||
| macver, _, nativearch = platform.mac_ver() | ||
| if arch != nativearch: | ||
| x = self._env.setdefault('_PYTHON_HOST_PLATFORM', f'macosx-{macver}-{arch}') | ||
| if not x.endswith(arch): | ||
| raise ConfigError(f'$ARCHFLAGS={archflags!r} and $_PYTHON_HOST_PLATFORM={x!r} do not agree') | ||
| family = 'aarch64' if arch == 'arm64' else arch | ||
| cross_file_data = textwrap.dedent(f''' | ||
| [binaries] | ||
| c = ['cc', '-arch', {arch!r}] | ||
| cpp = ['cpp', '-arch', {arch!r}] | ||
| [host_machine] | ||
| system = 'Darwin' | ||
| cpu = {arch!r} | ||
| cpu_family = {family!r} | ||
| endian = 'little' | ||
| ''') | ||
| self._meson_cross_file.write_text(cross_file_data) | ||
| self._meson_args['setup'].extend(('--cross-file', os.fspath(self._meson_cross_file))) |
There was a problem hiding this comment.
Humm, I'd prefer the cross file not to be added to meson_args, but rather added in _configure directly. I think it would be alright to always have a cross file, and it just being empty when it is not needed. I feel like that is cleaner and better sets things up for the future if we need to use it for something else.
There was a problem hiding this comment.
Adding --cross-file for a non cross build sounds more confusing than helpful to me. I'd much prefer to not see that.
There was a problem hiding this comment.
I agree with @rgommers Also, at the moment I cannot imagine any other feature that would automagically generate a cross-file. If we will find one, we will revisit this. One main advantage of not having the --cross-file argument when not needed, is that its presence when scanning the build output is already a good indication that the environment variables are set correctly.
There was a problem hiding this comment.
This could still be done in optionally _configure, without hijacking the user args.
Use sysconfig.get_platform() instead of platform.mac_ver() to base the platform tag computation. The ability to overwrite the value returned by former via the _PYTHON_HOST_PLATFORM environment variable is used in macOS extension modules cross-compilation.
See #222 for background.