Hardcoded / stored compiler and compiler flags may cause inability to build perl modules

[mojca]

(I'm sorry for violating the template, but my case doesn't fit too well into the prescribed module.)

Request

We really need a way to avoid using hardcoded paths to compiler and some other options in perl.

Some tickets for reference:

• https://trac.macports.org/ticket/50894
• https://trac.macports.org/ticket/59755
• https://trac.macports.org/ticket/59786
• https://trac.macports.org/ticket/62440

While I understand why perl does what it does when storing options to Config.pm and Config_heavy.pl, that approach causes some headaches for us.

Background

We are packaging perl inside a package manager for macOS (MacPorts) that basically supports anything from Tiger 10.4 up to the latest version, and anything from powerpc to arm64.

The latest macOS versions usually use the default compiler (as provided by Xcode), but we ship a wide range of gcc and clang versions as packages and the older OS versions nowadays use nearly the latest clang compiler by default, so that we can properly support the latest C++ standards.

Every now and then we increase the version of the default compiler on those ancient systems (which means that at some point we start using /opt/local/bin/clang-mp-9.0 instead of /opt/local/bin/clang-mp-7.0 for example, but at the same time we don't recompile existing packages like perl). The users might have already removed the older compiler from their system, or maybe they never had it installed in the first place: they might have downloaded the package in binary form from the server which used an older compiler for building.

At the other side of the spectrum, whenever the developer tools on the latest macOS version get an update, the compiler name stays the same (/usr/bin/clang), but the compiler version changes, and the supported compiler options change. And most annoyingly, the path to SDK changes. For example, at the time of compiling perl for macOS 11 for the first time, the following flags were stored to Config_heavy.pl:

libspath=' /opt/local/lib /Library/Developer/CommandLineTools/usr/lib/clang/12.0.0/lib /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/usr/lib /Library/Developer/CommandLineTools/usr/lib /usr/lib'

but the path to SDK basically changes with every single "security update". It started with MacOSX11.0.sdk, then goes to MacOSX11.1.sdk, MacOSX11.2.sdk, etc. Needless to say, we don't recompile all the package with every OS update.

We often mix-and-match compilers. Sure, not all random combinations of compilers work together, but as long as the same libc++ (or the same stdlibc++) is being used, choosing an arbitrary version of clang seems to do the job for us. Perl modules get compiled by the package manager as well, and the package manager is pretty rigorous about providing all the necessary flags (specifying full path to the compiler, full path to libraries, etc.).

Problem Description

Whenever something on the system changes, either due to updates from Apple, or due to our own updates like deciding to use a newer compiler by default, building perl modules becomes broken.

Ideally I would like to see the build of perl modules to respect CC / CXX env. variables when they are defined. Or provide some alternative mechanism for specifying the flags.

At the moment a significant portion of our perl ecosystem is broken on macOS 11 due to hardcoded paths.

We can "manually" mess with Configue.pm (I'll probably do that very soon, else too many modules remain broken), but I would really like to find the best solution together with the perl developers.

@jmroot, @ryandesign, @kencu

[Leont]

Whenever something on the system changes, either due to updates from Apple, or due to our own updates like deciding to use a newer compiler by default, building perl modules becomes broken.

This is another case of "Apple breaking basic expectations of how unix toolchains work", they're making a habit out of it :-/. Though to be honest, the piece of code that uses these library paths is a mistake of our own. I suspect we'll have to rewrite that to dynamically get the include paths from the compiler (essentially the same code that we use in Configure).

[mojca]

In our case the include paths cannot really be deduced from compiler alone, but from a combination of CC, CXX, CFLAGS, LDFLAGS, ... etc. (at the time of compiling perl modules such as Tk). But that's probably not too relevant.

At the moment I believe we are looking at two separate tasks:

• figuring out the best workaround that can be applied relatively fast, so that we can at least get the latest perl version work correctly on the latest macOS (this can include arbitrarily patching perl on our side, either before or after the build)
• figuring out a clean way to solve the problem in the long term

Do you have any reasonable suggestions for the first one, or any time estimation for the second one (even if it counts in years)?

We could do various cleanups or hard-core patches for the first part (workaround), but I'm still not sure about the best way to solve this one, for example:

timeincl='/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/usr/include/sys/time.h '

[Leont]

This pull request might solve your issue. I haven't really tested it yet though, beyond "the unit tests pass".

[mojca]

To clarify a bit more: there are two problematic things:

• hardcoded compiler itself ($cc)
• flags

Regarding the flags, here are all the problematic variables that currently contain MacOSX11.0.sdk that no longer exists on an up-to-date system:

Config.pm:

libpth => '/opt/local/lib /Library/Developer/CommandLineTools/usr/lib/clang/12.0.0/lib /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/usr/lib /Library/Developer/CommandLineTools/usr/lib /usr/lib',

Config_heavy.pl:

ccflags='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.16 -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk  -fno-strict-aliasing -fstack-protector-strong -I/opt/local/include -DPERL_USE_SAFE_PUTENV'
config_arg23='-Acppflags=-I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk'
config_arg24='-Accflags=-pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk '
config_arg25='-Alddlflags=-L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk '
config_arg26='-Aldflags=-L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk '
cppflags='-I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk -fno-common -DPERL_DARWIN -mmacosx-version-min=10.16 -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk  -fno-strict-aliasing -fstack-protector-strong -I/opt/local/include'
cpprun='/usr/bin/clang  -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk -E'
cppstdin='/usr/bin/clang  -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk -E'
incpth='/opt/local/include /Library/Developer/CommandLineTools/usr/lib/clang/12.0.0/include /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/usr/include /Library/Developer/CommandLineTools/usr/include'
lddlflags=' -mmacosx-version-min=10.16 -bundle -undefined dynamic_lookup -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk  -fstack-protector-strong'
ldflags=' -mmacosx-version-min=10.16 -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk  -fstack-protector-strong'
libpth='/opt/local/lib /Library/Developer/CommandLineTools/usr/lib/clang/12.0.0/lib /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/usr/lib /Library/Developer/CommandLineTools/usr/lib /usr/lib'
libspath=' /opt/local/lib /Library/Developer/CommandLineTools/usr/lib/clang/12.0.0/lib /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/usr/lib /Library/Developer/CommandLineTools/usr/lib /usr/lib'
timeincl='/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/usr/include/sys/time.h '

so it's not just about the libpth.

Meanwhile I discovered that there is apparently a symlink pointing from MacOSX11.sdk to MacOSX11.x.sdk, so I could use the former, but I seem to remember that these SDKs might sometimes be gone (read as: potentially replaced by MacOSX12.x.sdk) as well once a newer macOS version gets released and if one installs a newer version of developer tools.

For reference, this is the full invocation (including env. variables) when we build perl:

CC='/usr/bin/clang'
CC_PRINT_OPTIONS='YES'
CC_PRINT_OPTIONS_FILE='/path/to/perl5.32/work/.CC_PRINT_OPTIONS'
CFLAGS='-pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk -arch arm64'
CPATH='/opt/local/include'
CPPFLAGS='-I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
CXX='/usr/bin/clang++'
CXXFLAGS='-pipe -Os -stdlib=libc++ -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk -arch arm64'
DEVELOPER_DIR='/Library/Developer/CommandLineTools'
F90FLAGS='-pipe -Os -m64'
FCFLAGS='-pipe -Os -m64'
FFLAGS='-pipe -Os -m64'
INSTALL='/usr/bin/install -c'
LC_ALL='C'
LDFLAGS='-L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk -arch arm64'
LIBRARY_PATH='/opt/local/lib'
MACOSX_DEPLOYMENT_TARGET='11.0'
OBJC='/usr/bin/clang'
OBJCFLAGS='-pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk -arch arm64'
OBJCXX='/usr/bin/clang++'
OBJCXXFLAGS='-pipe -Os -stdlib=libc++ -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk -arch arm64'
SDKROOT='/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'

"/path/to/perl5.32/work/perl-5.32.1" && sh Configure -des -Dprefix='/opt/local' -Dscriptdir='/opt/local/bin' -Dvendorprefix='/opt/local' -Dusemultiplicity=y -Dusethreads -Duseshrplib -Dcc="$CC" -Dman1ext=1pm -Dman3ext=3pm -Dinstallstyle='lib/perl5' -Dman1dir='/opt/local/share/man/man1p' -Dman3dir='/opt/local/share/man/man3p' -Dsitebin='/opt/local/libexec/perl5.32/sitebin' -Dsiteman1dir='/opt/local/share/perl5.32/siteman/man1' -Dsiteman3dir='/opt/local/share/perl5.32/siteman/man3' -Dvendorbin='/opt/local/libexec/perl5.32' -Dvendorman1dir='/opt/local/share/perl5.32/man/man1' -Dvendorman3dir='/opt/local/share/perl5.32/man/man3' -Dpager='/usr/bin/less -sR' -Dperlpath="/opt/local/bin/perl5.32" -Dstartperl="#!/opt/local/bin/perl5.32" -Acppflags="$CPPFLAGS" -Accflags="$CFLAGS" -Alddlflags="$LDFLAGS" -Aldflags="$LDFLAGS" 

I will do a bit more investigation about what can best be done about all those MacOSX11.sdk paths.

This is about one type of flags. There are occasionally other problematic flags (see https://trac.macports.org/ticket/50894#comment:4).

I will test your patch on macOS 11 with perl 5.34 next, I just doubt that it will address the complete issue.

[jmroot]

This pull request might solve your issue. I haven't really tested it yet though, beyond "the unit tests pass".

Looks like a step in the right direction, but I don't think it addresses everything here. AIUI, there are a number of config variables that reference a specific SDK path or compiler path, which simply may not exist at runtime (either because the user updated their toolchain since building perl, or the perl binary was built on a different system.)

Some related work: Python had pretty much the same issues, and their solutions are thus:

• Always use the compiler specified by CC if set in the environment, and if not, use the one recorded in sysconfig if it is available, otherwise use platform-specific knowledge to find the best compiler that is available on the system: https://github.com/python/cpython/blob/3.9/Lib/_osx_support.py#L198
• Remove any -isysroot option that points to a nonexistent SDK from the flags retrieved from sysconfig: https://github.com/python/cpython/blob/3.9/Lib/_osx_support.py#L331
• Ask the compiler where it looks for headers and libraries, if no sysroot has been specified: https://github.com/python/cpython/blob/3.9/Lib/_osx_support.py#L153

[Leont]

so it's not just about the libpth.

Fair, but that would be the most immediate problem. To be honest, I think libpth and lddlflags are the only ones that are used in the standard toolchain. Though about two dozen dists on CPAN use cpprun and like one or two use ldflags. The rest is generally only used when building perl.

config_arg25='-Alddlflags=-L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk '

-L/opt/local/lib would be automatically adopted from ldflags by Configure, and I would guess -Wl,-syslibroot should also be. Intuitively I would guess headerpad_max_install_names isn't necessary there (lddlflags is only used for extensions, ldflags is for executable such as perl itself), but I'm not a Mach-O expert. Manually adding to lddlflags is generally a bad idea.

[mojca]

Manually adding to lddlflags is generally a bad idea.

-Alddlflags have been added in macports/macports-ports@041e3a80e967 by @MarcusCalhoun-Lopez with a reference to the following ticket:

• https://trac.macports.org/ticket/52739

I still have no clue what .bundle files are, but judging from the ticket we could have achieved the same effect by adding only -arch flags there, which are needed to tell the compiler what target architecture to build for (basically Apple's shortcut for cross-compiling). It is common to compile any package including perl for multiple architectures (say, for both x86_64 and arm64 by using just clang -arch x86_64 -arch arm64 and compiling the sources just once). At least that lets us getting rid of other "weird" flags.

I'm happy to change that part of the code if I get some suggestions about how to properly address "cross-compilations". (We generally want to allow building perl universally for multiple architectures in the same binary, but then allow building perl modules for any subset of those architectures depending on user preferences for that particular module.)

So, out of curiosity: when is specifying -Alddlflags not a bad idea (or rather: what's the reason the flag is there)? And what's the correct approach for "cross-compiling" then?

[ryandesign]

Meanwhile I discovered that there is apparently a symlink pointing from MacOSX11.sdk to MacOSX11.x.sdk, so I could use the former,

In Xcode 12.5 and later. See https://trac.macports.org/ticket/63139.

but I seem to remember that these SDKs might sometimes be gone (read as: potentially replaced by MacOSX12.x.sdk) as well once a newer macOS version gets released and if one installs a newer version of developer tools.

Absolutely. If we must bake an sdk path into perl config files, it would be most robust to bake in the path to the unversioned MacOSX.sdk since that should always exist.

[mojca]

@Leont: I tested the patch, but I see absolutely no difference. I'm not saying that the patch isn't useful, all I'm saying is that in my case I haven't seen any changes in Config_heavy.pl.

@ryandesign: we don't need to bake the sdk path into the config files. Perl will just happily store whatever we pass to its own build flags. If we remove all the references to SDKs when building Perl, none should end up in the config files. I hope. There are still a bunch of random other flags that sometimes end up in the config files (like -fstack-protector-strong) that break when Xcode or CLT get updated.

There are some consequences though:

• we can no longer specify the sdk that perl has to use
• we can no longer specify the sdk that perl modules have to use

[Leont]

@Leont: I tested the patch, but I see absolutely no difference. I'm not saying that the patch isn't useful, all I'm saying is that in my case I haven't seen any changes in Config_heavy.pl.

It shouldn't affect Config_heavy.pl, but it should fix any breakage related to libpth.

There are some consequences though:

* we can no longer specify the sdk that perl has to use

* we can no longer specify the sdk that perl modules have to use

What would the consequences of that be?

[jmroot]

Any program that checks for features at build time (only) needs to be built against the SDK matching the oldest OS version that the program is intended to run on. Otherwise it will crash when attempting to call any function that was in the SDK but is not present at runtime.

If you're not on the very latest OS version, Apple's toolchain has a good chance of defaulting to using the SDK for a newer OS version. This is because the Apple Way is to always build against the very latest SDK, using -Wunguarded-availability, and fix all the warnings by adding availability checks around all use of weak-linked symbols. Needless to say, almost nothing that is not specifically designed for an Apple OS does that. And that's the main reason why it's important to be able to control which SDK is used.

[oodler577]

Last I checked gcc on MacOS was actually clang in some gcc compatibility mode. So I wonder where this confusion is leading.

computer:~$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/c++/4.2.1
Apple clang version 12.0.0 (clang-1200.0.32.29)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

computer:~$ perl -MData::Dumper -MConfig -e 'print Dumper(%Config)' | grep cc
$VAR97 = 'byacc';
$VAR98 = 'byacc';
$VAR107 = 'cc';
$VAR108 = 'cc';
$VAR109 = 'cccdlflags';
$VAR111 = 'ccdlflags';
$VAR113 = 'ccflags';
$VAR115 = 'ccflags_uselargefiles';
$VAR117 = 'ccname';
$VAR118 = 'gcc';
$VAR119 = 'ccsymbols';
$VAR121 = 'ccversion';
$VAR167 = 'cppccsymbols';
$VAR176 = 'cc  -E';
$VAR178 = 'cc  -E';
$VAR221 = 'd_accept4';
$VAR223 = 'd_access';
$VAR225 = 'd_accessx';
$VAR375 = 'd_eaccess';
$VAR719 = 'd_locconv';
$VAR1385 = 'gccansipedantic';
$VAR1387 = 'gccosandvers';
$VAR1389 = 'gccversion';
$VAR1440 = '/usr/shlib  /lib /usr/lib /usr/lib/386 /lib/386 /usr/ccs/lib /usr/ucblib /usr/local/lib ';
$VAR1585 = 'i_sysaccess';
$VAR1746 = 'env MACOSX_DEPLOYMENT_TARGET=10.3 cc';
$VAR1830 = 'ccflags="-DUSE_PERL_SBRK -DPERL_SBRK_VIA_MALLOC $ccflags"';
$VAR2417 = 'yacc';
$VAR2418 = 'yacc';
$VAR2419 = 'yaccflags';
$VAR2425 = 'ccflags_nolargefiles';
$VAR2433 = 'ccwarnflags';
$VAR2435 = 'ccstdflags';

computer:~$ perl -MData::Dumper -MConfig -e 'print Dumper(%Config)' | grep clang
$VAR180 = '_FORTIFY_SOURCE=2 _LP64=1 __BYTE_ORDER__=1234 __GNUC_MINOR__=2 __GNUC__=4 __LITTLE_ENDIAN__=1 __LP64__=1 __MACH__=1 __PIC__=2 __STDC__=1 __amd64=1 __amd64__=1 __clang__=1 __pic__=2 __x86_64=1 __x86_64__=1';
$VAR1390 = '4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)';
$VAR1670 = '/usr/local/include /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/include /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include /usr/include';
$VAR1766 = '/usr/local/lib /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/lib /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib /usr/lib';
$VAR1776 = ' /usr/local/lib /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/lib /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib /usr/lib';

@mojca - does MacPorts use the base /usr/bin/perl for modules or does it attempt to build it's own perl using actual gcc? Just curious - I only came here to mention something that I have not seen in this issue, i.e., clang is operating in a compatibility mode and that Config:: is clearly confused by this.

[mojca]

We try to avoid using the system perl and nearly always build our own and depend on it whenever possible. (There might be some exceptions to the rule, but really only minor ones.) Note also that Apple is probably removing perl (and python and ruby and php and ...) from the tools that come preinstalled by default (my naive bet would be that this will happen one year from now, see for example https://twitter.com/mikeymikey/status/1404510362087555074), making it even more important to depend on the self-built version.

We gravitate towards clang, gcc is only used for very specific software with specific needs (Atlas etc.), or perhaps on 10.4 / 10.5 PowerPC where clang is not feature-complete. But we never call the compiler via gcc, it's always called as ${prefix}/bin/gcc-mp-9 or so when we need gcc, and the default is always called as /usr/bin/clang.

What exactly is strange in the above example (other than clang claiming to be gcc 4.2, but it's done that since the beginning, and it also does it if you call it as clang, I think)?

As a temporary workaround many of the hardcoded values are now filtered out, but we really hope that this will be addressed properly in the perl core: macports/macports-ports@f35a2c5