Skip to content

[build.ps1] don't use host Python any more #76594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Dec 2, 2024
Merged

[build.ps1] don't use host Python any more #76594

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c621c76
[build.ps1] don't use host Python any more
stevapple Sep 19, 2024
eb359bb
[build.ps1] fix compatibility with Windows Powershell
stevapple Sep 20, 2024
b6ea4cc
Try to address CI failure
stevapple Nov 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 46 additions & 14 deletions utils/build.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,14 +188,6 @@ $WiXVersion = "4.0.5"
# Avoid $env:ProgramFiles in case this script is running as x86
$UnixToolsBinDir = "$env:SystemDrive\Program Files\Git\usr\bin"

$python = "${env:ProgramFiles(x86)}\Microsoft Visual Studio\Shared\Python39_64\python.exe"
if (-not (Test-Path $python)) {
$python = (where.exe python) | Select-Object -First 1
if (-not (Test-Path $python)) {
throw "Python.exe not found"
}
}

if ($Android -and ($AndroidSDKs.Length -eq 0)) {
# Enable all android SDKs by default.
$AndroidSDKs = @("aarch64","armv7","i686","x86_64")
Expand Down Expand Up @@ -352,6 +344,10 @@ function Get-BisonExecutable {
return Join-Path -Path $BinaryCache -ChildPath "win_flex_bison\win_bison.exe"
}

function Get-PythonExecutable {
return Join-Path -Path $BinaryCache -ChildPath "Python$($HostArch.CMakeName)-$PythonVersion\tools\python.exe"
}

function Get-InstallDir($Arch) {
if ($Arch -eq $HostArch) {
$ProgramFilesName = "Program Files"
Expand Down Expand Up @@ -740,10 +736,43 @@ function Fetch-Dependencies {
}
}

function Ensure-PythonModules($Python) {
# First ensure pip is installed, else bootstrap it
try {
Invoke-Program -OutNull $Python -m pip *> $null
} catch {
Write-Output "Installing pip ..."
Invoke-Program -OutNull $Python '-I' -m ensurepip -U --default-pip
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to just perform the installation assuming it is not present without any adverse effects?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t think that’s a good idea, but it’s certainly doable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had a simple try and found that we cannot easily reuse Extract-ZipFile because Expand-Archive requires .zip as file extension, and wheels aren’t extracted to standalone directories. Given that pip is already included (no download required) and simply just not installed, it shouldn’t be a real problem.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that using pip is fine, just trying to see if we can simplify some of the logic here by just unconditionally doing python -I -m ensurepip -U --default-pip.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running conditionally should work, but I intended to log precisely about what’s happening. Also need to notice that this command will output some noise when you already have pip installed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this @stevapple! The move away from host Python solved an issue for me accessing modules in LLDB tests. And I can re-use your approach for installing modules: 942538f What else do you think will be necessary to merge this PR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@weliveindetail Glad to hear that it helped you! Let’s wait for @compnerd to kick off a new CI run and see if there’s any other problem.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great. Then I will wait for it to land, rebase my changes on top of it and add my Python modules for lldb tests. Maybe I will have to factor out a function or so, that would be no problem to do. What do you think @compnerd?

# 'packaging' is required for building LLVM 18+
try {
Invoke-Program -OutNull $Python -c 'import packaging' *> $null
} catch {
$WheelURL = "https://files.pythonhosted.org/packages/08/aa/cc0199a5f0ad350994d660967a8efb233fe0416e4639146c089643407ce6/packaging-24.1-py3-none-any.whl"
$WheelHash = "5b8f2217dbdbd2f7f384c41c628544e6d52f2d0f53c6d0c3ea61aa5d1d7ff124"
DownloadAndVerify $WheelURL "$BinaryCache\python\packaging-24.1-py3-none-any.whl" $WheelHash
Write-Output "Installing 'packaging-24.1-py3-none-any.whl' ..."
Invoke-Program -OutNull $Python '-I' -m pip install "$BinaryCache\python\packaging-24.1-py3-none-any.whl" --disable-pip-version-check
}
# 'setuptools' provides 'distutils' module for Python 3.12+, required for SWIG support
# https://github.com/swiftlang/llvm-project/issues/9289
try {
Invoke-Program -OutNull $Python -c 'import distutils' *> $null
} catch {
$WheelURL = "https://files.pythonhosted.org/packages/ff/ae/f19306b5a221f6a436d8f2238d5b80925004093fa3edea59835b514d9057/setuptools-75.1.0-py3-none-any.whl"
$WheelHash = "35ab7fd3bcd95e6b7fd704e4a1539513edad446c097797f2985e0e4b960772f2"
DownloadAndVerify $WheelURL "$BinaryCache\python\setuptools-75.1.0-py3-none-any.whl" $WheelHash
Write-Output "Installing 'setuptools-75.1.0-py3-none-any.whl' ..."
Invoke-Program -OutNull $Python '-I' -m pip install "$BinaryCache\python\setuptools-75.1.0-py3-none-any.whl" --disable-pip-version-check
}
}

Download-Python $HostArchName
if ($IsCrossCompiling) {
Download-Python $BuildArchName
}
# Ensure Python modules that are required as host build tools
Ensure-PythonModules "$(Get-PythonExecutable)"

if ($Android) {
# Only a specific NDK version is supported right now.
Expand Down Expand Up @@ -1411,6 +1440,9 @@ function Build-Compilers() {
}
}

$PythonRoot = "$BinaryCache\Python$($Arch.CMakeName)-$PythonVersion\tools"
$PythonLibName = "python{0}{1}" -f ([System.Version]$PythonVersion).Major, ([System.Version]$PythonVersion).Minor

# The STL in VS 17.10 requires Clang 17 or higher, but Swift toolchains prior to version 6 include older versions
# of Clang. If bootstrapping with an older toolchain, we need to relax to relax this requirement with
# ALLOW_COMPILER_AND_STL_VERSION_MISMATCH.
Expand Down Expand Up @@ -1442,10 +1474,10 @@ function Build-Compilers() {
LLVM_NATIVE_TOOL_DIR = $BuildTools;
LLVM_TABLEGEN = (Join-Path $BuildTools -ChildPath "llvm-tblgen.exe");
LLVM_USE_HOST_TOOLS = "NO";
Python3_EXECUTABLE = "$python";
Python3_INCLUDE_DIR = "$BinaryCache\Python$($Arch.CMakeName)-$PythonVersion\tools\include";
Python3_LIBRARY = "$BinaryCache\Python$($Arch.CMakeName)-$PythonVersion\tools\libs\python39.lib";
Python3_ROOT_DIR = "$BinaryCache\Python$($Arch.CMakeName)-$PythonVersion\tools";
Python3_EXECUTABLE = (Get-PythonExecutable);
Python3_INCLUDE_DIR = "$PythonRoot\include";
Python3_LIBRARY = "$PythonRoot\libs\$PythonLibName.lib";
Python3_ROOT_DIR = $PythonRoot;
SWIFT_BUILD_SWIFT_SYNTAX = "YES";
SWIFT_CLANG_LOCATION = (Get-PinnedToolchainTool);
SWIFT_ENABLE_EXPERIMENTAL_CONCURRENCY = "YES";
Expand Down Expand Up @@ -1715,7 +1747,7 @@ function Build-Runtime([Platform]$Platform, $Arch) {
})
}

Invoke-Program $python -c "import plistlib; print(str(plistlib.dumps({ 'DefaultProperties': { 'DEFAULT_USE_RUNTIME': 'MD' } }), encoding='utf-8'))" `
Invoke-Program "$(Get-PythonExecutable)" -c "import plistlib; print(str(plistlib.dumps({ 'DefaultProperties': { 'DEFAULT_USE_RUNTIME': 'MD' } }), encoding='utf-8'))" `
-OutFile "$($Arch.SDKInstallRoot)\SDKSettings.plist"
}

Expand Down Expand Up @@ -1942,7 +1974,7 @@ function Build-Testing([Platform]$Platform, $Arch, [switch]$Test = $false) {

function Write-PlatformInfoPlist($Arch) {
$PList = Join-Path -Path $Arch.PlatformInstallRoot -ChildPath "Info.plist"
Invoke-Program $python -c "import plistlib; print(str(plistlib.dumps({ 'DefaultProperties': { 'XCTEST_VERSION': 'development', 'SWIFT_TESTING_VERSION': 'development', 'SWIFTC_FLAGS': ['-use-ld=lld'] } }), encoding='utf-8'))" `
Invoke-Program "$(Get-PythonExecutable)" -c "import plistlib; print(str(plistlib.dumps({ 'DefaultProperties': { 'XCTEST_VERSION': 'development', 'SWIFT_TESTING_VERSION': 'development', 'SWIFTC_FLAGS': ['-use-ld=lld'] } }), encoding='utf-8'))" `
-OutFile "$PList"
}

Expand Down