Skip to content

feat: Adding support for python 3.13 #1666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 47 commits into from
Apr 25, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
a1fb445
Proxy Worker: Initial Commit
gavin-aguiar Feb 5, 2025
4d3b892
Updated worker config to include 3.13
gavin-aguiar Feb 6, 2025
a6c80f1
Updated test_setup
gavin-aguiar Feb 7, 2025
5343e0e
Updated worker.py
gavin-aguiar Feb 7, 2025
d1390d5
Updated dispatcher
gavin-aguiar Feb 11, 2025
bcad815
Updated syspath in worker.py
gavin-aguiar Feb 11, 2025
f720195
Updated path in worker.py
gavin-aguiar Feb 12, 2025
f7f4795
Updated worker.py
gavin-aguiar Feb 14, 2025
43d29a3
Removed reload in dispatcher
gavin-aguiar Feb 14, 2025
bcdf819
Updating v1 library worker name
gavin-aguiar Feb 18, 2025
bc4a520
Added dispatcher logs
gavin-aguiar Feb 18, 2025
7c7cc36
Added dispatcher try/catch logs
gavin-aguiar Feb 18, 2025
4764ce7
Updated sys path
gavin-aguiar Feb 20, 2025
53a3d4e
Dispatcher and dependency manager updates
gavin-aguiar Feb 25, 2025
5caa922
Updated dispatcher and pyproject
gavin-aguiar Mar 27, 2025
357ed59
Merge branch 'dev' of github.com:Azure/azure-functions-python-worker …
gavin-aguiar Mar 28, 2025
dabaec6
Testing updates and refactoring
gavin-aguiar Apr 9, 2025
7f038bb
Bug fixes and refactoring
gavin-aguiar Apr 9, 2025
3ff686d
Added more unit tests
gavin-aguiar Apr 10, 2025
5ade8e3
Added tests and fixed test setup
gavin-aguiar Apr 10, 2025
c465406
Updated test_setup
gavin-aguiar Apr 10, 2025
6083d7f
Updated test setup to add grpc dir copy
gavin-aguiar Apr 11, 2025
be60b01
build: proxy worker build & test setup (#1664)
hallvictoria Apr 11, 2025
2992d42
Merging changes
gavin-aguiar Apr 11, 2025
dc94eb1
linting fixes
gavin-aguiar Apr 11, 2025
fb5a154
Resolving conflicts
gavin-aguiar Apr 11, 2025
66175ce
Addressed comments
gavin-aguiar Apr 11, 2025
f439425
Updated unit test and added missing protos files
gavin-aguiar Apr 11, 2025
fb1823c
fix e2e test reference
hallvictoria Apr 14, 2025
4ac586f
lint, mypy, add 3.13 to unittests
Apr 15, 2025
f6b6551
correct version check
Apr 15, 2025
1e7e732
syntax
Apr 15, 2025
b364962
syntax
Apr 15, 2025
3b1f961
fix unit tests, mypy
hallvictoria Apr 18, 2025
4de671d
oops
hallvictoria Apr 18, 2025
b376b46
format
hallvictoria Apr 18, 2025
2419e78
lint
hallvictoria Apr 18, 2025
661de36
fix unittest dir for proxy
Apr 22, 2025
7fccc3a
set env variable
Apr 22, 2025
df125fd
update pyproject to use real deps
hallvictoria Apr 23, 2025
9613706
bump to a2
hallvictoria Apr 23, 2025
c9be882
bump v2 to a3
hallvictoria Apr 23, 2025
5ac75e1
Import v2 by default for LC
gavin-aguiar Apr 23, 2025
2429bdd
Merge branch 'gaaguiar/proxy_worker' of github.com:Azure/azure-functi…
gavin-aguiar Apr 23, 2025
b841184
Merge branch 'dev' into gaaguiar/proxy_worker
gavin-aguiar Apr 24, 2025
dec9d34
Refactoring and minor fixes
gavin-aguiar Apr 24, 2025
0873832
Merge branch 'gaaguiar/proxy_worker' of github.com:Azure/azure-functi…
gavin-aguiar Apr 24, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .flake8
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
ignore = W503,E402,E731

exclude = .git, __pycache__, build, dist, .eggs, .github, .local, docs/,
Samples, azure_functions_worker/protos/,
Samples, azure_functions_worker/protos/, proxy_worker/protos/,
azure_functions_worker/_thirdparty/typing_inspect.py,
tests/unittests/test_typing_inspect.py,
tests/unittests/broken_functions/syntax_error/main.py,
Expand Down
1 change: 0 additions & 1 deletion azure_functions_worker/protos/.gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,2 @@
/_src
*_pb2.py
*_pb2_grpc.py
2 changes: 2 additions & 0 deletions eng/templates/jobs/ci-emulator-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ jobs:
PYTHON_VERSION: '3.11'
Python312:
PYTHON_VERSION: '3.12'
Python313:
PYTHON_VERSION: '3.13'
steps:
- task: UsePythonVersion@0
inputs:
Expand Down
21 changes: 19 additions & 2 deletions eng/templates/jobs/ci-unit-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ jobs:
PYTHON_VERSION: '3.11'
Python312:
PYTHON_VERSION: '3.12'

Python313:
PYTHON_VERSION: '3.13'
steps:
- task: UsePythonVersion@0
inputs:
Expand All @@ -34,8 +35,24 @@ jobs:
displayName: 'Install dependencies'
condition: and(eq(variables.isSdkRelease, false), eq(variables.isExtensionsRelease, false), eq(variables['USETESTPYTHONSDK'], false), eq(variables['USETESTPYTHONEXTENSIONS'], false))
- bash: |
python -m pytest -q -n auto --dist loadfile --reruns 4 --instafail --cov=./azure_functions_worker --cov-report xml --cov-branch tests/unittests
PY_VER="$(PYTHON_VERSION)"
echo "Python version: $PY_VER"

# Extract minor version
PY_MINOR="${PY_VER#*.}"

if [ "$PY_MINOR" -ge 13 ]; then
echo "Running proxy_worker tests (Python >= 3.13)..."
python -m pytest -q -n auto --dist loadfile --reruns 4 --instafail \
--cov=./proxy_worker --cov-report xml --cov-branch tests/unittest_proxy
else
echo "Running unittests (Python < 3.13)..."
python -m pytest -q -n auto --dist loadfile --reruns 4 --instafail \
--cov=./azure_functions_worker --cov-report xml --cov-branch tests/unittests
fi
displayName: "Running $(PYTHON_VERSION) Unit Tests"
# Skip running tests for SDK and Extensions release branches. Public pipeline doesn't have permissions to download artifact.
condition: and(eq(variables.isSdkRelease, false), eq(variables.isExtensionsRelease, false), eq(variables['USETESTPYTHONSDK'], false), eq(variables['USETESTPYTHONEXTENSIONS'], false))
env:
PYTHON_VERSION: $(PYTHON_VERSION)

15 changes: 15 additions & 0 deletions eng/templates/official/jobs/build-artifacts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ jobs:
Python312V4:
pythonVersion: '3.12'
workerPath: 'python/prodV4/worker.py'
Python313V4:
pythonVersion: '3.13'
workerPath: 'python/proxyV4/worker.py'
templateContext:
outputParentDirectory: $(Build.ArtifactStagingDirectory)
outputs:
Expand Down Expand Up @@ -62,6 +65,9 @@ jobs:
Python312V4:
pythonVersion: '3.12'
workerPath: 'python/prodV4/worker.py'
Python313V4:
pythonVersion: '3.13'
workerPath: 'python/proxyV4/worker.py'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can the worker Path be a variable at the top?

templateContext:
outputParentDirectory: $(Build.ArtifactStagingDirectory)
outputs:
Expand Down Expand Up @@ -100,6 +106,9 @@ jobs:
Python312V4:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the long run - We should simplify these blocks - the yaml is unnecessarily long.

pythonVersion: '3.12'
workerPath: 'python/prodV4/worker.py'
Python313V4:
pythonVersion: '3.13'
workerPath: 'python/proxyV4/worker.py'
templateContext:
outputParentDirectory: $(Build.ArtifactStagingDirectory)
outputs:
Expand Down Expand Up @@ -137,6 +146,9 @@ jobs:
Python312V4:
pythonVersion: '3.12'
workerPath: 'python/prodV4/worker.py'
Python313V4:
pythonVersion: '3.13'
workerPath: 'python/proxyV4/worker.py'
templateContext:
outputParentDirectory: $(Build.ArtifactStagingDirectory)
outputs:
Expand Down Expand Up @@ -168,6 +180,9 @@ jobs:
Python312V4:
pythonVersion: '3.12'
workerPath: 'python/prodV4/worker.py'
Python313V4:
pythonVersion: '3.13'
workerPath: 'python/proxyV4/worker.py'
templateContext:
outputParentDirectory: $(Build.ArtifactStagingDirectory)
outputs:
Expand Down
20 changes: 18 additions & 2 deletions eng/templates/official/jobs/ci-e2e-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,15 @@ jobs:
SQL_CONNECTION: $(LinuxSqlConnectionString312)
EVENTGRID_URI: $(LinuxEventGridTopicUriString312)
EVENTGRID_CONNECTION: $(LinuxEventGridConnectionKeyString312)
Python313:
PYTHON_VERSION: '3.13'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3.13 is added to the E2E and emulator pipelines right now. E2E tests are re-using 312 resources for now until the appropriate 313 resources are created (cosmosdb, sql, eventgrid). Adding 3.13 to unit tests is pending still -- we don't want to run proxy worker tests for <=3.12, and vice versa. This will involve some more logic

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

E2E tests are re-using 312 resources for now until the appropriate 313 resources are created (cosmosdb, sql, eventgrid).

For this - is there a helper script to smoothen the process of creation? If not, please add it in the 3.13 backlog.

STORAGE_CONNECTION: $(LinuxStorageConnectionString312)
COSMOSDB_CONNECTION: $(LinuxCosmosDBConnectionString312)
EVENTHUB_CONNECTION: $(LinuxEventHubConnectionString312)
SERVICEBUS_CONNECTION: $(LinuxServiceBusConnectionString312)
SQL_CONNECTION: $(LinuxSqlConnectionString312)
EVENTGRID_URI: $(LinuxEventGridTopicUriString312)
EVENTGRID_CONNECTION: $(LinuxEventGridConnectionKeyString312)
steps:
- task: UsePythonVersion@0
inputs:
Expand Down Expand Up @@ -120,7 +129,14 @@ jobs:
Write-Host "pipelineVarSet: $pipelineVarSet"
$branch = "$(Build.SourceBranch)"
Write-Host "Branch: $branch"
if($branch.StartsWith("refs/heads/sdk/") -or $pipelineVarSet -eq "true")

$PY_VER = "$(PYTHON_VERSION)"
Write-Host "Python version: $PY_VER"
# Extract minor version as integers
$PY_MINO = "${PY_VER#*.}"
Write-Host "Branch: PY_MINOR"

if($branch.StartsWith("refs/heads/sdk/") -or $pipelineVarSet -eq "true" -or $PY_MINOR -ge 13 )
{
Write-Host "##vso[task.setvariable variable=skipTest;]true"
}
Expand All @@ -129,7 +145,6 @@ jobs:
Write-Host "##vso[task.setvariable variable=skipTest;]false"
}
displayName: 'Set skipTest variable'
condition: or(eq(variables.isSdkRelease, true), eq(variables['USETESTPYTHONSDK'], true))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What did this do?

- powershell: |
Write-Host "skipTest: $(skipTest)"
displayName: 'Display skipTest variable'
Expand All @@ -145,4 +160,5 @@ jobs:
AzureWebJobsEventGridTopicUri: $(EVENTGRID_URI)
AzureWebJobsEventGridConnectionKey: $(EVENTGRID_CONNECTION)
skipTest: $(skipTest)
PYAZURE_WEBHOST_DEBUG: true
displayName: "Running $(PYTHON_VERSION) Python E2E Tests"
5 changes: 5 additions & 0 deletions pack/Microsoft.Azure.Functions.V4.PythonWorker.nuspec
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,11 @@
<file src="..\3.12_LINUX_X64\**" target="tools\3.12\LINUX\X64" />
<file src="..\3.12_OSX_X64\**" target="tools\3.12\OSX\X64" />
<file src="..\3.12_OSX_X64\**" target="tools\3.12\OSX\Arm64" />
<file src="..\3.13_WINDOWS_X64\**" target="tools\3.13\WINDOWS\X64" />
<file src="..\3.13_WINDOWS_X86\**" target="tools\3.13\WINDOWS\X86" />
<file src="..\3.13_LINUX_X64\**" target="tools\3.13\LINUX\X64" />
<file src="..\3.13_OSX_X64\**" target="tools\3.13\OSX\X64" />
<file src="..\3.13_OSX_X64\**" target="tools\3.13\OSX\Arm64" />
<file src="..\python\prodV4\worker.config.json" target="tools" />
<file src="Microsoft.Azure.Functions.PythonWorker.targets" target="build" />
<file src="..\_manifest\spdx_2.2\manifest.spdx.json" target="SBOM\manifest.spdx.json" />
Expand Down
8 changes: 7 additions & 1 deletion pack/scripts/mac_arm64_deps.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,10 @@ python -m invoke -c test_setup build-protos

cd ..
cp .artifactignore "$BUILD_SOURCESDIRECTORY/deps"
cp -r azure_functions_worker/protos "$BUILD_SOURCESDIRECTORY/deps/azure_functions_worker"

version_minor=$(echo $1 | cut -d '.' -f 2)
if [[ $version_minor -lt 13 ]]; then
cp -r azure_functions_worker/protos "$BUILD_SOURCESDIRECTORY/deps/azure_functions_worker"
else
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in the long run - the name shouldn't change - can it be done in this iteration itself?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this seems unnecessary.

cp -r proxy_worker/protos "$BUILD_SOURCESDIRECTORY/deps/proxy_worker"
fi
8 changes: 7 additions & 1 deletion pack/scripts/nix_deps.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,10 @@ python -m invoke -c test_setup build-protos

cd ..
cp .artifactignore "$BUILD_SOURCESDIRECTORY/deps"
cp -r azure_functions_worker/protos "$BUILD_SOURCESDIRECTORY/deps/azure_functions_worker"

version_minor=$(echo $1 | cut -d '.' -f 2)
if [[ $version_minor -lt 13 ]]; then
cp -r azure_functions_worker/protos "$BUILD_SOURCESDIRECTORY/deps/azure_functions_worker"
else
cp -r proxy_worker/protos "$BUILD_SOURCESDIRECTORY/deps/proxy_worker"
fi
16 changes: 14 additions & 2 deletions pack/scripts/win_deps.ps1
Original file line number Diff line number Diff line change
@@ -1,11 +1,16 @@
param (
[string]$pythonVersion
)
$versionParts = $pythonVersion -split '\.' # Splitting by dot
$versionMinor = [int]$versionParts[1]

python -m venv .env
.env\Scripts\Activate.ps1
python -m pip install --upgrade pip

python -m pip install .

$depsPath = Join-Path -Path $env:BUILD_SOURCESDIRECTORY -ChildPath "deps"
$protosPath = Join-Path -Path $depsPath -ChildPath "azure_functions_worker/protos"

python -m pip install . azure-functions --no-compile --target $depsPath.ToString()

Expand All @@ -15,4 +20,11 @@ python -m invoke -c test_setup build-protos

cd ..
Copy-Item -Path ".artifactignore" -Destination $depsPath.ToString()
Copy-Item -Path "azure_functions_worker/protos/*" -Destination $protosPath.ToString() -Recurse -Force

if ($versionMinor -lt 13) {
$protosPath = Join-Path -Path $depsPath -ChildPath "azure_functions_worker/protos"
Copy-Item -Path "azure_functions_worker/protos/*" -Destination $protosPath.ToString() -Recurse -Force
} else {
$protosPath = Join-Path -Path $depsPath -ChildPath "proxy_worker/protos"
Copy-Item -Path "proxy_worker/protos/*" -Destination $protosPath.ToString() -Recurse -Force
}
35 changes: 35 additions & 0 deletions pack/templates/macos_64_env_gen.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,19 @@ steps:
inputs:
versionSpec: ${{ parameters.pythonVersion }}
addToPath: true
- bash: |
major=$(echo $(pythonVersion) | cut -d. -f1)
minor=$(echo $(pythonVersion) | cut -d. -f2)
echo "##vso[task.setvariable variable=pythonMajor]$major"
echo "##vso[task.setvariable variable=pythonMinor]$minor"
echo $pythonMinor
displayName: 'Parse pythonVersion'
- task: ShellScript@2
inputs:
disableAutoCwd: true
scriptPath: 'pack/scripts/mac_arm64_deps.sh'
args: '${{ parameters.pythonVersion }}'
displayName: 'Install Dependencies'
- bash: |
pip install pip-audit
pip-audit -r requirements.txt
Expand Down Expand Up @@ -41,4 +50,30 @@ steps:
!pkg_resources/**
!*.dist-info/**
!werkzeug/debug/shared/debugger.js
!proxy_worker/**
targetFolder: '$(Build.ArtifactStagingDirectory)'
condition: in(variables['pythonMinor'], '7', '8', '9', '10', '11', '12')
displayName: 'Copy azure_functions_worker files'
- task: CopyFiles@2
inputs:
sourceFolder: '$(Build.SourcesDirectory)/deps'
contents: |
**
!grpc_tools/**/*
!grpcio_tools*/*
!build/**
!docs/**
!pack/**
!python/**
!tests/**
!setuptools*/**
!_distutils_hack/**
!distutils-precedence.pth
!pkg_resources/**
!*.dist-info/**
!werkzeug/debug/shared/debugger.js
!azure_functions_worker/**
!dateutil/**
targetFolder: '$(Build.ArtifactStagingDirectory)'
condition: in(variables['pythonMinor'], '13')
displayName: 'Copy proxy_worker files'
35 changes: 35 additions & 0 deletions pack/templates/nix_env_gen.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,19 @@ steps:
inputs:
versionSpec: ${{ parameters.pythonVersion }}
addToPath: true
- bash: |
major=$(echo $(pythonVersion) | cut -d. -f1)
minor=$(echo $(pythonVersion) | cut -d. -f2)
echo "##vso[task.setvariable variable=pythonMajor]$major"
echo "##vso[task.setvariable variable=pythonMinor]$minor"
echo $pythonMinor
displayName: 'Parse pythonVersion'
- task: ShellScript@2
inputs:
disableAutoCwd: true
scriptPath: 'pack/scripts/nix_deps.sh'
args: '${{ parameters.pythonVersion }}'
displayName: 'Install Dependencies'
- bash: |
pip install pip-audit
pip-audit -r requirements.txt
Expand Down Expand Up @@ -41,4 +50,30 @@ steps:
!pkg_resources/**
!*.dist-info/**
!werkzeug/debug/shared/debugger.js
!proxy_worker/**
targetFolder: '$(Build.ArtifactStagingDirectory)'
condition: in(variables['pythonMinor'], '7', '8', '9', '10', '11', '12')
displayName: 'Copy azure_functions_worker files'
- task: CopyFiles@2
inputs:
sourceFolder: '$(Build.SourcesDirectory)/deps'
contents: |
**
!grpc_tools/**/*
!grpcio_tools*/*
!build/**
!docs/**
!pack/**
!python/**
!tests/**
!setuptools*/**
!_distutils_hack/**
!distutils-precedence.pth
!pkg_resources/**
!*.dist-info/**
!werkzeug/debug/shared/debugger.js
!dateutil/**
!azure_functions_worker/**
targetFolder: '$(Build.ArtifactStagingDirectory)'
condition: in(variables['pythonMinor'], '13')
displayName: 'Copy proxy_worker files'
34 changes: 34 additions & 0 deletions pack/templates/win_env_gen.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,17 @@ steps:
versionSpec: ${{ parameters.pythonVersion }}
architecture: ${{ parameters.architecture }}
addToPath: true
- bash: |
major=$(echo $(pythonVersion) | cut -d. -f1)
minor=$(echo $(pythonVersion) | cut -d. -f2)
echo "##vso[task.setvariable variable=pythonMajor]$major"
echo "##vso[task.setvariable variable=pythonMinor]$minor"
echo $pythonMinor
displayName: 'Parse pythonVersion'
- task: PowerShell@2
inputs:
filePath: 'pack\scripts\win_deps.ps1'
arguments: '${{ parameters.pythonVersion }}'
- bash: |
pip install pip-audit
pip-audit -r requirements.txt
Expand Down Expand Up @@ -41,4 +49,30 @@ steps:
!pkg_resources\**
!*.dist-info\**
!werkzeug\debug\shared\debugger.js
!proxy_worker\**
targetFolder: '$(Build.ArtifactStagingDirectory)'
condition: in(variables['pythonMinor'], '7', '8', '9', '10', '11', '12')
displayName: 'Copy azure_functions_worker files'
- task: CopyFiles@2
inputs:
sourceFolder: '$(Build.SourcesDirectory)\deps'
contents: |
**
!grpc_tools\**\*
!grpcio_tools*\*
!build\**
!docs\**
!pack\**
!python\**
!tests\**
!setuptools*\**
!_distutils_hack\**
!distutils-precedence.pth
!pkg_resources\**
!*.dist-info\**
!werkzeug\debug\shared\debugger.js
!dateutil\**
!azure_functions_worker\**
targetFolder: '$(Build.ArtifactStagingDirectory)'
condition: in(variables['pythonMinor'], '13')
displayName: 'Copy proxy_worker files'
2 changes: 2 additions & 0 deletions proxy_worker/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License.
6 changes: 6 additions & 0 deletions proxy_worker/__main__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License.
from proxy_worker import start_worker

if __name__ == '__main__':
start_worker.start()
Loading
Loading