Add support for aarch64 release builds

[toolboc]

My team recently encountered a situation where we needed to run the azure-pipelines-agent on an aarch64 (ARM 64bit) device.

At this time, the official release builds only support:

• Windows x86/64
• macOS
• Linux x64/ARM32

We were able to install and run the Linux ARM32 release flavor by installing the armhf dependencies on an aarch64 device (Jetson TX2), however, it would be nice to have first class support for this architecture due to it's prevalence in the IoT space (Raspberry Pi 3B+, PINE64, Nividia Jetson etc.).

#Install the ARM32 gcc toolchain
sudo apt-get install gcc-arm-linux-gnueabihf

#Symlink the armhf ld kernel module (this allows loading ARM32 bins on ARM64 OS and comes with the ARM32 toolchain)
sudo ln -s /usr/arm-linux-gnueabihf/lib/ld-linux-armhf.so.3 /lib/ld-linux-armhf.so.3

#Add arm-linux-gnueabihf libs to LD_LIBRARY_PATH 
#(This tells ld where it can find libs in addition to the default /lib dir,
#Point it to the default path for armhf libs
export LD_LIBRARY_PATH="/usr/arm-linux-gnueabihf/lib/"

#OR For a persistent version of the above
#Symlink the relevant kernel modules
sudo ln -s /usr/arm-linux-gnueabihf/lib/ld-linux-armhf.so.3 /lib/ld-linux-armhf.so.3
sudo ln -s /usr/arm-linux-gnueabihf/lib/libdl.so.2 /lib/libdl.so.2
sudo ln -s /usr/arm-linux-gnueabihf/lib/libpthread.so /lib/libpthread.so
sudo ln -s /usr/arm-linux-gnueabihf/lib/libpthread.so.0 /lib/libpthread.so.0
sudo ln -s /usr/arm-linux-gnueabihf/lib/libstdc++.so.6 /lib/libstdc++.so.6
sudo ln -s /usr/arm-linux-gnueabihf/lib/libm.so.6 /lib/libm.so.6
sudo ln -s /usr/arm-linux-gnueabihf/lib/libgcc_s.so.1 /lib/libgcc_s.so.1
sudo ln -s /usr/arm-linux-gnueabihf/lib/libc.so.6 /lib/libc.so.6
sudo ln -s /usr/arm-linux-gnueabihf/lib/librt.so.1 /lib/librt.so.1

#Enable installation of armhf packages
dpkg --add-architecture armhf

#Install armhf version of libcurl and libicu
apt-get install libcurl4-openssl-dev:armhf libicu55:armhf

#Install dotnet core (aarch 64 build prolly works here too)
wget https://download.visualstudio.microsoft.com/download/pr/10b96626-02d8-415a-be85-051a2a48d0c2/5ec51d3d9f092ba558fb5f1f03d26699/dotnet-sdk-2.1.403-linux-arm.tar.gz
mkdir -p $HOME/dotnet && tar zxf dotnet-sdk-2.1.403-linux-arm.tar.gz -C $HOME/dotnet
export DOTNET_ROOT=$HOME/dotnet
export PATH=$PATH:$HOME/dotnet

[TingluoHuang]

@toolboc we are waiting for dotnet core to add ARM64 support, it's part of dotnet core 3.0 which will release in 2019.
dotnet/announcements#82

[toolboc]

@TingluoHuang , looks like current dotnet core (v2.1.5) already supports ARM64 => https://www.microsoft.com/net/download/dotnet-core/2.1

Is there any interest in adding support sooner?

[TingluoHuang]

@toolboc i really wish. :)
https://github.com/dotnet/core/blob/master/release-notes/2.1/2.1-supported-os.md#linux

From what i have been told, it's runnable but dotnet core team won't support it when you runs into issue.
The official support will be added as part of dotnet core 3.0.

[vtbassmatt]

ARM64 support in .NET Core: dotnet/announcements#82

[nicolasg-rocketleap]

Now that .NET Core 3.0 is officially released, can we expect an updated agent which supports ARM64 soon?

[BrianMouncer]

What is the status on this work?

I am looking into self hosting a Linux ARM32 and 64 build agent, as cross compiling in docker is adding several hours to our projects build time, and my hope is that compiling with native toolset could reduce our build time by an hour or so.

[nicolasg-rocketleap]

I have the same issue. I have to run arm64 docker builds manually on my machine using the QEMU emulation layer which is painfully slow.

Having proper arm64 support for the Azure pipeline agent is critical for us. Hopefully this gets prioritized...

[BrianMouncer]

Also, when Linux ARM64 support is available, will we then see a "Microsoft Hosted Agent" for this platform, just like we have for Ubuntu X64?

[freyshen]

Also waiting for the ARM64 version agent on linux ARM64 platform.
Is there any release plan for it?

[vtbassmatt]

We haven't scheduled the work. I hope we can get to it this calendar year, though that's not a commitment.

@BrianMouncer no, we don't plan a hosted pool of ARM64 agents.

[vkrasnov]

I would love to see ARM64 support as well.
FWIW the changes to current version to make it run on ARM64 are very minimal:

https://github.com/vkrasnov/azure-pipelines-agent/tree/vlad/arm64_fix

[eXXL]

Support for ARM64 or guidance how to build and deploy agent on my own on raspberry 4 would be great.

[kaysan13]

Hello,

Here is a vsts version that supports arm64, I was able to create a build from a ubuntu 18.04.3 arm64 LTS.

https://github.com/kaysan13/azure-pipelines-agent.git

To build the project, go to the src directory and then

./dev build
./dev.sh package

then you download the azure agent, you replace the bin directory by your bin builder directory.

I hope I can help with this little fix,

Let me know if you have any problems

[liminany]

Has any roadmap for this ?

[vtbassmatt]

Sorry, nothing publicly tracking it. Same state as before: we need to move the agent to .NET Core 3.0 and then this becomes easy.

[BrianMouncer]

@vtbassmatt back in October.

We haven't scheduled the work. I hope we can get to it this calendar year, though that's not a commitment.

It's December, is there any update on this work (will it still make this calendar year)?

I see there was an initial PR for the agent changes, but needed test cases and lab hardware to finish., Then another that re-did this work on the current state of master, but no definitive indication when the PR will be completed, or when the new agent should be available in production.

#2256

#1911

[isgallagher]

Any update to this? Would be great to have a hosted agent running for e.g. multiarch/ubuntu-core:arm64-xenial to build arm64 packages.

[vtbassmatt]

There will not be hosted agents running ARM64, even once we ship an ARM64 agent. You'll be able to use self-hosted or elastic self-hosted pools, though.

[NGPixel]

@vtbassmatt Any ETA or status update when it will be available?

[vtbassmatt]

Unfortunately no. I've made soft promises before and had to break them, so I won't do that here. Hopefully soon!

[jkunkee]

Note that on ARM64 Windows desktop, the x86 client should run under emulation. I have not tried this, so of course YMMV, but it should be on par with the x86 client on x64.

Once the client is on .NET Core and .NET Core supports ARM64 Windows, then it should(TM) Just Work.

[vtbassmatt]

@toolboc @nicolasg-playster @BrianMouncer @freyshen @vkrasnov @eXXL @liminany @NGPixel @Qinka @michalpenka @philipktlin - good news, there is an ARM64 agent now. Sorry for the looong delay.

[vtbassmatt]

(Leaving open until we're officially official with support)

[michalpenka]

This is excellent, thanks @vtbassmatt for an update. I'll give it a try later today and will report any bug should I find any.

Like your ARM build btw!

[vtbassmatt]

In case anyone is interested in the details, it's a PicoCluster 5H with Raspberry Pi 4Bs running Ubuntu Server 18.04 64-bit. The bottom two, currently disconnected, units will probably get 32-bit Raspbian or Ubuntu Server, as the Pi4 is about twice as fast as our current ARM32 build machines.

[michalpenka]

It's just such a shame PicoCluster is so expensive. But looks great and neat!

Anyways, well done @vtbassmatt for all the work on the agent.

[patschaefer]

Thanks @vtbassmatt for this. We have been running a hacked up version based on 2.159 for a few months, I upgraded my agents to the 2.166.2 last week and have seen no issues so far.

[amaslenn]

Link to arm64 package is broken in v2.166.3, both arm and arm64 point to arm file.
This one https://vstsagentpackage.azureedge.net/agent/2.166.3/vsts-agent-linux-arm64-2.166.3.tar.gz doesn't exist.

[vtbassmatt]

Thanks @amaslenn. I'll file a separate issue to make sure we track that.

[michalpenka]

I've successfully deployed 2.166.2 (.3 link really not available) to my Kubernetes cluster and so far I have found no issues. Will keep you updated should I find any. Thanks again!

[kushalkoolwal]

Thanks, @vtbassmatt for all your hard work to get to this point. Are there any remaining tasks to complete before we close this one out? Seems like Arm/Arm64 builds are now available with latest releases up to 2.168.2

[vtbassmatt]

Thanks for the reminder. Yes, this is official now.

[philipktlin]

Btw I think we need to update the New Agent download page in DevOps Website (click "New Agent" button on DevOps website) to include ARM64 with download link and steps.

[mthalman]

Just wanted to drop a note saying that we've begun using this for our CI builds of the official .NET Docker images: dotnet/docker-tools#564. We make use of NVIDIA Jetson machines that we've updated to install the ARM64 version of the agent. Originally, we had AMD64-based Linux VMs that run the pipeline agent software and acted as a proxy, sending remote Docker commands to the Jetson machines. Now we can completely get rid of those proxy machines. Everything's working great at this point.

[BrianMouncer]

I’m assuming the “NVIDIA Jetson machines” are selfhosted agents? Now that we have a functional ARM agent, are there any plans to have ARM agents in the MS hosted pools? Thanks, Brian. From: Matt Thalman <[email protected]> Sent: Monday, June 22, 2020 11:37 AM To: microsoft/azure-pipelines-agent <[email protected]> Cc: Brian Mouncer <[email protected]>; Mention <[email protected]> Subject: Re: [microsoft/azure-pipelines-agent] Add support for aarch64 release builds (#1911) Just wanted to drop a note saying that we've begun using this for our CI builds of the official .NET Docker images: dotnet/docker-tools#564<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fdocker-tools%2Fpull%2F564&data=02%7C01%7CBrian.Mouncer%40microsoft.com%7C5b9dd52303944241f14d08d816db48ca%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637284478359974795&sdata=PVqMnelKeGhZ%2FjCW%2BNje0boDVFfkv1JTiltnZt7tKh0%3D&reserved=0>. We make use of NVIDIA Jetson machines that we've updated to install the ARM64 version of the agent. Originally, we had AMD64-based Linux VMs that run the pipeline agent software and acted as a proxy, sending remote Docker commands to the Jetson machines. Now we can completely get rid of those proxy machines. Everything's working great at this point. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fazure-pipelines-agent%2Fissues%2F1911%23issuecomment-647702295&data=02%7C01%7CBrian.Mouncer%40microsoft.com%7C5b9dd52303944241f14d08d816db48ca%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637284478359984787&sdata=S6%2B%2BssUQM4tPSMmNX3ltRgTv66R%2B7jYoeLyaVifN8lk%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALUD54MMXG3PZZVB6RI6UGTRX6QFVANCNFSM4F75RUNA&data=02%7C01%7CBrian.Mouncer%40microsoft.com%7C5b9dd52303944241f14d08d816db48ca%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637284478359984787&sdata=sC5yeMoC41hpCdvsb3TusdYPHsICAF6dBcPr3qTKEgM%3D&reserved=0>.

[vtbassmatt]

No plans for a hosted pool of these.

[azimuthdeveloper]

Are there any plans for a Hosted pool of these? If not, can we use docker buildx to build arm64 images on an azure pipeline?

[dennisameling]

@lewcianci I'm working on adding arm64 images for DDEV using docker buildx on a x64 host. They're using CircleCI (Ubuntu 16.04) but I assume you can use similar or the same steps in Azure Pipelines (include everything that's in the Prepare docker buildx for multi-arch builds step):
https://github.com/drud/ddev/blob/66f4640f97c57769e59e9fb7fff55b209583e350/.circleci/config.yml#L269-L286

Double-check if Docker 19.03+ is installed on your pipeline, this is a prerequisite to be able to use buildx.

Then you can e.g. run docker buildx build --platform linux/amd64,linux/arm64 ...... further down in the pipeline as described here: https://github.com/docker/buildx#documentation

[vtbassmatt]

@lewcianci no plans for a hosted pool.

[azimuthdeveloper]

@lewcianci no plans for a hosted pool.

I find that strange considering that Microsoft has invested quite heavily in IoT and yet we can't use the build agents to build ARM64 images natively and have to use QEMU to build instead. But it's perhaps not in high enough demand yet 😄

Thanks for getting back to me

[rajeshaz09]

@lewcianci no plans for a hosted pool.

I find that strange considering that Microsoft has invested quite heavily in IoT and yet we can't use the build agents to build ARM64 images natively and have to use QEMU to build instead. But it's perhaps not in high enough demand yet 😄

Thanks for getting back to me

It's surprise to me as well.