x/build: add LUCI linux-loong64 builder

[abner-chenc]

Proposal Details

@gopherbot, please add label new-builder

[abner-chenc]

linux-loong64-builder.csr.txt

[mauri870]

Thanks for working on this!

[mknyszek]

I'm on rotation so I'll try to get to this this week.

[dr2chase]

Does anyone know offhand if this is a slower-than-usual builder? (I'm on rotation this week, in the middle of doing the addition.)

[dr2chase]

and it does have network access?

[mknyszek]

@dr2chase The SLOW_HOST list you're referring to can be updated later if we deem them necessary.

NO_NETWORK does not have to do with having network access or not. They're platforms where we set up a special environment to run tests that explicitly need to run in a no-network environment.

[gopherbot]

Change https://go.dev/cl/561339 mentions this issue: main.star: add linux-loong64 builder

[abner-chenc]

I haven't fully configured loong64's builder yet, and some additional settings may be needed for the builder to work properly.
But sorry, I'm on vacation recently (about a week), I'll try to finish it in advance if possible.

[dr2chase]

The certificate you provided, it is for "linux-loong64-builder", could it instead be for "linux-loong64"? That would match the pattern for all the other builders. Thanks much, sorry for the hiccup, no rush if you are enjoying your vacation.

[abner-chenc]

linux-loong64.csr.txt

This is the certificate I regenerated using hostname "linux-loong64". Please replace linux-loong64-builder.csr.txt with the current linux-loong64.csr.txt.　

Thanks

[abner-chenc]

Gentle ping.
Are the certificates ready?

[prattmic]

linux-loong64-1708541172.cert.txt

[abner-chenc]

Thank you for the certificate.

I've tried to launch the swarming bot, but I came across this error:

luci_machine_tokend -backend luci-token-server.appspot.com -cert-pem /tmp/linux-loong64-1708541172.cert.txt -pkey-pem /home/golang/linux-loong64.key -token-file=/var/lib/luci_machine_tokend/token.json
[I2024-02-22T15:15:37.646633+08:00 1526740 0 iface.go:167] tsmon is disabled because no endpoint is configured
[I2024-02-22T15:15:37.649157+08:00 1526740 0 main.go:237] The token is valid, skipping the update

/tmp/bootstrapswarm -hostname linux-loong64 
2024/02/22 15:15:42 Bootstrapping the swarming bot with certificate authentication
2024/02/22 15:15:42 retrieving the luci-machine-token from the token file /var/lib/luci_machine_tokend/token.json (default path for GOOS != windows)
2024/02/22 15:15:42 Downloading the swarming bot
2024/02/22 15:15:44 Starting the swarming bot /root/.swarming/swarming_bot.zip
1526774 2024-02-22 07:22:00.529 E: Unable to open given url, https://chromium-swarm.appspot.com/swarming/api/v1/bot/server_ping, after 30 attempts or 360 timeout.
HTTPSConnectionPool(host='chromium-swarm.appspot.com', port=443): Max retries exceeded with url: /swarming/api/v1/bot/server_ping (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fffe0619050>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
1526774 2024-02-22 07:22:00.530 E: No response from server_ping
1526774 2024-02-22 07:26:05.915 E: Unable to open given url, https://chromium-swarm.appspot.com/swarming/api/v1/bot/handshake, after 30 attempts or 240 timeout.
HTTPSConnectionPool(host='chromium-swarm.appspot.com', port=443): Max retries exceeded with url: /swarming/api/v1/bot/handshake (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fffe0634d90>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
1526774 2024-02-22 07:26:05.915 E: Failed to contact for handshake, retrying in 0 sec...
1526774 2024-02-22 07:30:10.905 E: Unable to open given url, https://chromium-swarm.appspot.com/swarming/api/v1/bot/handshake, after 30 attempts or 240 timeout.
HTTPSConnectionPool(host='chromium-swarm.appspot.com', port=443): Max retries exceeded with url: /swarming/api/v1/bot/handshake (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fffe0644ad0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
1526774 2024-02-22 07:30:10.905 E: Failed to contact for handshake, retrying in 1 sec...
1526774 2024-02-22 07:34:19.128 E: Unable to open given url, https://chromium-swarm.appspot.com/swarming/api/v1/bot/handshake, after 30 attempts or 240 timeout.
HTTPSConnectionPool(host='chromium-swarm.appspot.com', port=443): Max retries exceeded with url: /swarming/api/v1/bot/handshake (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fffe05fe490>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
1526774 2024-02-22 07:34:19.128 E: Failed to contact for handshake, retrying in 3 sec...

Is there something I'm doing wrong?
Thanks

[prattmic]

Can you access https://chromium-swarm.appspot.com/swarming/api/v1/bot/server_ping? That shouldn't require authentication or anything. It seems like there may be an issue with your network connection?

[abner-chenc]

Can you access https://chromium-swarm.appspot.com/swarming/api/v1/bot/server_ping? That shouldn't require authentication or anything. It seems like there may be an issue with your network connection?

Link https://chromium-swarm.appspot.com/swarming/api/v1/bot/server_ping I can access it normally, I try to curl the command to retrieve it and return Server up

This is the content of swarming_bot.log

[root@loongson-3a5000-01 logs]# cat swarming_bot.log 
977 2024-02-26 09:45:34.248 I: start_bot with args: []
977 2024-02-26 09:45:34.735 I: importing bot_main: /root/.swarming/swarming_bot.1.zip, d859187601d58b8378b1618189bcac3594a5269f12dd6b32a46eeee1bd54c1d6
977 2024-02-26 09:45:34.737 I: [singleton] acquire: /root/.swarming/swarming.lck = <_io.BufferedRandom name='/root/.swarming/swarming.lck'>
977 2024-02-26 09:45:34.744 D: Starting new HTTPS connection (1): chromium-swarm.appspot.com:443
977 2024-02-26 09:46:34.823 W: Unable to open url https://chromium-swarm.appspot.com/swarming/api/v1/bot/server_ping on attempt 0: HTTPSConnectionPool(host='chromium-swarm.appspot.com', port=443): Max retries exceeded with url: /swarming/api/v1/bot/server_ping (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fffe2748850>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
977 2024-02-26 09:46:36.771 W: Retrying request https://chromium-swarm.appspot.com/swarming/api/v1/bot/server_ping, attempt 1/30...
977 2024-02-26 09:46:36.772 D: Starting new HTTPS connection (2): chromium-swarm.appspot.com:443
977 2024-02-26 09:47:36.825 W: Unable to open url https://chromium-swarm.appspot.com/swarming/api/v1/bot/server_ping on attempt 1: HTTPSConnectionPool(host='chromium-swarm.appspot.com', port=443): Max retries exceeded with url: /swarming/api/v1/bot/server_ping (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fffe2749d10>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
977 2024-02-26 09:47:38.603 W: Retrying request https://chromium-swarm.appspot.com/swarming/api/v1/bot/server_ping, attempt 2/30...
977 2024-02-26 09:47:38.604 D: Starting new HTTPS connection (3): chromium-swarm.appspot.com:443
977 2024-02-26 09:48:38.684 W: Unable to open url https://chromium-swarm.appspot.com/swarming/api/v1/bot/server_ping on attempt 2: HTTPSConnectionPool(host='chromium-swarm.appspot.com', port=443): Max retries exceeded with url: /swarming/api/v1/bot/server_ping (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fffe274ae10>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

Based on this https://go.dev/wiki/DashboardBuilders Network access requirements, links https://remotebuildexecution.googleapis.com I am unable to access it, everything else is fine

[abner-chenc]

I am not sure what is the correct way to configure a proxy for the swarming bot. The source code for the swarming bot is at https://source.chromium.org/chromium/infra/infra/+/main:luci/appengine/swarming/swarming_bot/bot_code/bot_main.py;l=1785;drc=0d534b7e80a34a79a114fa0a05e5e984190fdbf7 (this line is, I believe, where the server_ping call comes from).

I am not very familiar with this code, but digging down several layers to https://source.chromium.org/chromium/infra/infra/+/main:luci/client/utils/net.py;l=810;drc=f9002fe7a0ae6b2837c9c48112b87d820bff9ffa, it looks like this ultimately uses Python requests, which looks like it should respect the environment variables: https://requests.readthedocs.io/en/latest/user/advanced/#proxies. Perhaps there is some additional configuration I missed.

This problem has been confirmed, that is, swarming_bot does not support system environment variable settings. Someone has reported this problem here luci/luci-py#323. After I tried adding this CL, I can see the builder of loong64 at https://7419-34ac013-dot-chromium-swarm.appspot.com/bot?id=linux-loong64

Here are my changes:

diff --git a/api/os_utilities.py b/api/os_utilities.py
index 445d3f7..3e51f9f 100644
--- a/api/os_utilities.py
+++ b/api/os_utilities.py
@@ -252,6 +252,9 @@ def get_cpu_type():
     return 'arm64'
   if machine == 'mips64':
     return 'mips'
+  if machine == 'loongarch64':
+    return 'loong64'
+
   return machine
 
 
diff --git a/client/cipd.py b/client/cipd.py
index 316fd06..97f39bb 100644
--- a/client/cipd.py
+++ b/client/cipd.py
@@ -307,6 +307,8 @@ def get_platform():
     arch = 'armv6l' if python_bits == 32 else 'arm64'
   elif arch == 'powerpc64':  # OpenBSD's name for ppc64
     arch = 'ppc64'
+  elif arch == 'loongarch64':
+    arch = 'loong64'
 
   elif not arch and os_name == 'windows':
     # On some 32bit Windows7, platform.machine() returns None.
diff --git a/utils/net.py b/utils/net.py
index 35392d0..8a403ef 100644
--- a/utils/net.py
+++ b/utils/net.py
@@ -808,8 +808,14 @@ class RequestsLibEngine:
   def __init__(self):
     super(RequestsLibEngine, self).__init__()
     self.session = requests.Session()
+    #proxies = {
+    #   'http':'http://x.x.x.x:yy',
+    #   'https':'http://x.x.x.x:yy',
+    #}
+    #self.session.proxies.update(proxies)
+
     # Configure session.
-    self.session.trust_env = False
+    self.session.trust_env = True
     self.session.verify = tools.get_cacerts_bundle()
     # Configure connection pools.
     for protocol in ('https://', 'http://'):

[abner-chenc]

I have enabled the system proxy environment settings and added Loong64 support to swarming-bot.

CL:
https://chromium-review.googlesource.com/c/infra/luci/luci-py/+/5334944
https://chromium-review.googlesource.com/c/infra/luci/luci-py/+/5334945

[abner-chenc]

I have enabled the system proxy environment settings and added Loong64 support to swarming-bot.

CL: https://chromium-review.googlesource.com/c/infra/luci/luci-py/+/5334944 https://chromium-review.googlesource.com/c/infra/luci/luci-py/+/5334945

＠prattmic　
　　These two CLs have been merged. Can you update swarming_bot.zip?
　　Thanks!

[abner-chenc]

This error occurred while running the task： https://7419-34ac013-dot-chromium-swarm.appspot.com/task?id=685629cf2b55df10&w=true

[prattmic]

cc @golang/release

[abner-chenc]

I found the following error after the service ran for a while：
HTTPSConnectionPool(host='chromium-swarm.appspot.com', port=443): Max retries exceeded with url: /swarming/api/v1/bot/rbe/session/update (Caused by ProxyError('Unable to connect to proxy', RemoteDisconnected('Remote end closed connection without response')))

But I have confirmed that the network is OK，　Here is the detailed log: swarming_bot.log

[abner-chenc]

This error occurred while running the task： https://7419-34ac013-dot-chromium-swarm.appspot.com/task?id=685629cf2b55df10&w=true

This problem seems to be happening again

[abner-chenc]

I am ready to connect multiple builders, but how can I distinguish the BOT_ID? The environment variable SWARMING_BOT_ID seems to be deprecated.

[abner-chenc]

I am ready to connect multiple builders, but how can I distinguish the BOT_ID? The environment variable SWARMING_BOT_ID seems to be deprecated.

Thanks, I know how to do this.

[abner-chenc]

This error occurred while running the task： https://7419-34ac013-dot-chromium-swarm.appspot.com/task?id=685629cf2b55df10&w=true

This problem seems to be happening again

Gentle ping.

[abner-chenc]

Can a task be triggered on this builder? If everything goes smoothly, I plan to take all the builders connected through the buildlet method offline and then switch them all to the LUCI access method

Thanks.

[abner-chenc]

Friendly ping. @golang/release

Can you solve this error? ( https://7419-34ac013-dot-chromium-swarm.appspot.com/task?id=68c4af6ef5417c10)?
I want to switch the builder’s access method to LUCI.

Thanks

[cagedmantis]

I'm working on resolving that error message. As soon as it is resolved, I will respond to this issue. Thanks for pushing this along.

[abner-chenc]

I'm working on resolving that error message. As soon as it is resolved, I will respond to this issue. Thanks for pushing this along.

Gentle ping.
Thanks.

[dmitshur]

We're still working on this on our side, and will update this after it's resolved or more progress is made. Thanks for your patience and for pushing this forward.

[abner-chenc]

Now the LUCI linux-loong64 builder can work happily. Of course, there are occasional network timeouts.
It seems that the LUCI method requires much greater network bandwidth than the previous buildlet method.

Thanks！

[cagedmantis]

We are glad to hear that it's working now. Now that the builder is up and running, is there any reason we can't close this issue? Thank you for all of your work.

[gopherbot]

Change https://go.dev/cl/584776 mentions this issue: main.star: remove linux-loong64 known issue

[gopherbot]

Change https://go.dev/cl/585175 mentions this issue: dashboard: mark all linux-loong64 builders as migrated to LUCI