Strategy for running CI-tests against GPU machines

[nils-braun]

With the recent integration of dask-cudf into dask-sql, it would be wise to start testing against GPU libraries and machines in the automated build triggered by GitHub actions.

@ayushdg mentioned on #220, that:

GpuCI is one option that could be used to test integration with rapids libraries. It's what the dask repository uses as well (dask/community#138).
I don't think it uses GitHub actions specifically but is enabled via a webhook to GitHub.

I would be very much interested, if this is also an option for dask-contrib packages (and if this would be supported by RAPIDS) and what the current experience with the setup at the dask main repository is. Maybe, @quasiben, @jrbourbeau, can you shortly summarize the current setup in the dask main repository and how the experiences are? Thanks!

[charlesbluca]

The basic setup for Dask's gpuCI consisted of:

• Marking relevant GPU tests with a custom pytest marker (see Add pytest marker for GPU tests dask/dask#7876)
• Enabling gpuCI on a fork of Dask and getting the required setup files working there (see Fill in gpuCI build script charlesbluca/dask#1); these files consist of
  • a shell script defining the steps to build the test environment and run the desired tests
  • a YAML file defining the build matrix
• Once gpuCI works in the forked branch, merging this branch into the main repo and enabling gpuCI there

Once enabled, gpuCI runs on all PRs opened by members on an allowlist (in Dask's case, this contains all members of the organization as well as some other people added using the commands listed here); initially, there was some noisiness as the gpuCI bot pinged a large chunk of the open Dask PRs for approval to run GPU tests, but this should be avoidable here if we populate the allowlist before enabling.

Another thing to consider is that gpuCI on Dask and Distributed uses Docker images (generated nightly based on specifications here) to speed up the overall test time; we might want to do something similar for dask-sql, but I'm interested in if it would be possible to reuse one of those images here.

[charlesbluca]

Closing this as we now have gpuCI set up for GPU testing