buildkite failure on julia v1.6

[CarloLucibello]

Lately, the buildkite gpu tests on julia v1.6 are failing, see
https://buildkite.com/julialang/flux-dot-jl/builds/3755#0186f06e-ec5d-4ddc-a1ea-3662edd19d53
as an example.
The error message is super long, truncated, and show a lot of CartesianIndex. It's hard to understand why this happens.

[ToucheSir]

I think that was a one-off spurious failiure, because I don't see it in any previous runs or on master. If the latest CI run completes successfully, we can probably close this.

[ToucheSir]

Ok, it happened again on a different PR: https://buildkite.com/julialang/flux-dot-jl/builds/3764#0186f1c1-c5df-4ebc-9a5d-9f0c24ee4969. I'm guessing something changed on the CUDA.jl side because we shouldn't have done anything on the FluxML side?

[christiangnrd]

This might be caused by GPUArrays 8.6.4 that was released yesterday.

[CarloLucibello]

now also v1.8 is failing. A different error?

julia v1.6 failure
https://buildkite.com/julialang/flux-dot-jl/builds/3771#018708eb-2773-4f2b-8cfc-212db49b6854

julia v1.8 failure
https://buildkite.com/julialang/flux-dot-jl/builds/3771#018708eb-2777-4255-94fb-b0873be0a97c

pinging @maleadt to see if he has any clues

[maleadt]

JuliaGPU/GPUCompiler.jl#410, already fixed, PkgServers are lagging more than a day behind. Requires LLVM 4.17.1

[CarloLucibello]

Failure of the day is
https://buildkite.com/julialang/flux-dot-jl/builds/3802#018728a6-e273-41ed-b265-322836527ac7

[CarloLucibello]

The problem is a CUDNN failure in a Float16 test on julia v1.6

[ToucheSir]

Strange that we'd suddenly get CUDNN_STATUS_BAD_PARAM and only on 1.6...