Pytorch ROCm Init is slow (3+ minutes)

[Delaunay]

After investigation of ROCm/pytorch#1232
we found the root cause of the issue to be https://github.com/pytorch/builder/blob/main/common/install_rocm_drm.sh#L100

• The code looks for "amdgpu.ids" at the root of the executable (removing /bin/exec) (i.e /path/to/bin/python => /path/to)

1. The check_for_location_of_amdgpuids always returns 0 so the search is never cancelled once the first file is found.
2. because the search is never cancelled the entire python installation is walked through
3. the file is not even installed close to the python installation so looking there is pointless anyway
4. the amgpu.ids is just a mapping of product id to marketing names; the features is far from critical but takes a significant amount of time for no reasons.
5. other part of the code will directly load /opt/amdgpu/share/libdrm/amdgpu.ids so why bother looking for it if we know where it is.

[jeffdaily]

(1) is a bug and we are working on a fix.

(2) agree, thanks for your analysis

(3) looking under the python installation is purposeful for our wheels because we must copy the amdgpu.ids file from its standard location at the time the wheel is built into the wheel to avoid annoying warnings about the missing amdgpu.ids file when using a ROCm PyTorch wheel without a full install of ROCm. We replace libdrm with this patched version so it can find this amdgpu.ids file we put there.

(4) agree, far from critical, but there were tens of warning messages about the amdgpu.ids file being missing otherwise and these were alarming our users even though they were harmless.

(5) Yes, one other part of the code will attempt to directly load the hard-coded path if the python search failed. The remaining parts of the code will print an error message referring to the incorrect path. We can fix this, too.