[Caffe2] GPU memory access fault while running OverFeat benchmark for batch size 1 and 4

[ashishfarmer]

python ../../caffe2/python/convnet_benchmarks.py --batch_size 1 --model OverFeat --net_type simple --layer_wise_benchmark True 2>&1 | tee caffe2_overfeat_bs1.txt
I0801 15:38:58.072324 30959 net_simple.cc:101] Starting benchmark.
I0801 15:38:58.072343 30959 net_simple.cc:102] Running warmup runs.
Memory access fault by GPU node-1 on address 0x524400000. Reason: Page not present or supervisor privilege.
OverFeat: running forward-backward.
*** Aborted at 1533163139 (unix time) try "date -d @1533163139" if you are using GNU date ***
PC: @ 0x7f307508d428 gsignal
*** SIGABRT (@0x3e8000078ef) received by PID 30959 (TID 0x7f3023c7c700) from PID 30959; stack trace: ***
@ 0x7f3075433390 (unknown)
@ 0x7f307508d428 gsignal
@ 0x7f307508f02a abort
@ 0x7f30423e5155 (unknown)
@ 0x7f30423ebafd (unknown)
@ 0x7f30423b6817 (unknown)
@ 0x7f30754296ba start_thread
@ 0x7f307515f41d clone
@ 0x0 (unknown)

[ashishfarmer]

Looks like a problem with a rocBLAS kernel. Working with the team on this

[Indrashish]

@ashishfarmer any update on this.

[ashishfarmer]

@Indrashish, The issue has been fixed in rocblas, but the packages have not yet been up on repo.radeon.com. To get the fix, instead of using apt-get install rocblas to install rocBLAS, try this:

wget https://github.com/ROCmSoftwarePlatform/rocBLAS/releases/download/v14.1.1/rocblas-0.14.1.101-Linux.deb
dpkg -i rocblas-0.14.1.101-Linux.deb

This should fix the problem

[ashishfarmer]

Issue fixed in the package releases with ROCm 1.9