Can't use AWS Instance GPU on GITLAB CI and CML-RUNNER

[leoitcode]

I have this gitlab-ci.yml:

stages:
  - test
  - deploy
  - train

sast:
  stage: test
include:
- template: Security/SAST.gitlab-ci.yml

deploy_job:
  stage: deploy
  when: always
  image: iterativeai/cml:0-dvc2-base1
  script:
    - cml-runner
      --cloud aws
      --cloud-region us-east-1
      --cloud-type g3.4xlarge
      --cloud-hdd-size 64
      --cloud-aws-security-group="cml-runners-sg"
      --labels=cml-runner-gpu
      --idle-timeout=120
train_job:
  stage: train
  when: on_success
  image: iterativeai/cml:0-dvc2-base1-gpu
  tags:
    - cml-runner-gpu
  before_script:
    - pip install poetry
    - poetry --version
    - poetry config virtualenvs.create false
    - poetry install -vv
    - nvdia-smi
  script:
    # DVC Stuff
    - dvc pull
    - dvc repro -m
    - dvc push
    # Report metrics
    - echo "## Metrics" >> report.md
    - echo "\`\`\`json" >> report.md
    - cat metrics/best-meta.json >> report.md
    - echo "\`\`\`" >> report.md
    # Report GPU details
    - echo "## GPU info" >> report.md
    - cat gpu_info.txt >> report.md
    # Send comment
    - cml-send-comment report.md

But, the container can't recognize driver or GPU, on nvidia-smi command I had the following error:

/usr/bin/bash: line 133: nvdia-smi: command not found

I realized that iterativeai/cml:0-dvc2-base1-gpu can't use instance GPU. How could I install nvidia drivers and the nvidia-docker and activate
--gpus option on this docker?

Thank you

[0x2b3bfa0]

👋 Hello, @leoitcode! Can you please connect to the instance through SSH and retrieve some additional information?

$ npm --version
$ cat /var/log/*cloud*
$ nvidia-smi

[leoitcode]

Hello

wave Hello, @leoitcode! Can you please connect to the instance through SSH and retrieve some additional information?

$ npm --version
$ cat /var/log/*cloud*
$ nvidia-smi

I tried to follow the tutorial but I'm having this error:

Unknown argument: --cloud-ssh-private=MY_RSA_KEY

I'm using this command:
cml-runner --cloud aws --cloud-region us-east-1 --cloud-type g3.4xlarge --cloud-hdd-size 64 --cloud-aws-security-group="cml-runners-sg" --labels=cml-runner-gpu --idle-timeout=-1 --cloud-ssh-private="$(cat cml.pem)"

[leoitcode]

Just adding the job log on CI of the deploy_job step:
deploy_job.txt

and the train_job step:
job_log.txt

[dacbd]

@leoitcode My guess is that it is having trouble parsing the output from cat as an argument with the spaces/dashes, and the like...

🤔 there might be a bug in handling that option, this is the first time I've seen it used. In the meantime, you can probably use the AWS web console to connect to the instance instead of trying to pass your private key.

[leoitcode]

@leoitcode My guess is that it is having trouble parsing the output from cat as an argument with the spaces/dashes, and the like...

thinking there might be a bug in handling that option, this is the first time I've seen it used. In the meantime, you can probably use the AWS web console to connect to the instance instead of trying to pass your private key.

I tried access by AWS Console, but I got the following error:

Some usernames that I tried:
ubuntu
ec2-user
root

[gitdoluquita]

@leoitcode My guess is that it is having trouble parsing the output from cat as an argument with the spaces/dashes, and the like...

thinking there might be a bug in handling that option, this is the first time I've seen it used. In the meantime, you can probably use the AWS web console to connect to the instance instead of trying to pass your private key.

@dacbd I managed to make it work by adding EOF to my pem file:

<< EOF
-----BEGIN RSA PRIVATE KEY-----
MY PRIVATE KEY HERE
-----END RSA PRIVATE KEY-----
EOF

To be honest I have no idea of how this works, I just imagined it could be that by looking at what you did here:
iterative/terraform-provider-iterative#232 (comment)

Maybe there is a more elegant way of doing this 😆

[gitdoluquita]

Me and @leoitcode are working together at this.

I connected to the deployed instance and managed to execute the nvidia-smi command:

ubuntu@ip-10-0-0-224:~$ nvidia-smi
Fri Dec 17 18:41:49 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla M60           Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   43C    P0    37W / 150W |      0MiB /  7618MiB |     92%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

But no cuda-smi command

ubuntu@ip-10-0-0-224:~$ cuda-smi
cuda-smi: command not found

I thought this might be a problem by looking at this line, but I'm not sure:
https://github.com/iterative/cml/blob/master/src/drivers/gitlab.js#L178

Testing container

Inside of the instance, if I do this:
docker run -it iterativeai/cml:0-dvc2-base1-gpu bash
I can't run nvidia-smi, as expected.

But if I do this:
docker run -it --gpus all iterativeai/cml:0-dvc2-base1-gpu bash
It works.

I couldn't find where something like this (--gpus all) is setted.

Maybe there is a config file for the gitlab-runner that should be changed for something like this: https://docs.gitlab.com/runner/configuration/gpus.html

I also notice that at Github Actions there is this option, like in this snippet:

  run:
    needs: deploy-runner
    runs-on: [self-hosted,cml-runner]
    container: 
      image: docker://iterativeai/cml:0-dvc2-base1-gpu
      options: --gpus all
 
    steps:
    - uses: actions/checkout@v2

I'm attaching the result of cat /var/log/*cloud* too, as @0x2b3bfa0 has asked:
result.log
I just edited one line for removing the GL_TOKEN that is printed, for security reasons.

Let us know if we can provide any other information.

Thanks.

[dacbd]

Wish I could help but I think we need to wait for the world to rotate back to @0x2b3bfa0 side 🌍

I see some npm install errors, you could try and add a startup script with sudo apt-get update && sudo apt-get upgrade -y && sudo apt-get install build-essential ? but I wouldn't hold my breath for that to make it work.

are your using gitlab-ci or github actions?

[DavidGOrtega]

I connected to the deployed instance and managed to execute the nvidia-smi command:

Having seen that nvidia-smi works cml should have setup the runner with the nvidia executor automatically

cml/src/drivers/gitlab.js

Line 204 in e338266

--docker-runtime "${gpu ? 'nvidia' : ''}" \

[gitdoluquita]

Having seen that nvidia-smi works cml should have setup the runner with the nvidia executor automatically

@DavidGOrtega this lines wouldn't make gpu false if cuda-smi is not working?

cml/src/drivers/gitlab.js

Lines 177 to 181 in e338266

	try {
	await exec('cuda-smi');
	} catch (err) {
	gpu = false;
	}

I think this might be the problem.

If not that what else could I be missing?

are your using gitlab-ci or github actions?

@dacbd I'm using gitlab-ci, with the yaml file that @leoitcode has posted in the first comment of the issue.

[0x2b3bfa0]

If nvidia-smi works, these lines won't run at all.