Update existing tests for disconnected automation

[abhijeet-dhumal]

Issue link

RHOAIENG-8735

What changes have been made

• Updated existing tests to support disconnected testing
• Added YAML file and support methods for S3 compatible storage instance deployment.
• Added readme documentation for disconnected environment setup.

Verification steps

Make sure you have following pre-requisites setup for your disconnected cluster :

1. Mirror registry : Make sure you have this minio image mentioned in deployment yaml present mirror registry - For example :

quay.io/minio/minio@sha256:08dbb97c6ddcbd15138fdbd258137c4ddac5c72fdd1994a9b2d42255a98fc77b 

3. Minio storage instance : In case you don't have storage instance configured, can use disconnected_support.py file's apply_k8s_yaml and delete_k8s yaml methods using minio-deployment.yaml (Store MNIST datasets in a storage bucket similar to example shown below)
4. Pypi mirror index : To install python package requirements as a part of rayjob.
5. Export environment variables in below mentioned format :

export RAY_IMAGE=quay.io/project-codeflare/ray@sha256:72b9972b4c9fd39bfd8a1450d8227890fbda75425859a710f3514f17412ae20f \
PIP_INDEX_URL=https://bastion.dis-210gpu.aws.rhaiseng.com:9443/root/pypi/+simple/ \
PIP_TRUSTED_HOST=bastion.dis-210gpu.aws.rhaiseng.com \
MINIO_ENDPOINT=bastion.dis-210gpu.aws.rhaiseng.com:9000 \
MINIO_ACCESS_KEY=minioadmin \
MINIO_SECRET_KEY=minioadmin \
MINIO_STORAGE_BUCKET=dw-datasets \
AWS_STORAGE_BUCKET_MNIST_DIR=mnist-datasets

Run :
poetry run pytest -v -s ./tests/e2e/mnist_raycluster_sdk_oauth_test.py

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Srihari1192, sutaakar
Once this PR has been reviewed and has the lgtm label, please assign dimakis for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment