Skip to content

people-and-planet-ai.image-classification.e2e_test: test_predict failed #7595

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
flaky-bot bot opened this issue Mar 15, 2022 · 30 comments
Closed

people-and-planet-ai.image-classification.e2e_test: test_predict failed #7595

flaky-bot bot opened this issue Mar 15, 2022 · 30 comments
Assignees
Labels
flakybot: flaky Tells the Flaky Bot not to close or comment on this issue. flakybot: issue An issue filed by the Flaky Bot. Should not be added manually. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. 🚨 This issue needs some love. samples Issues that are directly related to samples. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@flaky-bot
Copy link

flaky-bot bot commented Mar 15, 2022

Note: #6734 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.


commit: aa47539
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes
@flaky-bot flaky-bot bot added flakybot: issue An issue filed by the Flaky Bot. Should not be added manually. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Mar 15, 2022
@product-auto-label product-auto-label bot added the samples Issues that are directly related to samples. label Mar 15, 2022
@flaky-bot
Copy link
Author

flaky-bot bot commented Mar 16, 2022

commit: 1b45e8e
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes

@flaky-bot
Copy link
Author

flaky-bot bot commented Mar 18, 2022

commit: 7c48837
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes

@flaky-bot
Copy link
Author

flaky-bot bot commented Mar 19, 2022

commit: 1397651
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes

@flaky-bot
Copy link
Author

flaky-bot bot commented Mar 21, 2022

commit: 558844a
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes

@flaky-bot
Copy link
Author

flaky-bot bot commented Mar 22, 2022

commit: 63d1e3d
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes

@yoshi-automation yoshi-automation added the 🚨 This issue needs some love. label Mar 22, 2022
@flaky-bot
Copy link
Author

flaky-bot bot commented Mar 24, 2022

commit: 83e8ee9
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes

@parthea
Copy link
Collaborator

parthea commented Mar 24, 2022

I've filed a request to increase the quota for project python-docs-samples-tests-py37. I received a reply

This message is to confirm that we've received your quota request for 
project 'python-docs-samples-tests-py37'. Quota increase requests typically 
take two business days to process. If this quota request is very urgent, 
please respond to this email so that our support agents can have full 
context when processing your quota increase request.

@parthea parthea assigned parthea and unassigned busunkim96 Mar 24, 2022
@flaky-bot
Copy link
Author

flaky-bot bot commented Mar 25, 2022

commit: 3f81e36
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes

@flaky-bot
Copy link
Author

flaky-bot bot commented Mar 26, 2022

commit: fe8dd4c
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes

@busunkim96
Copy link
Contributor

@parthea Has that quota increase been granted?

@parthea
Copy link
Collaborator

parthea commented Mar 28, 2022

Not yet

@flaky-bot
Copy link
Author

flaky-bot bot commented Mar 29, 2022

commit: 3b6d797
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes

@flaky-bot
Copy link
Author

flaky-bot bot commented Mar 29, 2022

commit: 029e84c
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 111, in model_endpoint_id
    deployed_model_id = deploy_model.deploy_model(
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes

@flaky-bot
Copy link
Author

flaky-bot bot commented Mar 30, 2022

commit: 6c4c8de
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes

@parthea
Copy link
Collaborator

parthea commented Mar 30, 2022

I received an email that says quota has been adjusted but the error still appears

Your quota request for python-docs-samples-tests-py37 has been approved and your project quota
has been adjusted accordingly. To verify the quota change, please navigate to
https://console.cloud.google.com/, select your project, select Google Cloud
AutoML under APIs, and then click on "Quotas" tab on the top of the page.
________________________ ERROR at setup of test_predict ________________________
[gw2] linux -- Python 3.7.12 /workspace/people-and-planet-ai/image-classification/.nox/py-3-7/bin/python
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes

image

@parthea
Copy link
Collaborator

parthea commented Mar 30, 2022

It turns out that the project with a quota issue is python-docs-samples-tests not python-docs-samples-tests-py37. I saw python-docs-samples-tests-py37 in the build log so I expect it to use that project.

I've submitted a new request for the correct project.

image

@flaky-bot
Copy link
Author

flaky-bot bot commented Mar 31, 2022

commit: 3b0d547
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes

@flaky-bot
Copy link
Author

flaky-bot bot commented Apr 1, 2022

commit: f7df2bb
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes

@parthea
Copy link
Collaborator

parthea commented Apr 4, 2022

I’m going to close this issue and see if it reappears. I already submitted a request to increase the quota.

@parthea parthea closed this as completed Apr 4, 2022
@flaky-bot flaky-bot bot reopened this Apr 5, 2022
@flaky-bot flaky-bot bot added the flakybot: flaky Tells the Flaky Bot not to close or comment on this issue. label Apr 5, 2022
@flaky-bot
Copy link
Author

flaky-bot bot commented Apr 5, 2022

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.


commit: c35e0a6
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes

@parthea
Copy link
Collaborator

parthea commented Apr 5, 2022

Still in progress:

This is an update for your quota increase request. Our Systems engineers are now working on your case. We will keep you informed as to any and all significant updates as soon as we receive them from the concerned teams. Please expect an update from us in the next 5 days.

@parthea
Copy link
Collaborator

parthea commented Apr 5, 2022

I'm going to close this issue. Flaky bot will re-open it if there is still a quota issue.

@parthea parthea closed this as completed Apr 5, 2022
@flaky-bot flaky-bot bot reopened this Apr 6, 2022
@flaky-bot
Copy link
Author

flaky-bot bot commented Apr 6, 2022

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.


commit: 1062b16
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes

@parthea
Copy link
Collaborator

parthea commented Apr 8, 2022

The quota request is still in progress. I'm going to close this issue. Flaky bot will re-open it if there is still a quota issue.

@parthea parthea closed this as completed Apr 8, 2022
@flaky-bot
Copy link
Author

flaky-bot bot commented Apr 8, 2022

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.


commit: cf1643c
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.NOT_FOUND
	details = "Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found."
	debug_error_string = "{"created":"@1649418272.249736443","description":"Error received from peer ipv4:74.125.142.95:443","file":"src/core/lib/surface/call.cc","file_line":903,"grpc_message":"Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found.","grpc_status":5}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 111, in model_endpoint_id
deployed_model_id = deploy_model.deploy_model(
File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 65, in deploy_model
response = client.deploy_model(
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/cloud/aiplatform_v1/services/endpoint_service/client.py", line 1157, in deploy_model
response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py", line 145, in call
return wrapped_func(*args, **kwargs)
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "", line 3, in raise_from
google.api_core.exceptions.NotFound: 404 Model projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768 is not found.

@flaky-bot flaky-bot bot reopened this Apr 8, 2022
@parthea
Copy link
Collaborator

parthea commented Apr 9, 2022

The quota request is still in progress. I'm going to close this issue. Flaky bot will re-open it if there is still a quota issue.

@parthea parthea closed this as completed Apr 9, 2022
@flaky-bot flaky-bot bot reopened this Apr 10, 2022
@flaky-bot
Copy link
Author

flaky-bot bot commented Apr 10, 2022

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.


commit: 15aaa57
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.NOT_FOUND
	details = "Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found."
	debug_error_string = "{"created":"@1649586003.269965779","description":"Error received from peer ipv4:74.125.195.95:443","file":"src/core/lib/surface/call.cc","file_line":903,"grpc_message":"Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found.","grpc_status":5}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 77, in deploy_model
traffic_split={"0": 100},
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/cloud/aiplatform_v1/services/endpoint_service/client.py", line 1157, in deploy_model
response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 145, in call
return wrapped_func(*args, **kwargs)
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "", line 3, in raise_from
google.api_core.exceptions.NotFound: 404 Model projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768 is not found.

@parthea
Copy link
Collaborator

parthea commented Apr 11, 2022

Current status on quota request:

We are still working on your request. Rest assured that we are keeping a close eye on this for any changes on your case. Please expect an update from us in the next 5 days or as soon as we receive an update from the engineering team.

@parthea parthea closed this as completed Apr 11, 2022
@flaky-bot
Copy link
Author

flaky-bot bot commented Apr 12, 2022

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.


commit: 568d19b
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.NOT_FOUND
	details = "Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found."
	debug_error_string = "{"created":"@1649758879.742826862","description":"Error received from peer ipv4:74.125.195.95:443","file":"src/core/lib/surface/call.cc","file_line":903,"grpc_message":"Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found.","grpc_status":5}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 77, in deploy_model
traffic_split={"0": 100},
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/cloud/aiplatform_v1/services/endpoint_service/client.py", line 1157, in deploy_model
response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 145, in call
return wrapped_func(*args, **kwargs)
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "", line 3, in raise_from
google.api_core.exceptions.NotFound: 404 Model projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768 is not found.

@flaky-bot flaky-bot bot reopened this Apr 12, 2022
@parthea
Copy link
Collaborator

parthea commented Apr 14, 2022

Googlers see b/227460894

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flakybot: flaky Tells the Flaky Bot not to close or comment on this issue. flakybot: issue An issue filed by the Flaky Bot. Should not be added manually. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. 🚨 This issue needs some love. samples Issues that are directly related to samples. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

No branches or pull requests

3 participants