-
Notifications
You must be signed in to change notification settings - Fork 6.5k
people-and-planet-ai.image-classification.e2e_test: test_predict failed #7595
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
commit: 1b45e8e Test outputTraceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes |
commit: 7c48837 Test outputTraceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes |
commit: 1397651 Test outputTraceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes |
commit: 558844a Test outputTraceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes |
commit: 63d1e3d Test outputTraceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes |
commit: 83e8ee9 Test outputTraceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes |
I've filed a request to increase the quota for project python-docs-samples-tests-py37. I received a reply
|
commit: 3f81e36 Test outputTraceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes |
commit: fe8dd4c Test outputTraceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes |
@parthea Has that quota increase been granted? |
Not yet |
commit: 3b6d797 Test outputTraceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes |
commit: 029e84c Test outputTraceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 111, in model_endpoint_id deployed_model_id = deploy_model.deploy_model( File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes |
commit: 6c4c8de Test outputTraceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes |
I received an email that says quota has been adjusted but the error still appears
|
commit: 3b0d547 Test outputTraceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes |
commit: f7df2bb Test outputTraceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes |
I’m going to close this issue and see if it reappears. I already submitted a request to increase the quota. |
Looks like this issue is flaky. 😟 I'm going to leave this open and stop commenting. A human should fix and close this. commit: c35e0a6 Test outputTraceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes |
Still in progress:
|
I'm going to close this issue. Flaky bot will re-open it if there is still a quota issue. |
Oops! Looks like this issue is still flaky. It failed again. 😬 I reopened the issue, but a human will need to close it again. commit: 1062b16 Test outputTraceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes |
The quota request is still in progress. I'm going to close this issue. Flaky bot will re-open it if there is still a quota issue. |
Oops! Looks like this issue is still flaky. It failed again. 😬 I reopened the issue, but a human will need to close it again. commit: cf1643c Test outputTraceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable return callable_(*args, **kwargs) File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.NOT_FOUND details = "Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found." debug_error_string = "{"created":"@1649418272.249736443","description":"Error received from peer ipv4:74.125.142.95:443","file":"src/core/lib/surface/call.cc","file_line":903,"grpc_message":"Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found.","grpc_status":5}" > |
The quota request is still in progress. I'm going to close this issue. Flaky bot will re-open it if there is still a quota issue. |
Oops! Looks like this issue is still flaky. It failed again. 😬 I reopened the issue, but a human will need to close it again. commit: 15aaa57 Test outputTraceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable return callable_(*args, **kwargs) File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.NOT_FOUND details = "Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found." debug_error_string = "{"created":"@1649586003.269965779","description":"Error received from peer ipv4:74.125.195.95:443","file":"src/core/lib/surface/call.cc","file_line":903,"grpc_message":"Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found.","grpc_status":5}" > |
Current status on quota request:
|
Oops! Looks like this issue is still flaky. It failed again. 😬 I reopened the issue, but a human will need to close it again. commit: 568d19b Test outputTraceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable return callable_(*args, **kwargs) File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.NOT_FOUND details = "Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found." debug_error_string = "{"created":"@1649758879.742826862","description":"Error received from peer ipv4:74.125.195.95:443","file":"src/core/lib/surface/call.cc","file_line":903,"grpc_message":"Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found.","grpc_status":5}" > |
Googlers see b/227460894 |
Note: #6734 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.
commit: aa47539
buildURL: Build Status, Sponge
status: failed
Test output
The text was updated successfully, but these errors were encountered: