people-and-planet-ai.image-classification.e2e_test: test_predict failed

[flaky-bot]

Note: #6463 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.

---

commit: 35873ab
buildURL: Build Status, Sponge
status: failed

Test output

Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "{"created":"@1632652201.879730366","description":"Error received from peer ipv4:74.125.142.95:443","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):

File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id

PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id

File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model

deployed_model = response.result()

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 130, in result

self._blocking_poll(timeout=timeout, **kwargs)

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 108, in blocking_poll

retry(self._done_or_raise)(**kwargs)

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/retry.py", line 291, in retry_wrapped_func

on_error=on_error,

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/retry.py", line 189, in retry_target

return target()

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 86, in _done_or_raise

if not self.done(**kwargs):

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/operation.py", line 170, in done

self._refresh_and_update(retry)

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/operation.py", line 158, in _refresh_and_update

self._operation = self._refresh(retry=retry)

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/operations_v1/operations_client.py", line 143, in get_operation

request, retry=retry, timeout=timeout, metadata=metadata

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 145, in call

return wrapped_func(*args, **kwargs)

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/retry.py", line 291, in retry_wrapped_func

on_error=on_error,

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/retry.py", line 189, in retry_target

return target()

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout

return func(*args, **kwargs)

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable

six.raise_from(exceptions.from_grpc_error(exc), exc)

File "", line 3, in raise_from

google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded

[flaky-bot]

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

---

When run at the same commit (35873ab), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).

[davidcavazos]

It looks like a transient error when deploying the model into Vertex AI.

Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "{"created":"@1632652201.879730366","description":"Error received from peer ipv4:74.125.142.95:443","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 130, in result
    self._blocking_poll(timeout=timeout, **kwargs)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 108, in _blocking_poll
    retry_(self._done_or_raise)(**kwargs)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/retry.py", line 291, in retry_wrapped_func
    on_error=on_error,
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 86, in _done_or_raise
    if not self.done(**kwargs):
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/operation.py", line 170, in done
    self._refresh_and_update(retry)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/operation.py", line 158, in _refresh_and_update
    self._operation = self._refresh(retry=retry)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/operations_v1/operations_client.py", line 143, in get_operation
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
    return wrapped_func(*args, **kwargs)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/retry.py", line 291, in retry_wrapped_func
    on_error=on_error,
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded

[leahecole]

@davidcavazos is this issue still an issue? if not, we can close as obsolete

[davidcavazos]

I haven't seen it reproduce. Let's close this for now.

[flaky-bot]

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

---

commit: 238dbf2
buildURL: Build Status, Sponge
status: failed

Test output

Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes

[davidcavazos]

Quota issue, should be transient. Closing issue.

google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes