people-and-planet-ai.image-classification.e2e_test: test_predict failed

[flaky-bot]

Note: #5758 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.

---

commit: 6abb1e5
buildURL: Build Status, Sponge
status: failed

Test output

Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 134, in result
    raise self._exception
google.api_core.exceptions.InternalServerError: 500 INTERNAL

[flaky-bot]

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

---

When run at the same commit (6abb1e5), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).

[dandhlee]

Hey @davidcavazos, it seems like it ran into a 500 issue:

Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 134, in result
    raise self._exception
google.api_core.exceptions.InternalServerError: 500 INTERNAL

the tests are passing now. Do you think it's worth taking a look? Or would it be okay to close it for now and investigate further if it happens again?

[dandhlee]

The test hasn't failed since, going to close this off for now and investigate further later.

[flaky-bot]

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

---

commit: e774986
buildURL: Build Status, Sponge
status: failed

Test output

Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/future/polling.py", line 134, in result
    raise self._exception
google.api_core.exceptions.InternalServerError: 500 INTERNAL

[dandhlee]

Investigating, reaching out to necessary DPEs and service team to try and mitigate the 500 errors.

[dandhlee]

Filed an internal bug to investigate, will update back on this issue once I hear back.

[davidcavazos]

We've seen that 500 error intermittently before. We suspect it's some issue with the product service. Unfortunately, I'm not sure if we can do anything about it other than retrying.

[leahecole]

Sounds good. @dandhlee can you just put the bug number here so we can follow along? Instead of the link you can just call it "internal bug "

[dandhlee]

Noted.
Internal bug filed: 188940374

[dandhlee]

As all tests have been passing lately, will update this issue again and reopen if needed.