Skip to content

flaky tests: AutoML Vision Test failing with Quota Exceeded #10242

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
holtskinner opened this issue Jun 14, 2023 · 1 comment
Closed

flaky tests: AutoML Vision Test failing with Quota Exceeded #10242

holtskinner opened this issue Jun 14, 2023 · 1 comment
Assignees
Labels
api: automl Issues related to the AutoML API. priority: p2 Moderately-important priority. Fix may not be included in next release. samples Issues that are directly related to samples. triage me I really want to be triaged. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@holtskinner
Copy link
Contributor

Traceback (most recent call last):
  File "/workspace/automl/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 72, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/automl/snippets/.nox/py-3-10/lib/python3.10/site-packages/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/automl/snippets/.nox/py-3-10/lib/python3.10/site-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.RESOURCE_EXHAUSTED
	details = "Quota exceeded for quota metric 'Model creation requests' and limit 'Model creation requests per minute' of service 'automl.googleapis.com' for consumer 'project_number:1012616486416'."
	debug_error_string = "UNKNOWN:Error received from peer ipv4:142.250.107.95:443 {grpc_message:"Quota exceeded for quota metric \'Model creation requests\' and limit \'Model creation requests per minute\' of service \'automl.googleapis.com\' for consumer \'project_number:1012616486416\'.", grpc_status:8, created_time:"2023-06-13T22:15:51.484313121+00:00"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/automl/snippets/vision_object_detection_create_model_test.py", line 25, in test_vision_object_detection_create_model
    vision_object_detection_create_model.create_model(
  File "/workspace/automl/snippets/vision_object_detection_create_model.py", line 44, in create_model
    response = client.create_model(parent=project_location, model=model)
  File "/workspace/automl/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/cloud/automl_v1/services/auto_ml/client.py", line 1646, in create_model
    response = rpc(
  File "/workspace/automl/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py", line 113, in __call__
    return wrapped_func(*args, **kwargs)
  File "/workspace/automl/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
  File "/workspace/automl/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for quota metric 'Model creation requests' and limit 'Model creation requests per minute' of service 'automl.googleapis.com' for consumer 'project_number:1012616486416'. [reason: "RATE_LIMIT_EXCEEDED"
domain: "googleapis.com"
metadata {
  key: "service"
  value: "automl.googleapis.com"
}
metadata {
  key: "quota_metric"
  value: "automl.googleapis.com/create_model_requests"
}
metadata {
  key: "quota_location"
  value: "global"
}
metadata {
  key: "quota_limit"
  value: "CreateModelRequestsPerMinutePerProject"
}
metadata {
  key: "quota_limit_value"
  value: "30"
}
metadata {
  key: "consumer"
  value: "projects/1012616486416"
}
, links {
  description: "Request a higher quota limit."
  url: "https://cloud.google.com/docs/quota#requesting_higher_quota"
}
]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/automl/snippets/vision_object_detection_create_model_test.py", line 31, in test_vision_object_detection_create_model
    assert "Dataset does not exist." in e.message
AssertionError: assert 'Dataset does not exist.' in "Quota exceeded for quota metric 'Model creation requests' and limit 'Model creation requests per minute' of service 'automl.googleapis.com' for consumer 'project_number:1012616486416'."
 +  where "Quota exceeded for quota metric 'Model creation requests' and limit 'Model creation requests per minute' of service 'automl.googleapis.com' for consumer 'project_number:1012616486416'." = ResourceExhausted("Quota exceeded for quota metric 'Model creation requests' and limit 'Model creation requests per minute' of service 'automl.googleapis.com' for consumer 'project_number:1012616486416'.").message
@holtskinner holtskinner added triage me I really want to be triaged. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Jun 14, 2023
@product-auto-label product-auto-label bot added the samples Issues that are directly related to samples. label Jun 14, 2023
@holtskinner holtskinner added the api: automl Issues related to the AutoML API. label Jun 14, 2023
@leahecole
Copy link
Collaborator

Related to #10227

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: automl Issues related to the AutoML API. priority: p2 Moderately-important priority. Fix may not be included in next release. samples Issues that are directly related to samples. triage me I really want to be triaged. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

No branches or pull requests

3 participants