Skip to content

flaky tests: AutoML Vision Test failing with Quota Exceeded #10242

@holtskinner

Description

@holtskinner
Traceback (most recent call last):
  File "/workspace/automl/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 72, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/automl/snippets/.nox/py-3-10/lib/python3.10/site-packages/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/automl/snippets/.nox/py-3-10/lib/python3.10/site-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.RESOURCE_EXHAUSTED
	details = "Quota exceeded for quota metric 'Model creation requests' and limit 'Model creation requests per minute' of service 'automl.googleapis.com' for consumer 'project_number:1012616486416'."
	debug_error_string = "UNKNOWN:Error received from peer ipv4:142.250.107.95:443 {grpc_message:"Quota exceeded for quota metric \'Model creation requests\' and limit \'Model creation requests per minute\' of service \'automl.googleapis.com\' for consumer \'project_number:1012616486416\'.", grpc_status:8, created_time:"2023-06-13T22:15:51.484313121+00:00"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/automl/snippets/vision_object_detection_create_model_test.py", line 25, in test_vision_object_detection_create_model
    vision_object_detection_create_model.create_model(
  File "/workspace/automl/snippets/vision_object_detection_create_model.py", line 44, in create_model
    response = client.create_model(parent=project_location, model=model)
  File "/workspace/automl/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/cloud/automl_v1/services/auto_ml/client.py", line 1646, in create_model
    response = rpc(
  File "/workspace/automl/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py", line 113, in __call__
    return wrapped_func(*args, **kwargs)
  File "/workspace/automl/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
  File "/workspace/automl/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for quota metric 'Model creation requests' and limit 'Model creation requests per minute' of service 'automl.googleapis.com' for consumer 'project_number:1012616486416'. [reason: "RATE_LIMIT_EXCEEDED"
domain: "googleapis.com"
metadata {
  key: "service"
  value: "automl.googleapis.com"
}
metadata {
  key: "quota_metric"
  value: "automl.googleapis.com/create_model_requests"
}
metadata {
  key: "quota_location"
  value: "global"
}
metadata {
  key: "quota_limit"
  value: "CreateModelRequestsPerMinutePerProject"
}
metadata {
  key: "quota_limit_value"
  value: "30"
}
metadata {
  key: "consumer"
  value: "projects/1012616486416"
}
, links {
  description: "Request a higher quota limit."
  url: "https://cloud.google.com/docs/quota#requesting_higher_quota"
}
]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/automl/snippets/vision_object_detection_create_model_test.py", line 31, in test_vision_object_detection_create_model
    assert "Dataset does not exist." in e.message
AssertionError: assert 'Dataset does not exist.' in "Quota exceeded for quota metric 'Model creation requests' and limit 'Model creation requests per minute' of service 'automl.googleapis.com' for consumer 'project_number:1012616486416'."
 +  where "Quota exceeded for quota metric 'Model creation requests' and limit 'Model creation requests per minute' of service 'automl.googleapis.com' for consumer 'project_number:1012616486416'." = ResourceExhausted("Quota exceeded for quota metric 'Model creation requests' and limit 'Model creation requests per minute' of service 'automl.googleapis.com' for consumer 'project_number:1012616486416'.").message

Metadata

Metadata

Assignees

Labels

api: automlIssues related to the AutoML API.priority: p2Moderately-important priority. Fix may not be included in next release.samplesIssues that are directly related to samples.triage meI really want to be triaged.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions