Skip to content

dataproc.snippets.update_cluster_test: test_update_cluster failed #9445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
flaky-bot bot opened this issue Mar 31, 2023 · 19 comments · Fixed by #9523, #9764, #9825, #9826 or #9894
Closed

dataproc.snippets.update_cluster_test: test_update_cluster failed #9445

flaky-bot bot opened this issue Mar 31, 2023 · 19 comments · Fixed by #9523, #9764, #9825, #9826 or #9894
Assignees
@leahecole
Labels
api: dataproc Issues related to the Dataproc API. flakybot: flaky Tells the Flaky Bot not to close or comment on this issue. flakybot: issue An issue filed by the Flaky Bot. Should not be added manually. priority: p2 Moderately-important priority. Fix may not be included in next release. samples Issues that are directly related to samples. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@flaky-bot
Copy link

flaky-bot bot commented Mar 31, 2023

Note: #8894 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.

commit: a0b5ecf
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/update_cluster_test.py", line 79, in test_update_cluster
    update_cluster.update_cluster(PROJECT_ID, REGION, CLUSTER_NAME, NEW_NUM_INSTANCES)
  File "/workspace/dataproc/snippets/update_cluster.py", line 65, in update_cluster
    updated_cluster = operation.result()
  File "/workspace/dataproc/snippets/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 137, in result
    raise self._exception
google.api_core.exceptions.Cancelled: 499 Operation partially failed:
 - 3 of 3 instances could not be added to cluster, they were deleted
 - The resource 'projects/python-docs-samples-tests/regions/us-central1/subnetworks/default' is not ready
 - The resource 'projects/python-docs-samples-tests/regions/us-central1/subnetworks/default' is not ready
 - The resource 'projects/python-docs-samples-tests/regions/us-central1/subnetworks/default' is not ready
@flaky-bot flaky-bot bot added flakybot: issue An issue filed by the Flaky Bot. Should not be added manually. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Mar 31, 2023
@product-auto-label product-auto-label bot added api: dataproc Issues related to the Dataproc API. samples Issues that are directly related to samples. labels Mar 31, 2023
@flaky-bot
Copy link
Author

flaky-bot bot commented Mar 31, 2023

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

When run at the same commit (a0b5ecf), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).

@flaky-bot flaky-bot bot added the flakybot: flaky Tells the Flaky Bot not to close or comment on this issue. label Mar 31, 2023
@engelke engelke closed this as completed Apr 6, 2023
@flaky-bot flaky-bot bot reopened this Apr 6, 2023
@flaky-bot
Copy link
Author

flaky-bot bot commented Apr 6, 2023

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

commit: 161f9cb
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/update_cluster_test.py", line 59, in setup_teardown
    operation.result()
  File "/workspace/dataproc/snippets/.nox/py-3-9/lib/python3.9/site-packages/google/api_core/future/polling.py", line 261, in result
    raise self._exception
google.api_core.exceptions.InvalidArgument: 400 The resource 'projects/python-docs-samples-tests/regions/us-central1/subnetworks/default' is not ready

@leahecole leahecole assigned leahecole and unassigned kweinmeister Apr 7, 2023
@leahecole leahecole mentioned this issue Apr 7, 2023
Merged
8 tasks
@flaky-bot
Copy link
Author

flaky-bot bot commented Apr 12, 2023

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

commit: 0bf03c7
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/update_cluster_test.py", line 85, in test_update_cluster
    update_cluster.update_cluster(PROJECT_ID, REGION, CLUSTER_NAME, NEW_NUM_INSTANCES)
  File "/workspace/dataproc/snippets/update_cluster.py", line 65, in update_cluster
    updated_cluster = operation.result()
  File "/workspace/dataproc/snippets/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 261, in result
    raise self._exception
google.api_core.exceptions.Cancelled: 499 Operation partially failed:
 - 3 of 3 instances could not be added to cluster, they were deleted
 - The resource 'projects/python-docs-samples-tests/regions/us-central1/subnetworks/default' is not ready
 - The resource 'projects/python-docs-samples-tests/regions/us-central1/subnetworks/default' is not ready
 - The resource 'projects/python-docs-samples-tests/regions/us-central1/subnetworks/default' is not ready

@flaky-bot flaky-bot bot reopened this Apr 12, 2023
@leahecole
Copy link
Collaborator

Ooo, fun, a new to me error. I'll take a look.

@leahecole
Copy link
Collaborator

Closing this to see if #9666 helped

@flaky-bot flaky-bot bot reopened this Apr 19, 2023
@flaky-bot
Copy link
Author

flaky-bot bot commented Apr 19, 2023

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

commit: b7d8eb4
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/update_cluster_test.py", line 94, in test_update_cluster
    update_cluster.update_cluster(PROJECT_ID, REGION, CLUSTER_NAME, NEW_NUM_INSTANCES)
  File "/workspace/dataproc/snippets/update_cluster.py", line 65, in update_cluster
    updated_cluster = operation.result()
  File "/workspace/dataproc/snippets/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 261, in result
    raise self._exception
google.api_core.exceptions.Cancelled: 499 Operation partially failed:
 - 3 of 3 instances could not be added to cluster, they were deleted
 - The resource 'projects/python-docs-samples-tests/regions/us-central1/subnetworks/default' is not ready
 - The resource 'projects/python-docs-samples-tests/regions/us-central1/subnetworks/default' is not ready
 - The resource 'projects/python-docs-samples-tests/regions/us-central1/subnetworks/default' is not ready

@leahecole leahecole linked a pull request Apr 24, 2023 that will close this issue
fix: changes default region for dataproc quickstart test to us-central2 #9764
Merged
@leahecole
Copy link
Collaborator

I think that #9764 should have fixed this

@flaky-bot flaky-bot bot reopened this Apr 24, 2023
@flaky-bot
Copy link
Author

flaky-bot bot commented Apr 24, 2023

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

commit: 6681669
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/update_cluster_test.py", line 94, in test_update_cluster
    update_cluster.update_cluster(PROJECT_ID, REGION, CLUSTER_NAME, NEW_NUM_INSTANCES)
  File "/workspace/dataproc/snippets/update_cluster.py", line 65, in update_cluster
    updated_cluster = operation.result()
  File "/workspace/dataproc/snippets/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 261, in result
    raise self._exception
google.api_core.exceptions.Cancelled: 499 Operation partially failed:
 - 3 of 3 instances could not be added to cluster, they were deleted
 - The resource 'projects/python-docs-samples-tests/regions/us-central1/subnetworks/default' is not ready
 - The resource 'projects/python-docs-samples-tests/regions/us-central1/subnetworks/default' is not ready
 - The resource 'projects/python-docs-samples-tests/regions/us-central1/subnetworks/default' is not ready

@leahecole
Copy link
Collaborator

@nicain checking in on this - can you remind me why we ended up reverting to us-central after the experimenting with different regions? This seems to still be an issue

@nicain nicain mentioned this issue Apr 27, 2023
Merged
@flaky-bot flaky-bot bot reopened this Apr 30, 2023
@flaky-bot
Copy link
Author

flaky-bot bot commented Apr 30, 2023

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

commit: e419fcf
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 72, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.ALREADY_EXISTS
	details = "Already exists: Failed to create cluster: Cluster projects/python-docs-samples-tests/regions/us-central1/clusters/py-cc-test-6e496aa1-bed0-44a7-a1b0-f72b87650224"
	debug_error_string = "UNKNOWN:Error received from peer ipv4:209.85.146.95:443 {grpc_message:"Already exists: Failed to create cluster: Cluster projects/python-docs-samples-tests/regions/us-central1/clusters/py-cc-test-6e496aa1-bed0-44a7-a1b0-f72b87650224", grpc_status:6, created_time:"2023-04-30T13:06:58.306931859+00:00"}"
>

The above exception was the direct cause of the following exception:


Traceback (most recent call last):

File "/workspace/dataproc/snippets/update_cluster_test.py", line 88, in test_update_cluster

setup_cluster(cluster_client)

File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry

ret = target(*args, **kwargs)

File "/workspace/dataproc/snippets/update_cluster_test.py", line 61, in setup_cluster

operation = cluster_client.create_cluster(

File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/cloud/dataproc_v1/services/cluster_controller/client.py", line 631, in create_cluster

response = rpc(

File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py", line 113, in call

return wrapped_func(*args, **kwargs)

File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func

return retry_target(

File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/retry.py", line 191, in retry_target

return target()

File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout

return func(*args, **kwargs)

File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable

raise exceptions.from_grpc_error(exc) from exc

google.api_core.exceptions.AlreadyExists: 409 Already exists: Failed to create cluster: Cluster projects/python-docs-samples-tests/regions/us-central1/clusters/py-cc-test-6e496aa1-bed0-44a7-a1b0-f72b87650224

@flaky-bot flaky-bot bot reopened this May 2, 2023
@flaky-bot
Copy link
Author

flaky-bot bot commented May 2, 2023

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

commit: 6286564
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 72, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "Cluster 'py-cc-test-b425dd62-cbb0-4499-963e-65ecd16b377d' must be running before it can be updated, current cluster state is 'ERROR'."
	debug_error_string = "UNKNOWN:Error received from peer ipv4:173.194.196.95:443 {grpc_message:"Cluster \'py-cc-test-b425dd62-cbb0-4499-963e-65ecd16b377d\' must be running before it can be updated, current cluster state is \'ERROR\'.", grpc_status:3, created_time:"2023-05-02T13:03:19.77002831+00:00"}"
>

The above exception was the direct cause of the following exception:


Traceback (most recent call last):

File "/workspace/dataproc/snippets/update_cluster_test.py", line 94, in test_update_cluster

update_cluster.update_cluster(PROJECT_ID, REGION, CLUSTER_NAME, NEW_NUM_INSTANCES)

File "/workspace/dataproc/snippets/update_cluster.py", line 56, in update_cluster

operation = client.update_cluster(

File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/cloud/dataproc_v1/services/cluster_controller/client.py", line 863, in update_cluster

response = rpc(

File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py", line 113, in call

return wrapped_func(*args, **kwargs)

File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func

return retry_target(

File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/retry.py", line 191, in retry_target

return target()

File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout

return func(*args, **kwargs)

File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable

raise exceptions.from_grpc_error(exc) from exc

google.api_core.exceptions.InvalidArgument: 400 Cluster 'py-cc-test-b425dd62-cbb0-4499-963e-65ecd16b377d' must be running before it can be updated, current cluster state is 'ERROR'.

@leahecole leahecole mentioned this issue May 4, 2023
Merged
9 tasks
@flaky-bot
Copy link
Author

flaky-bot bot commented May 8, 2023

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

commit: 43304bc
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/update_cluster_test.py", line 98, in test_update_cluster
    assert response.status.state == ClusterStatus.State.RUNNING
AssertionError: assert  == 
 +  where  = state: ERROR\nstate_start_time {\n  seconds: 1683576210\n  nanos: 18671000\n}\n.state
 +    where state: ERROR\nstate_start_time {\n  seconds: 1683576210\n  nanos: 18671000\n}\n = project_id: "python-docs-samples-tests"\ncluster_name: "py-cc-test-2cc16589-2aa6-4ec7-b82a-6584a7819fb7"\nconfig {\n  config_bucket: "dataproc-28f7d075-c5ac-4938-a5ad-65eafd8032d3-us-central1"\n  temp_bucket: "dataproc-temp-us-central1-1012616486416-3kbzlm1c"\n  gce_cluster_config {\n    zone_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/zones/us-central1-f"\n    network_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/global/networks/default"\n    service_account_scopes: "https://www.googleapis.com/auth/bigquery"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.admin.table"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.data"\n    service_account_scopes: "https://www.googleapis.com/auth/cloud.useraccounts.readonly"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.full_control"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.read_write"\n    service_account_scopes: "https://www.googleapis.com/auth/logging.write"\n  }\n  master_config {\n    num_instances: 1\n    instance_names: "py-cc-test-2cc16589-2aa6-4ec7-b82a-6584a7819fb7-m"\n    image_uri...    properties {\n      key: "distcp:mapreduce.map.memory.mb"\n      value: "768"\n    }\n    properties {\n      key: "distcp:mapreduce.map.java.opts"\n      value: "-Xmx576m"\n    }\n    properties {\n      key: "core:hadoop.ssl.enabled.protocols"\n      value: "TLSv1,TLSv1.1,TLSv1.2"\n    }\n    properties {\n      key: "core:fs.gs.metadata.cache.enable"\n      value: "false"\n    }\n    properties {\n      key: "core:fs.gs.block.size"\n      value: "134217728"\n    }\n    properties {\n      key: "capacity-scheduler:yarn.scheduler.capacity.root.default.ordering-policy"\n      value: "fair"\n    }\n  }\n  endpoint_config {\n  }\n}\nlabels {\n  key: "goog-dataproc-location"\n  value: "us-central1"\n}\nlabels {\n  key: "goog-dataproc-cluster-uuid"\n  value: "25dc4ca2-fdb6-4340-943d-2a427c996286"\n}\nlabels {\n  key: "goog-dataproc-cluster-name"\n  value: "py-cc-test-2cc16589-2aa6-4ec7-b82a-6584a7819fb7"\n}\nlabels {\n  key: "goog-dataproc-autozone"\n  value: "enabled"\n}\nstatus {\n  state: ERROR\n  state_start_time {\n    seconds: 1683576210\n    nanos: 18671000\n  }\n}\nstatus_history {\n  state: CREATING\n  state_start_time {\n    seconds: 1683576207\n    nanos: 351596000\n  }\n}\ncluster_uuid: "25dc4ca2-fdb6-4340-943d-2a427c996286"\n.status
 +  and    = .RUNNING
 +    where  = ClusterStatus.State

@flaky-bot flaky-bot bot reopened this May 8, 2023
@leahecole
Copy link
Collaborator

WAI, failed on cluster in ERROR state

@flaky-bot flaky-bot bot reopened this May 11, 2023
@flaky-bot
Copy link
Author

flaky-bot bot commented May 11, 2023

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

commit: 0b7616f
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/update_cluster_test.py", line 98, in test_update_cluster
    assert response.status.state == ClusterStatus.State.RUNNING
AssertionError: assert  == 
 +  where  = state: ERROR\nstate_start_time {\n  seconds: 1683801581\n  nanos: 931735000\n}\n.state
 +    where state: ERROR\nstate_start_time {\n  seconds: 1683801581\n  nanos: 931735000\n}\n = project_id: "python-docs-samples-tests"\ncluster_name: "py-cc-test-a77243ee-9536-4ae6-93cf-8042d6d69a0c"\nconfig {\n  config_bucket: "dataproc-28f7d075-c5ac-4938-a5ad-65eafd8032d3-us-central1"\n  temp_bucket: "dataproc-temp-us-central1-1012616486416-3kbzlm1c"\n  gce_cluster_config {\n    zone_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/zones/us-central1-f"\n    network_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/global/networks/default"\n    service_account_scopes: "https://www.googleapis.com/auth/bigquery"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.admin.table"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.data"\n    service_account_scopes: "https://www.googleapis.com/auth/cloud.useraccounts.readonly"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.full_control"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.read_write"\n    service_account_scopes: "https://www.googleapis.com/auth/logging.write"\n  }\n  master_config {\n    num_instances: 1\n    instance_names: "py-cc-test-a77243ee-9536-4ae6-93cf-8042d6d69a0c-m"\n    image_uri...    properties {\n      key: "distcp:mapreduce.map.memory.mb"\n      value: "768"\n    }\n    properties {\n      key: "distcp:mapreduce.map.java.opts"\n      value: "-Xmx576m"\n    }\n    properties {\n      key: "core:hadoop.ssl.enabled.protocols"\n      value: "TLSv1,TLSv1.1,TLSv1.2"\n    }\n    properties {\n      key: "core:fs.gs.metadata.cache.enable"\n      value: "false"\n    }\n    properties {\n      key: "core:fs.gs.block.size"\n      value: "134217728"\n    }\n    properties {\n      key: "capacity-scheduler:yarn.scheduler.capacity.root.default.ordering-policy"\n      value: "fair"\n    }\n  }\n  endpoint_config {\n  }\n}\nlabels {\n  key: "goog-dataproc-location"\n  value: "us-central1"\n}\nlabels {\n  key: "goog-dataproc-cluster-uuid"\n  value: "6393dbd8-6c99-47e9-af75-9cef70116b42"\n}\nlabels {\n  key: "goog-dataproc-cluster-name"\n  value: "py-cc-test-a77243ee-9536-4ae6-93cf-8042d6d69a0c"\n}\nlabels {\n  key: "goog-dataproc-autozone"\n  value: "enabled"\n}\nstatus {\n  state: ERROR\n  state_start_time {\n    seconds: 1683801581\n    nanos: 931735000\n  }\n}\nstatus_history {\n  state: CREATING\n  state_start_time {\n    seconds: 1683801579\n    nanos: 24856000\n  }\n}\ncluster_uuid: "6393dbd8-6c99-47e9-af75-9cef70116b42"\n.status
 +  and    = .RUNNING
 +    where  = ClusterStatus.State

@leahecole leahecole added priority: p2 Moderately-important priority. Fix may not be included in next release. and removed priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. labels May 12, 2023
@leahecole
Copy link
Collaborator

bumping to p2 because it's not a service issue

@flaky-bot
Copy link
Author

flaky-bot bot commented May 31, 2023

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

commit: 360972a
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/update_cluster_test.py", line 98, in test_update_cluster
    assert response.status.state == ClusterStatus.State.RUNNING
AssertionError: assert  == 
 +  where  = state: ERROR\nstate_start_time {\n  seconds: 1685537022\n  nanos: 392280000\n}\n.state
 +    where state: ERROR\nstate_start_time {\n  seconds: 1685537022\n  nanos: 392280000\n}\n = project_id: "python-docs-samples-tests"\ncluster_name: "py-cc-test-67685216-d8b4-44d9-90a2-d3e78e6f98a3"\nconfig {\n  config_bucket: "dataproc-28f7d075-c5ac-4938-a5ad-65eafd8032d3-us-central1"\n  temp_bucket: "dataproc-temp-us-central1-1012616486416-3kbzlm1c"\n  gce_cluster_config {\n    zone_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/zones/us-central1-f"\n    network_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/global/networks/default"\n    service_account_scopes: "https://www.googleapis.com/auth/bigquery"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.admin.table"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.data"\n    service_account_scopes: "https://www.googleapis.com/auth/cloud.useraccounts.readonly"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.full_control"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.read_write"\n    service_account_scopes: "https://www.googleapis.com/auth/logging.write"\n  }\n  master_config {\n    num_instances: 1\n    instance_names: "py-cc-test-67685216-d8b4-44d9-90a2-d3e78e6f98a3-m"\n    image_uri...    properties {\n      key: "distcp:mapreduce.map.memory.mb"\n      value: "768"\n    }\n    properties {\n      key: "distcp:mapreduce.map.java.opts"\n      value: "-Xmx576m"\n    }\n    properties {\n      key: "core:hadoop.ssl.enabled.protocols"\n      value: "TLSv1,TLSv1.1,TLSv1.2"\n    }\n    properties {\n      key: "core:fs.gs.metadata.cache.enable"\n      value: "false"\n    }\n    properties {\n      key: "core:fs.gs.block.size"\n      value: "134217728"\n    }\n    properties {\n      key: "capacity-scheduler:yarn.scheduler.capacity.root.default.ordering-policy"\n      value: "fair"\n    }\n  }\n  endpoint_config {\n  }\n}\nlabels {\n  key: "goog-dataproc-location"\n  value: "us-central1"\n}\nlabels {\n  key: "goog-dataproc-cluster-uuid"\n  value: "e3de8c40-dbf7-4cd2-b787-ec774181f8f7"\n}\nlabels {\n  key: "goog-dataproc-cluster-name"\n  value: "py-cc-test-67685216-d8b4-44d9-90a2-d3e78e6f98a3"\n}\nlabels {\n  key: "goog-dataproc-autozone"\n  value: "enabled"\n}\nstatus {\n  state: ERROR\n  state_start_time {\n    seconds: 1685537022\n    nanos: 392280000\n  }\n}\nstatus_history {\n  state: CREATING\n  state_start_time {\n    seconds: 1685537020\n    nanos: 77109000\n  }\n}\ncluster_uuid: "e3de8c40-dbf7-4cd2-b787-ec774181f8f7"\n.status
 +  and    = .RUNNING
 +    where  = ClusterStatus.State

@flaky-bot flaky-bot bot reopened this May 31, 2023
@leahecole leahecole mentioned this issue Jun 15, 2023
Closed
@leahecole
Copy link
Collaborator

Closing to see if #10527 helped

@flaky-bot
Copy link
Author

flaky-bot bot commented Aug 25, 2023

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

commit: e440097
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/update_cluster_test.py", line 107, in test_update_cluster
    assert response.status.state == ClusterStatus.State.RUNNING
AssertionError: assert  == 
 +  where  = state: ERROR\nstate_start_time {\n  seconds: 1692968912\n  nanos: 827217000\n}\n.state
 +    where state: ERROR\nstate_start_time {\n  seconds: 1692968912\n  nanos: 827217000\n}\n = project_id: "python-docs-samples-tests"\ncluster_name: "py-cc-test-ed5888c3-0ebd-4a53-bdd7-2edc7e05a7df"\nconfig {\n  config_bucket: "dataproc-28f7d075-c5ac-4938-a5ad-65eafd8032d3-us-central1"\n  temp_bucket: "dataproc-temp-us-central1-1012616486416-3kbzlm1c"\n  gce_cluster_config {\n    zone_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/zones/us-central1-f"\n    network_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/global/networks/default"\n    service_account_scopes: "https://www.googleapis.com/auth/bigquery"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.admin.table"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.data"\n    service_account_scopes: "https://www.googleapis.com/auth/cloud.useraccounts.readonly"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.full_control"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.read_write"\n    service_account_scopes: "https://www.googleapis.com/auth/logging.write"\n    service_account_scopes: "https://www.googleapis.com/auth/monitoring.write"\n  }\n  master_config {\n    num_instances: 1\n    in...    properties {\n      key: "distcp:mapreduce.map.memory.mb"\n      value: "768"\n    }\n    properties {\n      key: "distcp:mapreduce.map.java.opts"\n      value: "-Xmx576m"\n    }\n    properties {\n      key: "core:hadoop.ssl.enabled.protocols"\n      value: "TLSv1,TLSv1.1,TLSv1.2"\n    }\n    properties {\n      key: "core:fs.gs.metadata.cache.enable"\n      value: "false"\n    }\n    properties {\n      key: "core:fs.gs.block.size"\n      value: "134217728"\n    }\n    properties {\n      key: "capacity-scheduler:yarn.scheduler.capacity.root.default.ordering-policy"\n      value: "fair"\n    }\n  }\n  endpoint_config {\n  }\n}\nlabels {\n  key: "goog-dataproc-location"\n  value: "us-central1"\n}\nlabels {\n  key: "goog-dataproc-cluster-uuid"\n  value: "a582e6a6-e167-43fe-bcdd-fa8d9812fb6e"\n}\nlabels {\n  key: "goog-dataproc-cluster-name"\n  value: "py-cc-test-ed5888c3-0ebd-4a53-bdd7-2edc7e05a7df"\n}\nlabels {\n  key: "goog-dataproc-autozone"\n  value: "enabled"\n}\nstatus {\n  state: ERROR\n  state_start_time {\n    seconds: 1692968912\n    nanos: 827217000\n  }\n}\nstatus_history {\n  state: CREATING\n  state_start_time {\n    seconds: 1692968910\n    nanos: 74745000\n  }\n}\ncluster_uuid: "a582e6a6-e167-43fe-bcdd-fa8d9812fb6e"\n.status
 +  and    = .RUNNING
 +    where  = ClusterStatus.State

@flaky-bot flaky-bot bot reopened this Aug 25, 2023
@flaky-bot
Copy link
Author

flaky-bot bot commented Oct 14, 2023

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

commit: 3b1bc75
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/update_cluster_test.py", line 107, in test_update_cluster
    assert response.status.state == ClusterStatus.State.RUNNING
AssertionError: assert  == 
 +  where  = state: ERROR\nstate_start_time {\n  seconds: 1697289314\n  nanos: 908115000\n}\n.state
 +    where state: ERROR\nstate_start_time {\n  seconds: 1697289314\n  nanos: 908115000\n}\n = project_id: "python-docs-samples-tests"\ncluster_name: "py-cc-test-a8b73c46-3ba3-4bb5-ae6d-602006b361e9"\nconfig {\n  config_bucket: "dataproc-28f7d075-c5ac-4938-a5ad-65eafd8032d3-us-central1"\n  temp_bucket: "dataproc-temp-us-central1-1012616486416-3kbzlm1c"\n  gce_cluster_config {\n    zone_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/zones/us-central1-c"\n    network_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/global/networks/default"\n    service_account_scopes: "https://www.googleapis.com/auth/bigquery"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.admin.table"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.data"\n    service_account_scopes: "https://www.googleapis.com/auth/cloud.useraccounts.readonly"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.full_control"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.read_write"\n    service_account_scopes: "https://www.googleapis.com/auth/logging.write"\n    service_account_scopes: "https://www.googleapis.com/auth/monitoring.write"\n  }\n  master_config {\n    num_instances: 1\n    in...   properties {\n      key: "distcp:mapreduce.map.memory.mb"\n      value: "768"\n    }\n    properties {\n      key: "distcp:mapreduce.map.java.opts"\n      value: "-Xmx576m"\n    }\n    properties {\n      key: "core:hadoop.ssl.enabled.protocols"\n      value: "TLSv1,TLSv1.1,TLSv1.2"\n    }\n    properties {\n      key: "core:fs.gs.metadata.cache.enable"\n      value: "false"\n    }\n    properties {\n      key: "core:fs.gs.block.size"\n      value: "134217728"\n    }\n    properties {\n      key: "capacity-scheduler:yarn.scheduler.capacity.root.default.ordering-policy"\n      value: "fair"\n    }\n  }\n  endpoint_config {\n  }\n}\nlabels {\n  key: "goog-dataproc-location"\n  value: "us-central1"\n}\nlabels {\n  key: "goog-dataproc-cluster-uuid"\n  value: "4e43e0f8-43ed-4489-884a-56ccc97ece40"\n}\nlabels {\n  key: "goog-dataproc-cluster-name"\n  value: "py-cc-test-a8b73c46-3ba3-4bb5-ae6d-602006b361e9"\n}\nlabels {\n  key: "goog-dataproc-autozone"\n  value: "enabled"\n}\nstatus {\n  state: ERROR\n  state_start_time {\n    seconds: 1697289314\n    nanos: 908115000\n  }\n}\nstatus_history {\n  state: CREATING\n  state_start_time {\n    seconds: 1697289311\n    nanos: 264192000\n  }\n}\ncluster_uuid: "4e43e0f8-43ed-4489-884a-56ccc97ece40"\n.status
 +  and    = .RUNNING
 +    where  = ClusterStatus.State

@flaky-bot flaky-bot bot reopened this Oct 14, 2023
@leahecole leahecole mentioned this issue Oct 17, 2023
Merged
9 tasks
@flaky-bot flaky-bot bot mentioned this issue Apr 6, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@leahecole leahecole

Labels
api: dataproc Issues related to the Dataproc API. flakybot: flaky Tells the Flaky Bot not to close or comment on this issue. flakybot: issue An issue filed by the Flaky Bot. Should not be added manually. priority: p2 Moderately-important priority. Fix may not be included in next release. samples Issues that are directly related to samples. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
3 participants
@engelke @leahecole @kweinmeister