Skip to content

Resource status not found #45

@vishalbollu

Description

@vishalbollu

Description

Sometimes a deployment fails in the aggregation step.

Application Configuration

Iris examples

To Reproduce

Not consistently reproducible.

  1. Deploy iris example

Stack Trace

cortex logs sepal_width_mean

ERROR:cortex:An error occurred, see cx logs raw_column petal_width for more details.
Traceback (most recent call last):
  File "/src/spark_job/spark_job.py", line 311, in run_job
    run_custom_aggregators(spark, ctx, cols_to_aggregate, raw_df)
  File "/src/spark_job/spark_job.py", line 209, in run_custom_aggregators
    ctx.upload_resource_status_success(*builtin_aggregates)
  File "/src/lib/context.py", line 442, in upload_resource_status_success
    self.upload_resource_status_end("succeeded", *resources)
  File "/src/lib/context.py", line 450, in upload_resource_status_end
    status = self.get_resource_status(resource)
  File "/src/lib/context.py", line 411, in get_resource_status
    return aws.read_json_from_s3(key, self.bucket)
  File "/src/lib/aws.py", line 154, in read_json_from_s3
    obj = read_bytes_from_s3(key, bucket, allow_missing, client_config).decode("utf-8")
AttributeError: 'NoneType' object has no attribute 'decode'
19/03/25 21:38:15 SPARK:WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)

Version

0.2

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions