Resource status not found

[vishalbollu]

Description

Sometimes a deployment fails in the aggregation step.

Application Configuration

Iris examples

To Reproduce

Not consistently reproducible.

1. Deploy iris example

Stack Trace

cortex logs sepal_width_mean

ERROR:cortex:An error occurred, see cx logs raw_column petal_width for more details.
Traceback (most recent call last):
  File "/src/spark_job/spark_job.py", line 311, in run_job
    run_custom_aggregators(spark, ctx, cols_to_aggregate, raw_df)
  File "/src/spark_job/spark_job.py", line 209, in run_custom_aggregators
    ctx.upload_resource_status_success(*builtin_aggregates)
  File "/src/lib/context.py", line 442, in upload_resource_status_success
    self.upload_resource_status_end("succeeded", *resources)
  File "/src/lib/context.py", line 450, in upload_resource_status_end
    status = self.get_resource_status(resource)
  File "/src/lib/context.py", line 411, in get_resource_status
    return aws.read_json_from_s3(key, self.bucket)
  File "/src/lib/aws.py", line 154, in read_json_from_s3
    obj = read_bytes_from_s3(key, bucket, allow_missing, client_config).decode("utf-8")
AttributeError: 'NoneType' object has no attribute 'decode'
19/03/25 21:38:15 SPARK:WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)

Version

0.2