-
Notifications
You must be signed in to change notification settings - Fork 607
Closed
Description
Description
Sometimes a deployment fails in the aggregation step.
Application Configuration
Iris examples
To Reproduce
Not consistently reproducible.
- Deploy iris example
Stack Trace
cortex logs sepal_width_mean
ERROR:cortex:An error occurred, see cx logs raw_column petal_width for more details.
Traceback (most recent call last):
File "/src/spark_job/spark_job.py", line 311, in run_job
run_custom_aggregators(spark, ctx, cols_to_aggregate, raw_df)
File "/src/spark_job/spark_job.py", line 209, in run_custom_aggregators
ctx.upload_resource_status_success(*builtin_aggregates)
File "/src/lib/context.py", line 442, in upload_resource_status_success
self.upload_resource_status_end("succeeded", *resources)
File "/src/lib/context.py", line 450, in upload_resource_status_end
status = self.get_resource_status(resource)
File "/src/lib/context.py", line 411, in get_resource_status
return aws.read_json_from_s3(key, self.bucket)
File "/src/lib/aws.py", line 154, in read_json_from_s3
obj = read_bytes_from_s3(key, bucket, allow_missing, client_config).decode("utf-8")
AttributeError: 'NoneType' object has no attribute 'decode'
19/03/25 21:38:15 SPARK:WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)
Version
0.2
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working