MIG gives null pointer exception and does not retry when MinIO is down #183
Description
Description
We currently have an e2e test for resiliency that requires briefly putting MinIO down and seeing how the retry is handled. Right now MIG seems to give an exception and then not retry.
Steps to reproduce
- Ensure all required containers are up minus MinIO
- Send a dicom through MIG
Expected behavior
MIG would not be able to proceed but would retry the request a configurable number of times with a configurable timeout
Actual behavior
The exception below is given and the payload is deleted instantly
2022-09-29T10:26:55.600071009Z [INFO] ORTHANC -> C-Store response [1]: Success
2022-09-29T10:26:55.601679439Z [INFO] ORTHANC <- Association release request
2022-09-29T10:26:55.601900042Z <6> 10:26:55 MONAI[209] => Association=#f1eac5db-c506-4857-bf62-68e9dd8c24b1 10.233.119
.56:58240 Association release request received.
2022-09-29T10:26:55.602642291Z [INFO] ORTHANC -> Association release response
2022-09-29T10:26:55.603977867Z [INFO] Connection closed
2022-09-29T10:26:58.361016184Z <4> 10:26:58 Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService[4000]
=> File ID=1.3.6.1.4.1.5962.99.1.2968617883.1314880426.1493322302363.3.0/1.3.6.1.4.1.5962.99.1.2968617883.1314880426.
1493322302363.4.0/1.3.6.1.4.1.5962.99.1.2968617883.1314880426.1493322302363.2.0, Correlation ID=f1eac5db-c506-4857-bf6
2-68e9dd8c24b1 Failed to upload file 1.3.6.1.4.1.5962.99.1.2968617883.1314880426.1493322302363.3.0/1.3.6.1.4.1.5962.99
.1.2968617883.1314880426.1493322302363.4.0/1.3.6.1.4.1.5962.99.1.2968617883.1314880426.1493322302363.2.0; added back t
o queue for retry. System.NullReferenceException: Object reference not set to an instance of an object. at Minio.Mi
nioClient.PutObjectAsync(PutObjectArgs args, CancellationToken cancellationToken) in /root/.q/sources/minio-dotnet/Min
io/ApiEndpoints/ObjectOperations.cs:line 710 at Monai.Deploy.Storage.MinIO.MinIoStorageService.PutObjectUsingClient
(MinioClient client, String bucketName, String objectName, Stream data, Int64 size, String contentType, Dictionary`2 m
etadata, CancellationToken cancellationToken) at Monai.Deploy.Storage.MinIO.MinIoStorageService.PutObjectAsync(Stri
ng bucketName, String objectName, Stream data, Int64 size, String contentType, Dictionary`2 metadata, CancellationToke
n cancellationToken) at Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService.<>c__DisplayClass22_0.
<<UploadData>b__1>d.MoveNext() in /app/src/InformaticsGateway/Services/Storage/ObjectUploadService.cs:line 226 --- End
of stack trace from previous location --- at Polly.AsyncPolicy.<>c__DisplayClass40_0.<<ImplementationAsync>b__0>d.
MoveNext() --- End of stack trace from previous location --- at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TR
esult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPr
edicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`
1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext) at Polly.AsyncPolicy.E
xecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
at Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService.UploadData(String identifier, StorageObjectMe
tadata storageObjectMetadata, String source, List`1 workflows, CancellationToken cancellationToken) in /app/src/Inform
aticsGateway/Services/Storage/ObjectUploadService.cs:line 228 at Monai.Deploy.InformaticsGateway.Services.Storage.O
bjectUploadService.ProcessObject(FileStorageMetadata blob) in /app/src/InformaticsGateway/Services/Storage/ObjectUploa
dService.cs:line 179
2022-09-29T10:27:01.079697046Z <3> 10:27:01 Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler[3014]
=> Correlation ID=f1eac5db-c506-4857-bf62-68e9dd8c24b1 Payload deleted due to upload failure(s) 1.3.6.1.4.1.5962.99.1
.2968617883.1314880426.1493322302363.3.0.