MIG doesnt complete the processing of inbound data #347
Description
Description
When sending data to MIG, it gets in a state where it doesnt complete the processing of an inbound association and then can no longer process any further associations. By process I mean move to MinIO and generate the WorkflowRequestEvent, the inbound associatiosna re fine and it is added to a queue for processing which never happens.
Steps to reproduce
Unfortunately there are no firm repeatable steps for this. Logs seen for this transaction are below
2023-03-10T12:39:13.597065418Z 2023-03-10 12:39:13.5958|4006|INFO|Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService|File ID=1.2.826.0.1.2112370.55.1.13410264/1.3.12.2.1107.5.2.18.141028.2021051414181560370700792/1.3.12.2.1107.5.2.18.141028.2021051414184438235300876, CorrelationId=f3146e9b-e020-4a4f-ae1a-ba5d3feb508c|File uploaded to temporary store at f3146e9b-e020-4a4f-ae1a-ba5d3feb508c/dcm/96b281c0-ae5a-4d80-b58b-65601a064f91.dcm.json. 2023-03-10T12:39:13.608835350Z 2023-03-10 12:39:13.6044|4010|DEBUG|Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService|File ID=1.2.826.0.1.2112370.55.1.13410264/1.3.12.2.1107.5.2.18.141028.2021051414181560370700792/1.3.12.2.1107.5.2.18.141028.2021051414184438235300876, CorrelationId=f3146e9b-e020-4a4f-ae1a-ba5d3feb508c|File /incoming/f3146e9b-e020-4a4f-ae1a-ba5d3feb508c/dcm/96b281c0-ae5a-4d80-b58b-65601a064f91.dcm.json exists=True. 2023-03-10T12:39:13.608841913Z 2023-03-10 12:39:13.6045|4002|DEBUG|Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService|File ID=1.2.826.0.1.2112370.55.1.13410264/1.3.12.2.1107.5.2.18.141028.2021051414181560370700792/1.3.12.2.1107.5.2.18.141028.2021051414184438235300876, CorrelationId=f3146e9b-e020-4a4f-ae1a-ba5d3feb508c|Uploading file to temporary store at f3146e9b-e020-4a4f-ae1a-ba5d3feb508c/dcm/96b281c0-ae5a-4d80-b58b-65601a064f91.dcm. 2023-03-10T12:39:13.652782416Z 2023-03-10 12:39:13.6505|4006|INFO|Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService|File ID=1.2.826.0.1.2112370.55.1.13410264/1.3.12.2.1107.5.2.18.141028.2021051414181560370700792/1.3.12.2.1107.5.2.18.141028.2021051414184438235300876, CorrelationId=f3146e9b-e020-4a4f-ae1a-ba5d3feb508c|File uploaded to temporary store at f3146e9b-e020-4a4f-ae1a-ba5d3feb508c/dcm/96b281c0-ae5a-4d80-b58b-65601a064f91.dcm. 2023-03-10T12:39:13.664870429Z 2023-03-10 12:39:13.6604|4010|DEBUG|Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService|File ID=1.2.826.0.1.2112370.55.1.13410264/1.3.12.2.1107.5.2.18.141028.2021051414181560370700792/1.3.12.2.1107.5.2.18.141028.2021051414184438235300876, CorrelationId=f3146e9b-e020-4a4f-ae1a-ba5d3feb508c|File /incoming/f3146e9b-e020-4a4f-ae1a-ba5d3feb508c/dcm/96b281c0-ae5a-4d80-b58b-65601a064f91.dcm exists=True. 2023-03-10T12:39:13.664874797Z 2023-03-10 12:39:13.6606|4001|DEBUG|Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService|File ID=1.2.826.0.1.2112370.55.1.13410264/1.3.12.2.1107.5.2.18.141028.2021051414181560370700792/1.3.12.2.1107.5.2.18.141028.2021051414184438235300876, CorrelationId=f3146e9b-e020-4a4f-ae1a-ba5d3feb508c|Upload statistics: 2 threads, 0.0747231 seconds. 2023-03-10T12:39:13.664877773Z 2023-03-10 12:39:13.6606|4007|DEBUG|Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadQueue||Items in queue 35. 2023-03-10T12:39:13.664880097Z 2023-03-10 12:39:13.6607|4002|DEBUG|Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService|File ID=1.2.826.0.1.2112370.55.1.13410264/1.3.12.2.1107.5.2.18.141028.2021051414181560370700792/1.3.12.2.1107.5.2.18.141028.2021051414183590186800860, CorrelationId=f3146e9b-e020-4a4f-ae1a-ba5d3feb508c|Uploading file to temporary store at f3146e9b-e020-4a4f-ae1a-ba5d3feb508c/dcm/39e2cbc0-bf90-4209-bca3-e6a73a91634b.dcm.json. 2023-03-10T12:39:13.672585365Z 2023-03-10 12:39:13.6703|4006|INFO|Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService|File ID=1.2.826.0.1.2112370.55.1.13410264/1.3.12.2.1107.5.2.18.141028.2021051414181560370700792/1.3.12.2.1107.5.2.18.141028.2021051414183590186800860, CorrelationId=f3146e9b-e020-4a4f-ae1a-ba5d3feb508c|File uploaded to temporary store at f3146e9b-e020-4a4f-ae1a-ba5d3feb508c/dcm/39e2cbc0-bf90-4209-bca3-e6a73a91634b.dcm.json. 2023-03-10T12:39:13.680695759Z 2023-03-10 12:39:13.6784|4010|DEBUG|Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService|File ID=1.2.826.0.1.2112370.55.1.13410264/1.3.12.2.1107.5.2.18.141028.2021051414181560370700792/1.3.12.2.1107.5.2.18.141028.2021051414183590186800860, CorrelationId=f3146e9b-e020-4a4f-ae1a-ba5d3feb508c|File /incoming/f3146e9b-e020-4a4f-ae1a-ba5d3feb508c/dcm/39e2cbc0-bf90-4209-bca3-e6a73a91634b.dcm.json exists=True. 2023-03-10T12:39:13.680703423Z 2023-03-10 12:39:13.6785|4002|DEBUG|Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService|File ID=1.2.826.0.1.2112370.55.1.13410264/1.3.12.2.1107.5.2.18.141028.2021051414181560370700792/1.3.12.2.1107.5.2.18.141028.2021051414183590186800860, CorrelationId=f3146e9b-e020-4a4f-ae1a-ba5d3feb508c|Uploading file to temporary store at f3146e9b-e020-4a4f-ae1a-ba5d3feb508c/dcm/39e2cbc0-bf90-4209-bca3-e6a73a91634b.dcm. 2023-03-10T12:39:13.728493904Z 2023-03-10 12:39:13.7251|4006|INFO|Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService|File ID=1.2.826.0.1.2112370.55.1.13410264/1.3.12.2.1107.5.2.18.141028.2021051414181560370700792/1.3.12.2.1107.5.2.18.141028.2021051414183590186800860, CorrelationId=f3146e9b-e020-4a4f-ae1a-ba5d3feb508c|File uploaded to temporary store at f3146e9b-e020-4a4f-ae1a-ba5d3feb508c/dcm/39e2cbc0-bf90-4209-bca3-e6a73a91634b.dcm. 2023-03-10T12:39:13.737501884Z 2023-03-10 12:39:13.7342|4010|DEBUG|Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService|File ID=1.2.826.0.1.2112370.55.1.13410264/1.3.12.2.1107.5.2.18.141028.2021051414181560370700792/1.3.12.2.1107.5.2.18.141028.2021051414183590186800860, CorrelationId=f3146e9b-e020-4a4f-ae1a-ba5d3feb508c|File /incoming/f3146e9b-e020-4a4f-ae1a-ba5d3feb508c/dcm/39e2cbc0-bf90-4209-bca3-e6a73a91634b.dcm exists=True. 2023-03-10T12:39:13.737506022Z 2023-03-10 12:39:13.7342|4001|DEBUG|Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService|File ID=1.2.826.0.1.2112370.55.1.13410264/1.3.12.2.1107.5.2.18.141028.2021051414181560370700792/1.3.12.2.1107.5.2.18.141028.2021051414183590186800860, CorrelationId=f3146e9b-e020-4a4f-ae1a-ba5d3feb508c|Upload statistics: 2 threads, 0.0735303 seconds. 2023-03-10T12:39:13.737508998Z 2023-03-10 12:39:13.7342|4007|DEBUG|Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadQueue||Items in queue 34. 2023-03-10T12:39:13.737511382Z 2023-03-10 12:39:13.7343|4002|DEBUG|Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService|File ID=1.2.826.0.1.2112370.55.1.13410264/1.3.12.2.1107.5.2.18.141028.2021051414181560370700792/1.3.12.2.1107.5.2.18.141028.202105141418552196200910, CorrelationId=f3146e9b-e020-4a4f-ae1a-ba5d3feb508c|Uploading file to temporary store at f3146e9b-e020-4a4f-ae1a-ba5d3feb508c/dcm/151e97a7-3852-4ed0-90e0-a3692a5747eb.dcm.json. 2023-03-10T12:39:13.744672241Z 2023-03-10 12:39:13.7434|4006|INFO|Monai.Deploy.InformaticsGateway.Services.Storage.ObjectUploadService|File ID=1.2.826.0.1.2112370.55.1.13410264/1.3.12.2.1107.5.2.18.141028.2021051414181560370700792/1.3.12.2.1107.5.2.18.141028.202105141418552196200910, CorrelationId=f3146e9b-e020-4a4f-ae1a-ba5d3feb508c|File uploaded to temporary store at f3146e9b-e020-4a4f-ae1a-ba5d3feb508c/dcm/151e97a7-3852-4ed0-90e0-a3692a5747eb.dcm.json. 2023-03-10T12:39:13.753328306Z 2023-03-10 12:39:13.7500|20000|ERROR|Monai.Deploy.Storage.MinIO.MinIoStorageService|File ID=1.2.826.0.1.2112370.55.1.13410264/1.3.12.2.1107.5.2.18.141028.2021051414181560370700792/1.3.12.2.1107.5.2.18.141028.202105141418552196200910, CorrelationId=f3146e9b-e020-4a4f-ae1a-ba5d3feb508c|Error listing objects in bucket 'monaideploy' with error: MinIO API responded with message=A timeout exceeded while waiting to proceed with the request, please reduce your request rate 2023-03-10T12:39:13.901061627Z 2023-03-10 12:39:13.8988|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:13.901070904Z 2023-03-10 12:39:13.8988|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:14.901206769Z 2023-03-10 12:39:14.8988|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:14.901245643Z 2023-03-10 12:39:14.8989|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:15.900595300Z 2023-03-10 12:39:15.8982|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:15.900620979Z 2023-03-10 12:39:15.8983|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:16.900563277Z 2023-03-10 12:39:16.8993|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:16.900585530Z 2023-03-10 12:39:16.8993|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:17.900908057Z 2023-03-10 12:39:17.8975|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:17.900936681Z 2023-03-10 12:39:17.8976|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:18.901371130Z 2023-03-10 12:39:18.8979|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:18.901408651Z 2023-03-10 12:39:18.8980|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:19.901172232Z 2023-03-10 12:39:19.8986|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:19.901216575Z 2023-03-10 12:39:19.8987|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:20.900985586Z 2023-03-10 12:39:20.8984|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:20.901021564Z 2023-03-10 12:39:20.8984|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:21.901840170Z 2023-03-10 12:39:21.8997|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:21.901872040Z 2023-03-10 12:39:21.8998|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:22.900816122Z 2023-03-10 12:39:22.8974|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:22.900840367Z 2023-03-10 12:39:22.8975|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:23.901407376Z 2023-03-10 12:39:23.8979|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:23.901442122Z 2023-03-10 12:39:23.8980|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:24.901456117Z 2023-03-10 12:39:24.8983|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:24.901483939Z 2023-03-10 12:39:24.8984|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:25.901266516Z 2023-03-10 12:39:25.8988|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:25.901299107Z 2023-03-10 12:39:25.8989|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:26.901241105Z 2023-03-10 12:39:26.8999|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:26.901270130Z 2023-03-10 12:39:26.9000|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:27.900941338Z 2023-03-10 12:39:27.8975|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:27.900955304Z 2023-03-10 12:39:27.8976|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:28.900921718Z 2023-03-10 12:39:28.8975|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:28.900959800Z 2023-03-10 12:39:28.8976|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:29.901254313Z 2023-03-10 12:39:29.8978|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:29.901307093Z 2023-03-10 12:39:29.8979|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:30.921583825Z 2023-03-10 12:39:30.9003|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:30.921617188Z 2023-03-10 12:39:30.9004|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:32.133337130Z 2023-03-10 12:39:32.1309|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:32.133364631Z 2023-03-10 12:39:32.1310|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:33.133594493Z 2023-03-10 12:39:33.1309|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:33.133623789Z 2023-03-10 12:39:33.1310|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:34.133137367Z 2023-03-10 12:39:34.1306|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:34.133176010Z 2023-03-10 12:39:34.1308|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:35.136755174Z 2023-03-10 12:39:35.1322|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:35.136775673Z 2023-03-10 12:39:35.1324|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:36.137115252Z 2023-03-10 12:39:36.1343|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:36.137142905Z 2023-03-10 12:39:36.1344|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:37.137665310Z 2023-03-10 12:39:37.1352|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:37.137709193Z 2023-03-10 12:39:37.1353|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:38.136726362Z 2023-03-10 12:39:38.1352|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:38.136755137Z 2023-03-10 12:39:38.1354|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:39.137121998Z 2023-03-10 12:39:39.1337|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:39.137152646Z 2023-03-10 12:39:39.1338|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:40.136813963Z 2023-03-10 12:39:40.1355|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:40.136849440Z 2023-03-10 12:39:40.1355|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:41.137739561Z 2023-03-10 12:39:41.1339|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:41.137773164Z 2023-03-10 12:39:41.1340|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:42.136732144Z 2023-03-10 12:39:42.1334|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:42.136768953Z 2023-03-10 12:39:42.1334|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:43.136698217Z 2023-03-10 12:39:43.1352|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:43.136713015Z 2023-03-10 12:39:43.1353|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:44.137491554Z 2023-03-10 12:39:44.1349|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:44.137526500Z 2023-03-10 12:39:44.1350|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:45.137422892Z 2023-03-10 12:39:45.1361|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:45.137448681Z 2023-03-10 12:39:45.1362|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:46.136940708Z 2023-03-10 12:39:46.1353|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:46.136964202Z 2023-03-10 12:39:46.1353|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:47.228781638Z 2023-03-10 12:39:47.1345|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:47.228834959Z 2023-03-10 12:39:47.2275|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:48.228872537Z 2023-03-10 12:39:48.2272|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:48.228907102Z 2023-03-10 12:39:48.2273|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:49.228728052Z 2023-03-10 12:39:49.2274|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:49.228759401Z 2023-03-10 12:39:49.2275|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:50.229319217Z 2023-03-10 12:39:50.2270|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:50.229349734Z 2023-03-10 12:39:50.2271|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:51.228532985Z 2023-03-10 12:39:51.2272|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:51.228554806Z 2023-03-10 12:39:51.2273|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:52.229589443Z 2023-03-10 12:39:52.2261|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:52.229625160Z 2023-03-10 12:39:52.2262|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:53.228672982Z 2023-03-10 12:39:53.2263|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:53.228699081Z 2023-03-10 12:39:53.2264|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:54.228779635Z 2023-03-10 12:39:54.2265|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:54.228805614Z 2023-03-10 12:39:54.2265|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:55.228666142Z 2023-03-10 12:39:55.2263|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:55.228688555Z 2023-03-10 12:39:55.2264|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:56.229236682Z 2023-03-10 12:39:56.2268|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:56.229275255Z 2023-03-10 12:39:56.2269|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:57.229287460Z 2023-03-10 12:39:57.2270|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:57.229303420Z 2023-03-10 12:39:57.2271|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:58.228845787Z 2023-03-10 12:39:58.2255|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:58.228864662Z 2023-03-10 12:39:58.2255|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:39:59.229005159Z 2023-03-10 12:39:59.2267|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:39:59.229038512Z 2023-03-10 12:39:59.2268|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:00.229508162Z 2023-03-10 12:40:00.2249|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:00.229536545Z 2023-03-10 12:40:00.2250|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:01.239422349Z 2023-03-10 12:40:01.2260|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:01.239459850Z 2023-03-10 12:40:01.2261|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:02.229369932Z 2023-03-10 12:40:02.2278|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:02.229405720Z 2023-03-10 12:40:02.2279|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:03.229375895Z 2023-03-10 12:40:03.2278|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:03.229407395Z 2023-03-10 12:40:03.2279|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:04.229439687Z 2023-03-10 12:40:04.2260|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:04.229502276Z 2023-03-10 12:40:04.2261|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:05.228734096Z 2023-03-10 12:40:05.2264|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:05.228753713Z 2023-03-10 12:40:05.2264|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:06.229453888Z 2023-03-10 12:40:06.2255|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:06.229493553Z 2023-03-10 12:40:06.2257|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:07.229438119Z 2023-03-10 12:40:07.2270|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:07.229481521Z 2023-03-10 12:40:07.2271|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:08.228720575Z 2023-03-10 12:40:08.2258|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:08.228757164Z 2023-03-10 12:40:08.2259|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:09.228501363Z 2023-03-10 12:40:09.2262|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:09.228522011Z 2023-03-10 12:40:09.2262|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:10.228594861Z 2023-03-10 12:40:10.2273|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:10.228622243Z 2023-03-10 12:40:10.2274|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:11.229397319Z 2023-03-10 12:40:11.2268|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:11.229424851Z 2023-03-10 12:40:11.2269|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:12.228788400Z 2023-03-10 12:40:12.2254|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:12.228823676Z 2023-03-10 12:40:12.2255|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:13.229039025Z 2023-03-10 12:40:13.2254|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:13.229083599Z 2023-03-10 12:40:13.2255|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:14.228847555Z 2023-03-10 12:40:14.2263|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:14.228863895Z 2023-03-10 12:40:14.2264|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:15.229393408Z 2023-03-10 12:40:15.2270|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:15.229414879Z 2023-03-10 12:40:15.2271|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:16.228682456Z 2023-03-10 12:40:16.2272|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:16.228715809Z 2023-03-10 12:40:16.2273|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:17.229527264Z 2023-03-10 12:40:17.2271|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:17.229545008Z 2023-03-10 12:40:17.2272|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:18.229248379Z 2023-03-10 12:40:18.2267|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:18.229274368Z 2023-03-10 12:40:18.2268|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:19.229086004Z 2023-03-10 12:40:19.2257|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:19.229102295Z 2023-03-10 12:40:19.2258|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:20.229452978Z 2023-03-10 12:40:20.2260|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:20.229483696Z 2023-03-10 12:40:20.2261|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:21.229214760Z 2023-03-10 12:40:21.2268|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1. 2023-03-10T12:40:21.229269303Z 2023-03-10 12:40:21.2268|3005|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Checking elapsed time for bucket: 1.2.826.0.1.2112370.55.1.13410264. 2023-03-10T12:40:22.228637420Z 2023-03-10 12:40:22.2273|3004|TRACE|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadAssembler||Number of incomplete payloads waiting for processing: 1.
Other information
(Please attach any logs available and remember to anonymize any PHI before sharing).