Try to debug s3 queue

[mariusandra]

Changes

• Already lost hours on this. If anyone has insights, please share.
• After setting up credentials to enable S3 queues in production, when testing locally, I noticed not that many events got queued.
• Digging deeper, it seems s3.upload and s3.putObject just... freeze. I'm sure I'm doing something wrong, but I have no idea. (putObject is for smaller files, upload contains chunking logic, both should be interchangeable)

The problem is that all test uploads/putObjects inside testS3Connection succeed:

[MAIN] 12:17:25 # s3.putObject {
  Bucket: 'posthog-cloud-plugin-server-queue',
  Key: 'jobs/CONNTEST/0179b27a-8869-0000-b767-1e78ac286247.test',
  Body: <Buffer 74 65 73 74>
}
[MAIN] 12:17:25 # ---validate
[MAIN] 12:17:25 # ---build
[MAIN] 12:17:25 # ---sign
[MAIN] 12:17:26 # ---httpUploadProgress
[MAIN] 12:17:26 # ---httpHeaders
[MAIN] 12:17:26 # ---httpDone
[MAIN] 12:17:26 # ---send
[MAIN] 12:17:26 # ---extractData
[MAIN] 12:17:26 # ---success
[MAIN] 12:17:26 # ---complete
[MAIN] 12:17:26 # s3.putObject DONE

but even the same code ran again in enqueue fails:

[___2] 12:17:30 🍿 Fetched all actions from DB anew
[___1] 12:17:30 # s3.putObject {
  Bucket: 'posthog-cloud-plugin-server-queue',
  Key: 'jobs/CONNTEST/0179b27a-9c2e-0000-91f6-0f4a47a642c4.test',
  Body: <Buffer 74 65 73 74>
}
[___1] 12:17:30 # ---validate
[___1] 12:17:30 # ---build
[___1] 12:17:30 # ---sign
[___3] 12:17:30 🧵 Starting Piscina worker thread 3…
[___3] 12:17:30 ✅ Connected to Redis!

The only lead on the internet was this, but here the user solved it by removing some mocking code. For me this happens even if I run the compiled dist code.

Any ideas anyone? Leaving this up to take a break.

Checklist

• Updated Settings section in README.md, if settings are affected
• Jest tests

[guidoiaquinti]

Hey @mariusandra, is this PR still relevant/in progress? Any chance we can push it to the finish line or close it?

I'm asking in preparation of the repo move. Thank you! 🙇

[mariusandra]

I think this can be closed. The problem was the S3 queue was running async in the background, yet we had something blocking the thread if no promises were explicitly tracked. More context here: #487 and here: PostHog/piscina#3

The S3 queue itself would be a worst-case slow backup dump for in-flight jobs and tasks in case we lose access to the main job queue. Someone should still test and enable it for production: PostHog/posthog#4537

I'm just so far removed from this work now that it's a big context shift, so I never got around to finishing it. 🙃