Allow duplicate jobs to be discarded

[mhenrixon]

Should resolve #176

I want to prevent the following:

These jobs can under no circumstances end up in this situation, discarding the new jobs instead of queuing them ensures this won't happen.

I took the simplest possible approach using terminology from sidekiq-unique-jobs. All feedback appreciated.

[rosa]

Hey @mhenrixon, thanks for this! There was already a PR for it: #523
But this idea of marking the job as finished is really clever! It'd allow for quite a few simplifications 🤔 I wonder if it's too clever, though.... would it lead to confusing scenarios where you don't know what happened to a job? Although I guess the same is true if the job is simply discarded, so this should be ok!

I need to review what happens here for the bulk enqueue case, which was the tricky part of that other PR.

[mhenrixon]

Hey @mhenrixon, thanks for this! There was already a PR for it: #523

I need to learn to read all the issues. Sorry about the extra PR.

But this idea of marking the job as finished is really clever! It'd allow for quite a few simplifications 🤔 I wonder if it's too clever, though.... would it lead to confusing scenarios where you don't know what happened to a job? Although I guess the same is true if the job is simply discarded, so this should be ok!

After working on sidekiq-unique-jobs, and with sidekiq. I have learned that some visibility is appreciated. One of the reasons to use a database-backed queue system is to maintain visibility and avoid paying a blackmailer for keeping the data safe.

I am happy to close this if you prefer the other one @rosa. I didn't look into the other PR in detail, but the bulk enqueue sounds important.

[rosa]

Oh, no worries! I think the idea here is better than what we were trying to do in the other PR, so I think combining these two would be the best! I need to review the other one again after some changes so I'll review these two together and will think how to proceed, maybe transplanting some commits over 🤔

[mhenrixon]

Feel free to add commits or let me know what to pick or how I can help land this in the main branch. I am at your service!

[elias19r]

would it lead to confusing scenarios where you don't know what happened to a job? Although I guess the same is true if the job is simply discarded, so this should be ok!

I just read this and thought: would it make sense, or be worth it, to add some kind of metadata field to the job to indicate that it has been discarded?

[mhenrixon]

I just read this and thought: would it make sense, or be worth it, to add some kind of metadata field to the job to indicate that it has been discarded?

I thought about that but opted for the simplicity while maintaining visibility.

I am open to the idea though.