[5.1] [5.2] Multiple queue workers using Redis

[m4tthumphrey]

Possible cause/result of #8576.

It seems that using several workers (php artisan queue:work --daemon ...) via supervisor cause the jobs to be picked up by multiple workers at the same time, causing all kinds of havoc.

I have 10 workers in my case, that download a file from an FTP server, save the file itself locally, inserts a row in a database and then delete the original file from the FTP site.

I've experienced several times now, that several rows appear in the database, and sometimes I get an error informing me that the file on the FTP site doesn't exist but the row appears in the database meaning that it has already been processed. I am also pummelled with the redis errors found in #8576 (Predis\Transaction\AbortedMultiExecException)

I should say that this doesn't happen all the time, only for some jobs...

FYI, I've temporarily reduced the number of workers to 1 and I haven't had a user error since, altho I do still get the Predis errors.

I've tried to reproduce this locally but cannot, it seems that it works as expected. I don't get the Predis error either. The only difference is that I'm using supervisor in production and 10 terminals on my Mac...

[raja]

+1

[raja]

@m4tthumphrey Any luck resolving this issue. I've seen similar behavior with running a large worker farm (+50 workers).

[GrahamCampbell]

Looks like a limitation with that driver. Perhaps use beanstalk - it's more suited to this use.

[m4tthumphrey]

@raja Yeh I definitely would not use Redis to power mission critical queues, at least not within Laravel. I switched to Beanstalkd and have had a 100% success rate. Also, the author of redis is working on a dedicated message queue system for Redis, Disque.

[trompx]

In case of multiple hosts, I would not advise to use beanstalkd as if one machine goes down, you would loose all your jobs on that machine.

Maybe implementing a lock algorithm in the RedisQueue.php driver in the pop function (maybe using blpop like here : http://www.justincarmony.com/blog/2012/01/10/php-workers-with-redis-solo/) or with something like https://github.com/ronnylt/redlock-php.

if I come up with something I'll let you know.

[reshadman]

I have experienced the problem with Redis, too. Still don't know why the issue has been closed.

If it's related to redis then it should be mentioned in the docs that multiple workers can not be run when the queue driver has been set to redis.

[henriquebim]

I had this problem, it happens for any job that runs for more than 60 seconds.

I have a thread in Laracasts that explain why this happen, how to avoid it and a lot of information about how Laravel handles Queues since I had to dig the source code a lot to understand how it works.

Check it for more information: https://laracasts.com/discuss/channels/laravel/running-multiple-queue-workers

[m4tthumphrey]

@henriquebim Amazing work man! I knew it was going to be todo with Laravel's implementation and not Redis itself. I think I did actually set the expire time to 0 or null and still encountered the issue.

I'm quite astounded that this has been closed though as it's definitely a bug within Laravel and at the very least should be mentioned in the documentation.

I'm still running with beanstalkd however with no problems whatsoever.

[henriquebim]

@m4tthumphrey if you set it to 0 you will make it worse, the code checks with is_null and is_null(0) = false. So with it set to 0, virtually all jobs will be able to duplicate (any job that takes more than 0 seconds), but this depends on the amount of jobs and workers running.

Setting it to null or to a really high number (so that no jobs run for more than that number in seconds) fixes the issue, BUT any job that fails will get stuck (won't go to failed_jobs nor run again) or will take that amount of seconds to be retried/put on failed_jobs.

For those that can use beanstalkd, I suggest doing so.

[phoenixgao]

Will this be fixed or .. will this issue be reopened?

[phoenixgao]

I agree with @GrahamCampbell , this is not a bug.

But it is defective.

I was also wondering why laravel was not using rpoplpush when I was working on port laravel redisqueue to symfony, since in redis documentation it was supposed to use for a reliable queue. http://redis.io/commands/rpoplpush#pattern-reliable-queue

This is not a bug, but I vote to reopen + 1

[taylorotwell]

Does this occur if you use another queue driver or is it particular to Redis?

[taylorotwell]

I would also be curious to know the values you are using for sleep and timeout on your queue worker configuration.