{"issue":{"title":"Multiple Workers = Lost jobs?","created_at":"2011-12-22T03:17:30Z","user":{"login":"RVN-BR","id":1218902,"url":"https://api.github.com/users/RVN-BR","gravatar_id":"e33af81d39f14364d23b04a622ce8594","avatar_url":"https://secure.gravatar.com/avatar/e33af81d39f14364d23b04a622ce8594?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png"},"milestone":null,"updated_at":"2011-12-23T19:17:02Z","state":"open","labels":[],"id":2632703,"pull_request":{"patch_url":null,"html_url":null,"diff_url":null},"closed_at":null,"html_url":"https://github.com/chrisboulton/php-resque/issues/32","body":"As per a comment on issue #30, I'm opening this issue in order to help gather more info and find a fix for the bug.\r\n\r\nI observed that when deploying multiple workers, some jobs dont get processed...\r\n\r\nBasically the test I did was:\r\n\r\nwatch output and number of jobs, launch 5 workers.\r\nEnqueue 10 jobs.  number of jobs in redis gets augmented to 10 (so 10 jobs queued), but only a few get really processed.  Nonetheless, they all get pulled out of the queue...\r\n\r\nSo basically it is \"loosing\" jobs.\r\n\r\nI will try and test more tomorrow, time permitting, and post back.\r\n\r\nChris said he had already noticed this, so if anyone else has noticed and/or fixed this, please let us know.\r\n\r\nImo this is crucial for any production environment, no?  Is anyone using phpresque in a production environment? Are you loosing jobs? Maybe you havent noticed?  Can you please check and revert?  \r\n\r\nthanks!","number":32,"url":"https://api.github.com/repos/chrisboulton/php-resque/issues/32","comments":1,"assignee":null},"comment":{"created_at":"2011-12-23T19:17:02Z","user":{"login":"andrewjshults","id":175454,"url":"https://api.github.com/users/andrewjshults","gravatar_id":"3e7813758deb7d6fa5cb8f3e9e6fce21","avatar_url":"https://secure.gravatar.com/avatar/3e7813758deb7d6fa5cb8f3e9e6fce21?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png"},"updated_at":"2011-12-23T19:17:02Z","id":3263823,"body":"We used php-resque in production at frid.ge and were running multiple worker and I can't remember running into a lost job issue. Our primary use was for sending notifications so it's quite possible that we did end up losing some jobs, but never as significant as you seem to be seeing (I'm forgetting exact numbers but iirc we were pushing about 1MM jobs/month). We shut everything down after our acquisition in July so, unfortunately, I can't check the logs any more.\r\n\r\nTechnically, resque is not a fully safe queue (according to Chris Wanstrath, the developer of the Ruby version - https://github.com/defunkt/resque/issues/93 ). It would be possible to use RPOPLPUSH (http://redis.io/commands/rpoplpush) to make it fully safe with some relatively external reworking (workers would need names/UUIDs to PUSH/POP into/from unique sets and there would need to be some interface to move failed jobs back into the main queue). In the short term, this could be a good way to investigate which worker is popping off the missing jobs (just PUSH everything into unique sets based on the worker and don't worry about cleaning up the processed sets).","url":"https://api.github.com/repos/chrisboulton/php-resque/issues/comments/3263823"},"action":"created"}