Why Sidekiq “reliable_fetch’’ Is Not Consistently Being Reliable

Rails development and Sidekiq go hand in hand. One or the other time you must have gone into problem where jobs go missing. Wondering why?
Sahil explains!!

Let's make learning fun!

Recently, in one of my projects, I came across a situation wherein I was required to parse a CSV uploaded by the end user. For performance reasons, I choose to create a new Sidekiq worker for each row in my CSV. All these rows (workers) were running in parallel (standard Sidekiq behaviour). I wanted to execute some logic ONLY after all rows (workers) had finished processing. Sidekiq Pro let’s you do this by creating a batch of workers. Sidekiq has a Batch API that provides success and failure callbacks for such scenarios.

Everything was working well in development and staging environment. Until a day when a customer reported that he is not receiving success/failure report for his jobs. I tried to reproduce the scenario described by the user but, couldn’t. Not even for the same customer account. But, then I was able to reproduce it and again I couldn’t.


View original post 529 more words

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.