Why Sidekiq “reliable_fetch’’ Is Not Consistently Being Reliable

Recently, in one of my projects, I came across a situation wherein I was required to parse a CSV uploaded by the end user. For performance reasons, I choose to create a new Sidekiq worker for each row in my CSV. All these rows (workers) were running in parallel (standard Sidekiq behaviour). I wanted to execute some logic ONLY after all rows (workers) had finished processing. Sidekiq Pro let’s you do this by creating a batch of workers. Sidekiq has a Batch API that provides success and failure callbacks for such scenarios.

Everything was working well in development and staging environment. Until a day when a customer reported that he is not receiving success/failure report for his jobs. I tried to reproduce the scenario described by the user but, couldn’t. Not even for the same customer account. But, then I was able to reproduce it and again I couldn’t.

Confused???…