Skip to main content

Background Workers with ActiveJob and Sidekiq Discussion

General • Asked by Chris Oliver

Gravatar
varma on

Great video, I deployed my app with capistrano every thing went well except sidekiq, the jobs are not running. I tried debugging but sidekiq command is not working on my production system, could you please help me on this.

Thanks,
Srinivas


Gravatar
JP W (10 XP) on

I'd love to see a follow up on deploying to heroku!

Gravatar
Chris Oliver (169,610 XP) on

That's a really good idea! I'll do that. :)

Gravatar
David Becerra  on

I second that! :-)

Gravatar
Chris Oliver (169,610 XP) on

@disqus_Uh95M6FOYo:disqus @disqus_4UMLsxjaNb:disqus Enjoy! https://gorails.com/episode...


Gravatar
Gerardo Raiden (330 XP) on

Thanks Chris, Great vídeo. Quick comment: I think that Activejob is not passing 6 arguments as said (around minute 22) but the argument "6". That should be the user ID. Looking forward to the follow up on deploying to digitalocean!


Gravatar
John Viercinski (700 XP) on

Chris, I'm having an issue with getting this background email to send on Heroku. Posted a SO question about it. Please take look at it if you get a chance. Thanks! Learning a ton from you. http://stackoverflow.com/qu...

Gravatar
Chris Oliver (169,610 XP) on

Did you get this working? You'll need to also make sure your background job runner is a second command added to your Procfile. For example, you'd add sidekiq to your Procfile as a background process and that will make sure these deliver_later emails get sent on Heroku.

Gravatar
John Viercinski (700 XP) on

I'm still working to correct this problem. I'm able to get deliver_later working in development by running redis-server in one terminal, and sidekiq -q mailers in another, but I keep running into "ERROR: ERR max number of clients reached" issues on heroku. It's an absolute beast to configure redis on heroku for a beginner like myself. I'm trying redis to go now and I'm still running into problems.

Gravatar
Chris Oliver (169,610 XP) on

You may just need to bump down your concurrency number for Sidekiq so it's not using so many connections or upgrade your Redis instance so you have one with more connections. Most of the Heroku things like this have some pretty low limitations on concurrency on the lower end.

Gravatar
John Viercinski (700 XP) on

I've actually done both of those things. Lowered concurrency settings in sidekiq.yml and increased redis to go plan to micro and I still run into this issue. Is there any way to see how many clients are attempting to load so I can see what concurrency number I need? This is so ridiculous.

Gravatar
Chris Oliver (169,610 XP) on

Have you read this thread? https://github.com/mperham/...

Gravatar
John Viercinski (700 XP) on

Got it working. Not 100% clear how. I followed the instructions here: (http://maso.logdown.com/pos... and mixed it with the sidekiq.rb initializer from the link you recommended. Curious as to how this will impact performance when I launch the app. That's a question for another day though.

Gravatar
John Viercinski (700 XP) on

Thanks for your help!


Gravatar
Aime Ngongang (440 XP) on

@Chris Oliver I followed this episode and it worked perfectly. One thing though I could not find good resource is how to update a job running time. In my case I have a reminder job, that send email to users 24 hours prior to the event. Now lets say the event is cancel/edited how do we update sidekiq to either modified the time or not run the job at all, can you please give me hint?

Thanks.

Gravatar
Aime Ngongang (440 XP) on

@excid3:disqus can you please address this when you have a chance?

Gravatar
Chris Oliver (169,610 XP) on

Hey Aime,

I'd probably recommend making a cron job for this case instead of scheduling jobs ahead of time. This way you can write a worker that runs every day or so, and checks for events that happen in the next 24 hours and sends out those email reminders.

By doing this you don't have to keep track of which events were canceled or rescheduled, and you're always looking at accurate data when the job runs so you don't have to worry about any of the cancellations or reschedules.

I made an episode on the whenever gem that you can use to build cron jobs that you might like to watch: https://gorails.com/episode...


Gravatar
SaschaM on

Is there a more "beginner-like" version of background workers available? I'm particularly interested in automating e-mails, csv downloads + notifying the user per web or per mail. There really is nothing fundamental available online.
Thanks!

Gravatar
Chris Oliver (169,610 XP) on

Since ActiveJob is just a wrapper for whatever background processor you use, all you have to do is write your code to use ActiveJob. All your emails already support ActiveJob, so to send them in the background you can just say "UserMailer.notification(user).deliver_later" and the deliver later part will create a job for it.

As for everything else like notifications and csv downloads, you'd just write your code in a job and have your controllers start the job. Nothing too fancy.

Is there anything specific I could cover for you that would help wrap your head around it?


Gravatar
Taylor Cooney (170 XP) on
Great video Chris

Gravatar
gafemoyano (220 XP) on
Hey Chris, do you think it's fine for an application to use the same Redis instance for Sidekiq and as cache store? I've seen a couple of Mike Perham's comments against it from a couple years back, but it just seems like a much simpler setup to only have Redis instead of Redis + Memcached.
Are there any special considerations/configs to take into account in your experience if you plan on using Redis for both?

Thanks!

Gravatar
Chris Oliver (169,610 XP) on
I've been using Redis for Rails cache + Sidekiq for years now. Absolutely no problems. There's probably some rare situation that could cause trouble, but I prefer the simplicity of managing less services in production and it's worked just fine.
Gravatar
gafemoyano (220 XP) on
Thanks for the reply Chris!

Login or create an account to join the conversation.