How to handle long-running external services?

My Rails app calls a long-running external service which can take over a minute to return a response. I'm calling the service from a Sidekiq job but I don't want to tie up my worker for over a minute on every job waiting for the response.

I looked into a 'fire and forget' approach - call the service with the params then don't wait for a response, just move on. If I did that I'd end up with more jobs and code to check if the external job was completed. Not ideal.

I can have the external service call a webhook endpoint in my app with the result but I can't make the external service respond to the original call any earlier.

What are some good approaches to handling long-running external services?

Some ideas to consider. Can you:

• use cron (or scheduler) on Heroku to kick off the job regularly so the long running service is not taking so long?
• break the long-running service up into smaller pieces?
• dedicate a worker just for this long-running service so you don't mind it being tied up?
• configure the external service so it pings you when the job is done?

Hey Ivan, thanks for your ideas. Good stuff.

Unfortunately scheduling won't really help here because the long-running job is user-initiated. Nor will breaking it up into smaller pieces since it's a single FFMPEG video creation process.

A dedicated worker that just handles these requests would help but I'll still be limited to one job per minute. I guess multiple dedicated workers would multiply that but it's early stage and I want to keep costs down. Running the whole app on a $15/mo Digital Ocean server right now.

Having the external service ping my app when the job is done was my original approach. I always wanted to porcess to be asynchronous and started by passing in a webhook URL for the job to call when finished but...

Calling the job via HTTP is the issue. The basis of HTTP requests is that they wait for a response and this gets to the crux of my issue - the job is a serverless function running on Zeit Now platform. These functions stop processing as soon as they send an HTTP response. So I can't make it respond right away with some kind of 'ACCEPTED' message because then the rest of the function won't run.

One hacky option is to set a short timeout on the HTTP request in my Rails app. Then just accept the error and move on. That way the external job doesn't send any response at all. It just calls the webhook URL in app with the result. Super hacky!