Maybe this is an opportunity to create a new kind of Show/Ask HN, called “Who’s got questions?”
For example. I’m a self-described senior electron/react/iOS app dev and I have created exactly one API in my life using node/express/GraphQL/mongo/redis on heroku. It works great! I had a ton of fun learning and deploying and now maintaining.
Now... I want to add some features such as long-running (30s-10m) batch jobs. How do I go about doing this? I noticed Redis has Bull ... but I’m still not clear on the architecture. I feel a bit stuck and nervous I’m going to pick something that’s going to fall over at the slightest traffic.
If you're talking about having periodic jobs to run you can use heroku's scheduler plugin and start a job.
It will use a different dyno from the one running your api.
You will end up paying more, but you won't have problems with resources sharing.
In a traditional VPS environment you would have needed to have a machine beefy enough to run both api and jobs or spin up a new machine.
Cron, Jenkins or another task runner could have achieved the same.
If you're talking about having long running requests initiated by users, a common pattern is to have a queue:
- User send a request
- API add request to a queue
- n workers wait for a queue message and process it once it arrives
Another simpler approach is to just spin another instance of the API and just forward long running requests to the new instance. Depending on the type of job, its dependencies, its frequency you may run into some problems that you wouldn't have with the queue based approach
Use a queue. For example, a redis list. Then you can have scalable workers lpop to process the jobs and save the results somewhere. Would require 2 endpoints, one to schedule the job and one to fetch the results. You can generate a uuid to id each job.
Or you can use a managed queue service like SQS as well.
For example. I’m a self-described senior electron/react/iOS app dev and I have created exactly one API in my life using node/express/GraphQL/mongo/redis on heroku. It works great! I had a ton of fun learning and deploying and now maintaining.
Now... I want to add some features such as long-running (30s-10m) batch jobs. How do I go about doing this? I noticed Redis has Bull ... but I’m still not clear on the architecture. I feel a bit stuck and nervous I’m going to pick something that’s going to fall over at the slightest traffic.