Scheduling system for sending reminder emails before events

[Zeko369]

@davidals Had an idea on how we should add a notification system that would send an email for example a day before an event on #276, so I moved it to a separate issue here.

Has there been any decision/discussion on how the scheduling will be handled?

We could either have something like https://github.com/kelektiv/node-cron , or another service responsible only for querying the DB for "events happening tomorrow" and sending out the e-mails. Or some lightweight task scheduler using the database.

Originally posted by @davidals in https://github.com/freeCodeCamp/chapter/issues/276#issuecomment-594071997

[Zeko369]

@davidals great idea, I think that this should be part of the MVP.

But there are some problems with this, so first we said that for simplicity we'll host this on Heroku and if we use node-cron we'll essentially be running a blank Dyno which will cost a lot of money. Same goes for a more complicated job scheduler which we certainly don't need for the MVP. Since we're already heavily invested in the Heroku Ecosystem for the MVP (Heroku Postgres, Heroku Dynos for hosting, using their easy to setup ENV configs), we could simply use their built in scheduler (which is really limited about what we can change, giving us an ability to run it every 10minutes, 1hour or 1 day but we can set it to run daily at midnight for example, which is what we need) to start a one-off dyno to check if it needs to send any emails and do so. By doing this we'll greatly reduce the price of hosting on Heroku and make it simpler of us to setup.

I'd be really happy if you'd work on this with us, what do you think about this heroku-scheduled idea?

Since someone somewhere will probably try to run it on their own server, they can easily just create a "regular" cron job that executes our script at midnight. (we can provide a simple snippet for them)

Later we can defiantly add something like Bull, but that will require us to use redis, which is another dependency that I think we can live without for the MVP.

[davidals]

For an MVP, couldn't we do something simple using Node Timers? The general idea would be:

On Server startup:

• Schedule a notificaton task to run in X minutes the task starts and: -- After X minutes, schedule this same function to be ran again (do this first to be fail safe on the next steps) -- Then, find out all the scheduled tasks (we can store this in Postgres easily, so it would query postgres and process any pending tasks)
• Repeat after X minutes

I'm new to both Node and Heroku, based on that, do you think would work?

[Zeko369]

Node Timers? @davidals By this you mean setTimeout/Interval()?

This seems like implementing something that already exists, which we would need to maintain. Also if this were to be be run on the main server/process to save money on another dyno, a crash in this process would kill the whole server. I know that a simple try/catch will catch the error and not crash our server, just saying that if we put everything in the same container we're risking one thing taking down something eles.

Heroku implements a cron like scheduler, where you say when (10min/1h/1day) to run something, then it spins up a Dyno (their name for a docker container) and executes it there.

So we don't need to build the whole scheduler ourselves and just give heroku which command to run node task.js, we could set it to run every 10minutes if we wanted to (Although I think that even 1hour could be to frequent) and it will run the code in a separate container, so an error wouldn't kill our main server.

Doing this we could focus on the code itself rather than on implementing our own scheduler.

Also your implementation relies on setTimeout to schedule it later, we could simply implement that by doing a setInterval which would just run something every n milliseconds.

For testing we could write or own "cron" that would just do a setInterval and execute the actual task.js code

[davidals]

Gotcha. Yes, I meant the setTimeout/Interval. And yes, it depends on what we optimize for. Having a container only doing this scheduling would be ideal. Then it sounds like the Heroku scheduler is what we want.

I can then work on creating such a script that will do the checks for sending the e-mails and create an integration point for when we decide how to actually send the e-mail.

Makes sense?

[Zeko369]

Yeah, go ahead.

Also a couple of things to look out for

• Add a dontBugMe field to users (name it something else) so people don't get emails if they don't want them
• You should create it a service, for example "EmailParticipants" where you pass an email and an event so this can be reused to send an email as an organiser from the UI

And add a node script that we can run to simulate a scheduler calling the function

[allella]

Regarding @Zeko369 's dontBugMe field, we'd previously suggested a rsvps.subscribed boolean on https://github.com/freeCodeCamp/chapter/issues/107#issuecomment-562432174 and https://github.com/freeCodeCamp/chapter/issues/107#issuecomment-565544468

Also, while that's being implemented we might as well add the user_chapters.subscribed mentioned in the same comments from #107, which will allow an unsubscribe email link to stop all chapter emails, whereas the rsvps.subscribed would be specific to a notifications for a specific rsvp.

I don't think this was suggested, but the Wordpress-style scheduler technique of running tasks that hook onto the end of a visitor's website request is generally a recipe for disaster. So, having an independent cron/scheduler, as was being discussed, is the ideal way to go.

[allella]

@davidals I've added another point about the "subscribed" fields for rsvps and user_chapters on https://github.com/freeCodeCamp/chapter/issues/107#issuecomment-596905797

I put the notes on #107 to not blow up the scope of this issue, but I do think it's very relevant and related. Also, since this issue is dealing with reminders, I hope you might lead the way and add those two "subscribed" fields so we get them in the schema in a related PR.

[davidals]

Yes! I was already doing that.

I'll make sure to update accordingly.

[imcodingideas]

We could open up a webhook on our end, and use Github Actions Scheduled events i.e. https://docs.github.com/en/actions/learn-github-actions/events-that-trigger-workflows#scheduled-events

[allella]

Thanks for the suggestion on Github Actions.

Each Chapter instance will_ be self-hosted by various organizations and each administrator will likely need to configure their own scheduler, as with the Heroku conversation above.