node-gtfs icon indicating copy to clipboard operation
node-gtfs copied to clipboard

Edge deployment and SQLite

Open laem opened this issue 1 year ago • 4 comments

Hi, I'm wondering if this project could be connected to a product like turso.tech.

The DB would then stored outside of the server, enabling edge usages (serverless platforms do not have permanent filesystems).

laem avatar Feb 06 '24 21:02 laem

I'd like to add support for more databases other than sqlite3, perhaps using Drizzle or Prisma.

I have had success deploying to Vercel by including sqlite generated from node-gtfs as part of the repo in a read-only fashion (following the baked-data pattern https://simonwillison.net/2021/Jul/28/baked-data/ )

brendannee avatar Feb 07 '24 07:02 brendannee

Thanks ! So if I understand well, you have a server (or your local computer) where you're building your db, and then on Vercel you're downloading (through git or S3) a prepared version of your DB to be consumed without any writes.

In other words, no importGTFS on the edge platform ?

laem avatar Feb 07 '24 13:02 laem

You could also run this in a Github action which would trigger a redeploy and store it in the repo. I have been playing around with storing the db on a remix server with Simple Transit Site. I am persisting the db in s3 using a cron job and pulling it down in the Dockerfile and then running the Gtfs update.

I know that there are other options like Litestream, but it does not work with the DigitalOcean platform for some reason.

https://github.com/ioTransit/simple-transit-site/tree/main

AvidDabbler avatar Feb 07 '24 15:02 AvidDabbler

@laem You are correct - I build the db locally and then check it into the git repo along with the code for the project and then deploy (via git) to Vercel. So it is all bundled together. Whenever there is an update to the GTFS, I make a new commit to the repo.

That might not work well with very large agencies GTFS or for GTFS that changes very frequently.

brendannee avatar Feb 08 '24 05:02 brendannee