postgres-migrations
postgres-migrations copied to clipboard
ThomWright
Allow JS migrations to be asynchronous
Hi,
Thanks for this, and sorry I didn't get back to you earlier.
Could you give me a quick overview of your use case for this?
Thanks!
I have a large csv of records that I want to import into the database. The file is read as an async stream to save memory and converted into SQL statements during the migration.
How large are we talking?
Does it get run as one migration? If so, streaming to save memory might not help if it gets turned into one big SQL string anyway.
Could you run this CSV > SQL transformation once, offline, and save the .sql file?
The reason I ask is I'm hesitant to add more features, and I worry about what adding async support would encourage. I'd like to keep this as simple as possible.
That's certainly a possibility! Though in that case I wonder why you have this feature at all, as surely all use cases could be solved this way? It strikes me that if you're going to allow the use of a js migration, you might as well allow it to be asynchronous, especially considering how small this change is, otherwise this particular feature just feels half-finished to me.
That's certainly a possibility! Though in that case I wonder why you have this feature at all, as surely all use cases could be solved this way?
Yes, I do slightly regret allowing JS migrations! I wouldn't encourage using it, and I think most use cases are better served with SQL files.
Every time the migrations are run, the generated SQL from previous migrations will be hashed and compared with the hashes in the database. If these don't match, then the migration will fail.
This means:
- the conversion from CSV to SQL will be run every time
- the CSV file should be treated as immutable
- the code that does the conversion must be repeatable, care needs to be taken with e.g. dependency changes
That said, if you're still sure you want async migrations, I'll merge this in.
I need to do a major version bump some time soon because of some recent hashing changes (only affecting JS migrations), so let me know and I'll get this included with that release.
Personally I don't think I'll need this feature again -- though if you decide not to accept this PR it might be worth deprecating the JS migrations with a warning and adding a note to the docs.
it might be worth deprecating the JS migrations
This feature is a major selling point for me, personally. Some things, like, for example, database-level auditing, can be extremely repetitive, and there are gains in having a generator with parameters, rather than doing the copy-pasting by hand.
On the other hand, though, can be done in SQL too, just clumsier.
Is there a chance for this change or something similar to this to get merged as well? Postgrator does something very similar allowing to call HTTP request during migrations for example
Is there a chance for this change or something similar to this to get merged as well? Postgrator does something very similar allowing to call HTTP request during migrations for example
The idea of that fills me with dread :sweat_smile:
Could you explain your use case for this feature?