Simon Willison

Support for compound (composite) foreign keys

3

It turns out SQLite supports composite foreign keys: https://www.sqlite.org/foreignkeys.html#fk_composite Their example looks like this: ```sql CREATE TABLE album( albumartist TEXT, albumname TEXT, albumcover BINARY, PRIMARY KEY(albumartist, albumname) ); CREATE TABLE...

Handle giant CSV files

4

It would be great if `csvs-to-sqlite` could quite happily churn through a multiple GB CSV file without using up all available RAM on a machine. https://plot.ly/python/big-data-analytics-with-pandas-and-sqlite/ has some good patterns...

csvs-to-sqlite 2.0: dropping Pandas in favour of sqlite-utils

6

My [sqlite-utils](https://github.com/simonw/sqlite-utils) library has evolved to the point where I think it would make a good foundation for the next version of `csvs-to-sqlite`. The main feature I'm excited about here...

`--split` for splitting extracted columns on their values

3

https://data.sfgov.org/Economy-and-Community/Mobile-Food-Facility-Permit/rqzj-sfat Say for example there's a `FoodItems` column in the CSV that has data in it like this: `Cold Truck: Cheeseburgers: Burgers: Chicken Bake: Chili Dogs: Hot Dogs: Corn Dogs:...

-c should be able to merge multiple columns into a single table

4

Consider this for example: https://san-francisco.datasettes.com/registered-business-locations-3d50679/Registered_Business_Locations_-_San_Francisco "NAICS Code" and " NAICS Code Description" are two columns that always reflect each other.

Progress bars would be useful

2

I'm processing a 100MB+ file at the moment and there's no indication of how long it will take.

Quite a few options (like `--shape`) are only listed in the `--help` output. This isn't enough information - the README should be expanded.

This should work: curl url-to-csv | csvs-to-sqlite - data.db

This tool has enough options now that it warrants more documentation than just the README and `--help`

Figure out a mechanism for interpreting dates as always in the 1900s

4

I created https://csvs-to-sqlite-date-demo.now.sh/antiquities-317d506/actions.under.antiquities.act like this: $ csvs-to-sqlite actions.under.antiquities.act.csv antiquities.db -d date Using the CSV from here: https://github.com/fivethirtyeight/data/blob/master/antiquities-act/actions_under_antiquities_act.csv Just one problem:  It would be nice if...