csvs-to-sqlite
csvs-to-sqlite copied to clipboard
simonw
Convert CSV files into a SQLite database
Pandas read_csv throws an exception when encountering a line that seems to have too many fields, but it can be made to skip these bad lines and then report them...
This would allow me to add `datasette publish now --csv=foo.csv --csv=bar.csv --csv-db-name=csvs` to https://github.com/simonw/datasette
```console > echo "a,b" > [minimal_example].csv > csvs-to-sqlite "[minimal_example].csv" out.db /opt/homebrew/lib/python3.10/site-packages/csvs_to_sqlite/utils.py:38: FutureWarning: The error_bad_lines argument has been deprecated and will be removed in a future version. Use on_bad_lines in the...
Currently it creates a table for each file, but there are many use cases where files with same structure are saved in a particular directory. It would be a great...
Using the following command: ``` csvs-to-sqlite -dt timestamp rm_rnb_history_pres.csv qsl_locs.csv rm_toucans_qsl. db ``` results in the error text ``` File "/usr/local/python/3.10.8/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3800, in get_loc return self._engine.get_loc(casted_key) File "pandas/_libs/index.pyx", line...
With new pandas version 2.0.0, the argument and method were removed, retrieving an error. Reference: https://pandas.pydata.org/docs/whatsnew/v2.0.0.html#removal-of-prior-version-deprecations-changes closes https://github.com/simonw/csvs-to-sqlite/issues/88 closes https://github.com/simonw/csvs-to-sqlite/issues/93
Executing with recent version of pandas (2.0.0), released on April 3, 2023, returns this error: ``` Loaded 1 dataframes Traceback (most recent call last): File "/usr/local/bin/csvs-to-sqlite", line 8, in sys.exit(cli())...
If I run something like this `csvs-to-sqlite some-data.csv database.db` How do I make the script skip this command if table 'some-data' already exists?
