sqlite-utils icon indicating copy to clipboard operation
sqlite-utils copied to clipboard

Add an option for specifying column names when inserting CSV data

Open hubgit opened this issue 3 years ago • 3 comments

https://sqlite-utils.datasette.io/en/stable/cli.html#csv-files-without-a-header-row

The first row of any CSV or TSV file is expected to contain the names of the columns in that file.

If your file does not include this row, you can use the --no-headers option to specify that the tool should not use that fist row as headers.

If you do this, the table will be created with column names called untitled_1 and untitled_2 and so on. You can then rename them using the sqlite-utils transform ... --rename command.

It would be nice to be able to specify the column names when importing CSV/TSV without a header row, via an extra command line option.

(renaming a column of a large table can take a long time, which makes it an inconvenient workaround)

hubgit avatar Aug 27 '22 15:08 hubgit

A couple of tricks I use here. Firstly, I often create the table before the import using the sqlite-utils create-table command: https://sqlite-utils.datasette.io/en/stable/cli.html#creating-tables

The other current option is to use the bulk command, which lets you construct a custom SQL query to execute against every row from a CSV file: https://sqlite-utils.datasette.io/en/stable/cli.html#executing-sql-in-bulk

Do either of those options work here or is there a useful new feature that would work better?

simonw avatar Aug 27 '22 20:08 simonw

Creating the table before inserting is a useful workaround, thanks. It does require figuring out the create table syntax and listing all the fields manually, though, which loses some of the magic of sqlite-utils.

I was expecting to find an option like --headers=foo,bar (or --header-row='foo\tbar', if that would be easier) - not necessarily that exact syntax, but something that would essentially be treated the same as having a header row in the file.

hubgit avatar Aug 28 '22 12:08 hubgit

Yeah, having a way of inserting a CSV file but saying "only I want the title column to come first" does make sense to me as a feature.

It can use -o for consistency with https://sqlite-utils.datasette.io/en/stable/cli-reference.html#search and https://sqlite-utils.datasette.io/en/stable/cli-reference.html#rows and https://sqlite-utils.datasette.io/en/stable/cli-reference.html#transform

simonw avatar Aug 31 '22 03:08 simonw