firebird icon indicating copy to clipboard operation
firebird copied to clipboard

Add CSV support for external tables

Open krilbe opened this issue 3 years ago • 5 comments

I propose to extend the external table feature to support CSV format. The existing binary format for external tables make them mostly useless for import and export of data except in really simple cases, and even for those you need to specially prepare the data, e.g. add padding.

I would suggest to take a look at the CSV format options available in MariaDB/MySQL load data infile/select into outfile, and implement all/most of them: choice of character encoding, field and row delimiters, quoting, null markers etc.

I realize that external tables in CSV format will be limited to appending records (insert) and selecting (all?) rows, but these limitations already apply to external tables in the existing binary format. The select abilities might need to be limited even further for CSV format, e.g. maybe disallow where clauses. But since the intended use is for import and export of data, I can't see that as a problem.

CSV-format external tables should either disallow non-(var)character types or provide ability to specify in the table definition how to convert each non-char column to/from string. Possibly, CSV format external tables could be specified as unidirectional, meaning only allow insert (export) or only allow select (import). That would allow to only specify either "to string" or "from string" (not both) in most cases.

krilbe avatar Jan 22 '22 08:01 krilbe

See also issue #2995 . I think adding CSV format support to external tables would be a better solution, but adding load data infile and select into outfile (with same or different keywords and syntax) would also be a workable solution.

krilbe avatar Jan 22 '22 08:01 krilbe

Read of CSV file may be easily done with external procedure in current days.

asfernandes avatar Jan 22 '22 13:01 asfernandes

There's a huge difference in effort between "import/export CSV using built-in SQL features" and "authour your own CSV implementation as a UDR to import/export CSV". If such a UDR were available, then fair enough, but otherwise...seriously?

krilbe avatar Jan 22 '22 13:01 krilbe

Not quite UDR, but you can import CVS using ODBC provider and MS Text File ODBC driver.

aafemt avatar Jan 22 '22 13:01 aafemt

Still a workaround that adds work, both in terms of figuring out how to do it and in terms of actually doing it.

krilbe avatar Jan 22 '22 14:01 krilbe