q
q copied to clipboard
harelba
"Too many columns" error
Hi,
Is there a limit to the number of columns. THis file had columns uptil "N"
$ q "select * from .\Extractor\xxx.csv" WARNING: file already exists but should not: C:\Users\pstep\AppData\Local\Temp_MEI10122\include\pyconfig.h query error: too many columns on temp_table_10001
Getting the same error with 24 columns. Works for me, with this file, with 23 columns but not 24.
I don't run into the issue when I use the right command-line options. In my case, -H -d, to specify that it's CSV and has a header row
Hi, I've released a new version for windows 1.7.4 which solves this extra warning issue. Just download the new windows version from the site (remember to refresh the page properly, so the new link will be used) and install it. The setup version should be setup-1.7.4.exe.
Hi,
It seems that the limit issue is not completely solved. Indeed, try this URL https://pmb-bordeaux.fr/getdata/query/Atacama/(samples,lcms_esi_pos)?format=xml&debug=1 The lcms_esi_pos file has 1976 columns and the query works well. Now, try this URL https://pmb-bordeaux.fr/getdata/query/Atacama/(samples,lcms_esi_neg)?format=xml&debug=1 The lcms_esi_pos has 2564 columns and we obtain an error: "query error: too many columns on temp_table_10002"
Is there something to tune ? if yes, at what level ? Thanks in advance for your reply
After some tests, it seems that there is a limit in the number of columns not to be exceeded i.e. 2000 in total i.e. with one or more files. Thus, the 'samples' file having 12 columns, a query combining it with the lcms_esi_neg file, forces to limit the number of columns to 1989 (1989+12 = 2000 + 1 for linking). Whereas the lcms_esi_neg file alone allows to go up to 2000 columns.
In fact the limitation in the number of columns is set by sqlite3: https://www.dbtalks.com/tutorials/learn-sqlite/what-are-the-limitations-of-sqlite.
By installing sqlite3 from source and changing the value of SQLITE_MAX_COLUMN in the file src/sqliteLimit.h (i.e. 32767), then queries on tables beyond 2000 columns work now very well. :-)