Update - REST API clarification

[amyshwang]

Describe the requested changes: Reported by @javier:

Hi. Yesterday I had to do some bulk importing. 6 tables with several timestamp columns, all of them with the same format, but not a default format on questdb. Following the docs was more confusing that it should have been. We have a section for bulk CSV imports, where we show some detail about working with timestamps, but not about some other basic params, like overriding the table name. Also, when talking about timestamps, we mention we can just put the text_loader.json at the conf directory and everything will work, but that is not the case. You need to place the file, and also to uncomment the config option with the file name and restart. Otherwise, even if using the default text_loader.json name, you will still get errors. The text_loader.json file in itself is a bag of surprises (as we can see on github issues also) and depending how you installed questdb it will be found or not. I tried placing it outside of the config dir and couldn’t make it work

Location: Importing tables via REST API

[javier]

Two issues here: timestamp format and table name

timestamp format

With the new proposed changes at https://github.com/questdb/questdb.io/pull/967, docs on importing look way better. But I believe there are still two things we can improve.

At several points we mention the option to use the text_loader.json file for formatting. The docs would imply that if you drop the file at the config folder it will magically work, but that is not the case. Apart from creating the file you need to set the configuration property cairo.sql.copy.formats.file=text_loader.json. Once you set that (and restart), the REST API and the COPY command both will honour the formats

table name

At the bulk import guide we show how to deal with schemas or data types, but we don't show how to define the target name for the table. Without that property, the table will have an ugly name with the same name as the file, including the .csv extension. I would add a section explaining we can override the name for the table. For the advanced properties I see the current link to the API docs sufficient, but the table name in my opinion is a basic use case

[amyshwang]

timestamp format:

This is impacting both CSV import guide and REST API guide. Current section locates in REST API guide and Configuration page.

We need to find a place where both guides and potentially refer to?

[amyshwang]

Where to present info/settings shared by both COPY and REST API import

Currently we have two import guides:

There are two pieces of information applying to both import guides:

1. text_loader.json setting: currently in Small CSV import guide https://questdb.io/docs/guides/importing-data-rest#text-loader-configuration
2. Supported encoding: UTF-8 example code to change encoding: iconv -f CP1251 -t UTF-8 -o test-utf.csv test.csv

Potential solutions

1. Adding info to each guide

• Drawback: repetition, difficult to keep in sync
• Advantage: The sidebar has two guides on second level (same as status quo) so more visibility

2. Nesting the two guides:

- Guide
    - CSV import
        - Import overview ( `text_loader.json` and encoding info go here)
        - Large CSV import (COPY)
        - Small CSV import (REST API)

• Drawback: more levels and less visibility
• Advantage: Shared info is presented in the overview page --> it is clear

[pswu11]

I'd say option 2. But if we go for two, maybe we should shuffle the content a bit (introductory sections on both guides etc)? Otherwise, the overview page doesn't really feel like an overview.

[amyshwang]

Elaborate on option 2 above:

- Guide
    - CSV import
        - Import overview ( `text_loader.json` and encoding info go here)
        - Large CSV import (COPY)
        - Small CSV import (REST API)

The Import overview page should have:

• High level summary of the two options, extracted from both guides
• Import configuration
  • Timestamp format
  • Encoding

Each import guide can shorten its introductory paragraph as a result.

[jerrinot]

+1 for #2. Otherwise as a user I have to guess where to click. "large vs small" is not a great differentiator as these terms are very relative.