improve support for import from files

[ingenthr]

Currently, there is a recipe for importing json files in a particular format. That works well, but unfortunately the format isn't always what one would like it to be. Maybe this can be mixed in with something like jq?

One thing I tried, for example, was unzipping the travel-sample database and try to reimport it. The format didn't quite match. I tried a few things listing all of the .json files and piping to reformat with builtins, but it seemed to be trying to do all of this in memory, so I killed it.

Scenario: Given a set of files in a directory that are in JSON format, as a user I want to be able to import them. If they are not in the right format, I want to be able to massage them easily into the right format. For example, the file is likely what I really want to go into the 'content' field that kv-upsert expects, but I may want to extract a field from that to be the ID. Or, I may want the ID to be a concatination of two or more fields. Or, I may want the ID to be a sequence, possibly concatinated on a field in the file.

This may be do-able with nushell built ins, but it is complicated enough that it wasn't obvious in a few minutes.

One possible implementation: allow for a file that defines reformatting rules that are passed in to kv-upsert. Or introduce another command like kv-import.

Another solution which we could play with is jq streaming edits, which seems to be a thing, but not something I'm familiar with.

[ingenthr]

Caputuring what we've talked about when thinking about this idea.

ls *.json | { open $it.name } | map --where type = "airport" --where name icao { id: .name, content: . } | kv insert -n ^^^ this would open all of the json files, match where they are airports have a name and an icao, turn it into another record with the name as the id and the content in the content.

[ingenthr]

also, think about xattrs

[ingenthr]

And from @blbryant 's post, also consider the case where you need to add a "type" attribute to a document.