transform on import?

[Analect]

Excuse the basic question. I can see you have transformSource for exports. Is there any way to use your import tools to 'transform' and input, as it gets processed? My use case is where I have a collection in MongoDB, where I want to push across one of the fielsd as a parent_id, and leverage your interface to the bulk API. Thanks.

[skratchdot]

That's a good question, and it's probably the place I should have allowed for the transforming of documents, but currently, the transform logic is only in the export. It should be pretty easy to move similar transform logic from here: https://github.com/skratchdot/elasticsearch-tools/blob/master/lib/es-export-bulk.js#L132-L146 to here: https://github.com/skratchdot/elasticsearch-tools/blob/master/lib/es-import-bulk.js#L89 (I'd just need to have the meta and source transform logic happen on the correct odd vs even lines).

There's a better way to do some of this stuff, but the project was thrown together quickly for a very limited use case, and at the time, it made sense (for the project I was working on) to transform at the export stage (so I could spot-check the files before importing).

Depending on how big your data set is, you could import all your data. Then export it using a transform. The reimport it.

There are also some command line json parsing/transforming tools available that you might want to look into (like jq and underscore-cli).