CLI: import CSV file

[sehz]

$ fluvio producer --csv <csv_file>

This would producer files int the CSV format

[github-actions[bot]]

Stale issue message

[biryukovmaxim]

I think about taking this issue to implement. @sehz , could you help with requirements? Should it support binary records containing non-valid utf-8? What about header record? Should it be handled by special cli flags? additional parameter for delimiter to support something like tsv too?

[ajhunyady]

This issue was filled before https://github.com/infinyon/fluvio/issues/2988 was resolved by @crajcan. With producers supporting smartmodule transformations it would be better to implement this in a smartmodule.

This approach would allow the csv transformation to be used by producers, consumers, and connectors. Here is an example of a smartmodule that ingest from RSS and transforms into json: https://github.com/infinyon/labs-rss-json-sm

The new CLI would be:

fluvio produce csv-topic --smartmodule infinyon/[email protected]

The connector configuration:

apiVersion: 0.1.0
meta:
  version: 0.1.0
  name: csv-transform
  type: http-source
  topic: csv-topic
http:
  endpoint: "https://example.com/csv-endpoint"
  interval: 10s
transforms:
  - uses: infinyon/[email protected]

For reference, (rss-json running example):

• https://github.com/infinyon/labs-projects/blob/main/data-pipelines/hackernews-reader.md

[fraidev]

Hi!

I created a smartmodule for csv-to-json. If it makes sense, I would like to transfer the ownership to infinyon organization.

https://github.com/fraidev/csv-json-sm

[ajhunyady]

@fraidev thank you for your contribution. It seems like we have smartmodules repositories all over the place, and there was also an internal repository for a special project that performed a custom implementation - https://github.com/infinyon/labs-csv-json-sm. I made this repository public.

As your implementation is generic, replacing the existing solution with yours would make sense. Can you please issue a pull request against this repository and add an additional test case using the files provided by @AlexMikhalev?

[fraidev]

@fraidev thank you for your contribution. It seems like we have smartmodules repositories all over the place, and there was also an internal repository for a special project that performed a custom implementation - https://github.com/infinyon/labs-csv-json-sm. I made this repository public.

As your implementation is generic, replacing the existing solution with yours would make sense. Can you please issue a pull request against this repository and add an additional test case using the files provided by @AlexMikhalev?

Oh, I didn't know that already exist one smart module for this.

Great, I created a PR there. https://github.com/infinyon/labs-csv-json-sm/pull/2

[digikata]

Released as infinyon/csv-json smartmodule