Dominik Moritz comments

Results 1544 comments of


                                            Dominik Moritz

Add unit and/or integration tests

Or we check in a test whether the arrow/parquet has the right values and types without going back to csv in the test.

Reading from stdin doesn't work

Yeah, for now don't use stdin. Would love a pull request to fix this.

Reading from stdin doesn't work

Pretty much, yes. The seekable reader we use across the libraries is in a shared crate in https://github.com/domoritz/arrow-tools/tree/main/crates/arrow-tools. We can move any shared functionality into that crate.

Reading from stdin doesn't work

This seems to work now. ``` $ cat data/simple.csv | csv2arrow /dev/stdin -n Schema: { "fields": [ { "name": "a", "data_type": "Int64", "nullable": true, "dict_id": 0, "dict_is_ordered": false, "metadata": {}...

json2parquet hangs and gets killed when writing large files

Hmm, that sounds like a bug. We should not try to infer the schema when it's provided.

Reading from stdin doesn't work

I don't know. Could you do a git bisect to find out?

json2parquet hangs and gets killed when writing large files

Awesome. I think the main things are in the GitHub issues. Better support for stdin and stdout and tests are the main things. Installing via homebrew would require some CI...

json2parquet hangs and gets killed when writing large files

Can you try again to see whether these issues are still there?

Reading from stdin doesn't work

Thanks, that seems to fix it. ``` arrow-tools/crates/json2arrow on  main [!⇡1] via 🦀 v1.81.0 ❯ echo '{"a": 1, "b": 2}' | jq -c 'to_entries|.[]' | cargo run -- -n...

json2parquet hangs and gets killed when writing large files

I'll close this for now since we have https://github.com/domoritz/arrow-tools/issues/14 which I think covers what's left here.