pandavro
pandavro copied to clipboard
fix read_avro kwargs + cleanup tests
This PR solves two issues:
- In a previous PR, the
columns
kwarg offrom_records
was promoted to read_avro to save RAM. However, this meant that all rows were read, ignoringnrows
. Same forexclude
to exclude columns. Here this is fixed - Tests were flaky and not working for me locally. This was due to an unspecified timezone, which makes avro dump as UTC but then when reading it infers what it can. I hypothesize this might be working since the tests are run in UTC located machines, but I hope this will work now for any place.
Let me know your thoughts
@ynqa and @ruben-trdj for exposure
FWIW, I can confirm from local testing (Ubuntu 20.04) that:
- tests on
yqna:master
fail with my system locale set to local time and pass when I set my locale to UTC - tests on this branch pass regardless of system locale.