pandavro icon indicating copy to clipboard operation
pandavro copied to clipboard

fix read_avro kwargs + cleanup tests

Open marctorsoc opened this issue 1 year ago • 1 comments

This PR solves two issues:

  1. In a previous PR, the columns kwarg of from_records was promoted to read_avro to save RAM. However, this meant that all rows were read, ignoring nrows. Same for exclude to exclude columns. Here this is fixed
  2. Tests were flaky and not working for me locally. This was due to an unspecified timezone, which makes avro dump as UTC but then when reading it infers what it can. I hypothesize this might be working since the tests are run in UTC located machines, but I hope this will work now for any place.

Let me know your thoughts

marctorsoc avatar Oct 15 '22 09:10 marctorsoc

@ynqa and @ruben-trdj for exposure

marctorsoc avatar Oct 15 '22 09:10 marctorsoc

FWIW, I can confirm from local testing (Ubuntu 20.04) that:

  • tests on yqna:master fail with my system locale set to local time and pass when I set my locale to UTC
  • tests on this branch pass regardless of system locale.

deleted avatar Oct 26 '22 19:10 deleted