turbodbc icon indicating copy to clipboard operation
turbodbc copied to clipboard

flag `truncate_timestamps` added to fetcharrow functions

Open JorgeGarciaIrazabal opened this issue 5 years ago • 8 comments

#207 Arrow only allows timestamps with years between 1400 to 9999. Adding the new boolean parameter truncate_timestamps to functions fetchallarrow and fetcharrowbatches to truncate the value to the valid range instead of throwing.

JorgeGarciaIrazabal avatar Apr 04 '19 19:04 JorgeGarciaIrazabal

It looks like OSX if failing but I don't think it is related to this PR (I might be wrong though)

JorgeGarciaIrazabal avatar Apr 05 '19 17:04 JorgeGarciaIrazabal

Hi @JorgeGarciaIrazabal - Could you please try rebasing to master and reopen the PR? OSX is fixed there.

MathMagique avatar Apr 08 '19 11:04 MathMagique

Codecov Report

Merging #209 into master will increase coverage by 0.05%. The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #209      +/-   ##
==========================================
+ Coverage   98.59%   98.65%   +0.05%     
==========================================
  Files         136      145       +9     
  Lines        2999     3349     +350     
==========================================
+ Hits         2957     3304     +347     
- Misses         42       45       +3
Impacted Files Coverage Δ
cpp/turbodbc/Library/src/time_helpers.cpp 100% <100%> (ø) :arrow_up:
cpp/turbodbc_arrow/Library/src/python_bindings.cpp 100% <100%> (ø) :arrow_up:
python/turbodbc/cursor.py 98.97% <100%> (ø)
...pp/turbodbc_arrow/Library/src/arrow_result_set.cpp 96.24% <100%> (-0.44%) :arrow_down:
...urbodbc_arrow/Test/tests/arrow_result_set_test.cpp 100% <100%> (ø) :arrow_up:
python/turbodbc/constructors.py 100% <0%> (ø)
python/turbodbc/exceptions.py 100% <0%> (ø)
python/turbodbc/__init__.py 100% <0%> (ø)
python/turbodbc/options.py 100% <0%> (ø)
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 8592759...c488cc5. Read the comment docs.

codecov-io avatar Apr 08 '19 13:04 codecov-io

Hi @JorgeGarciaIrazabal - Could you please try rebasing to master and reopen the PR? OSX is fixed there.

@MathMagique I have a different error after rebasing, I am not 100% sure why it failed this time though.

JorgeGarciaIrazabal avatar Apr 08 '19 15:04 JorgeGarciaIrazabal

@MathMagique running the job again, all the checks passed! let me know if you want me to change/add anything to the PR. Thanks in advance

JorgeGarciaIrazabal avatar Apr 15 '19 15:04 JorgeGarciaIrazabal

This is not an issue with Arrow but with boost::gregorian::date in the date_to_days function. I would rather fix that than have this truncation workaround.

xhochy avatar Apr 22 '19 19:04 xhochy

This is not an issue with Arrow but with boost::gregorian::date in the date_to_days function. I would rather fix that than have this truncation workaround.

Yeah, that makes more sense. Unfortunately, with my limited c++ experience, I can not think of a better alternative for the library. Are you guys planing on working on this any time soon?

JorgeGarciaIrazabal avatar Apr 27 '19 15:04 JorgeGarciaIrazabal

@wesm, @xhochy what would be the approrpriate changes to make in Arrow, I took at look at the datetime parts of arrow-cpp but it wasn't clear to me what we can use instead of boost::gregorian ?

mariusvniekerk avatar Jun 26 '19 19:06 mariusvniekerk