hash icon indicating copy to clipboard operation
hash copied to clipboard

Migrate from `arrow` (`apache/arrow-rs`) to `arrow2` (unofficial, safer)

Open teymour-aldridge opened this issue 3 years ago • 1 comments

🌟 What is the purpose of this PR?

This migrates the engine from arrow (https://github.com/apache/arrow-rs) to arrow2 (https://github.com/jorgecarleitao/arrow2).

🔗 Related links

🔍 What does this change?

  • Removes arrow and replaces it with arrow2. In many cases, the code is now a lot safer than it used to be.

⚠️ Known issues

  • warning: it compiles but in some cases I just took a guess as to what the correct implementation was (and will need to test this)
  • slower expected simulation run executions

🛡 What tests cover this?

  • The test suite (let us hope)

teymour-aldridge avatar Jul 21 '22 13:07 teymour-aldridge

Would love to see performance benchmarks between the two (although accept we're committed to this now anyway for reasons extensively internally documented).

vilkinsons avatar Jul 21 '22 15:07 vilkinsons

@teymour-aldridge I've tagged Tim and Alfie in as reviewers now this has been marked ready. If you'd prefer they hold, please Slack.

vilkinsons avatar Aug 16 '22 12:08 vilkinsons