hash
hash copied to clipboard
Migrate from `arrow` (`apache/arrow-rs`) to `arrow2` (unofficial, safer)
🌟 What is the purpose of this PR?
This migrates the engine from arrow (https://github.com/apache/arrow-rs) to arrow2 (https://github.com/jorgecarleitao/arrow2).
🔗 Related links
- Asana task (internal)
🔍 What does this change?
- Removes
arrowand replaces it witharrow2. In many cases, the code is now a lot safer than it used to be.
⚠️ Known issues
- warning: it compiles but in some cases I just took a guess as to what the correct implementation was (and will need to test this)
- slower expected simulation run executions
🛡 What tests cover this?
- The test suite (let us hope)
Would love to see performance benchmarks between the two (although accept we're committed to this now anyway for reasons extensively internally documented).
@teymour-aldridge I've tagged Tim and Alfie in as reviewers now this has been marked ready. If you'd prefer they hold, please Slack.