Fix ArrowInvalid for large variables
Fixes #7821
In addition to the solution proposed in the issue, I encountered we also need to support 64bit types when writing using the RecordBatchStreamWriter.
Not sure if we want to create such large objects in the CI, but this is the only way to test that the issue is fixed, therefore I added the unit test.
@lhoestq : from what I see in the CI logs ( tests/test_arrow_dataset.py::test_map_int32_overflow /opt/hostedtoolcache/Python/3.9.23/x64/lib/python3.9/site-packages/multiprocess/resource_tracker.py:219: UserWarning: resource_tracker: There appear to be 24 leaked shared_memory objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '), see here, the test crashes probably since it consumes too much memory, it works locally for me though.
Shall we keep the test, remove ti, or skip it, or is there a specific pytest flag you have for slow/memory-consuming tests I can add, so that it gets skipped automatically on the CI but might be run locally?
EDIT: I added the high_memory marker (see here), so this will not be run in the CI, which is not ideal but I guess the way if we want to keep the test