Does mongo-arrow provide real zero copy in the chain mongodb->arrow->pandas?

[sergun]

.. or zero copy appear only between arrow->pandas but not here mongodb->arrow?

In other words are arrow data types used in mongodb?

[ShaneHarvey]

The MongoDB -> Arrow portion is not zero copy. This library translates (copies) MongoDB's BSON document result structure into Arrow's data format. There are no native Arrow types in MongoDB. Alternatively, the app can store Arrow data as opaque BSON binary data in MongoDB but then the app looses the ability to query the data.

[sergun]

The MongoDB -> Arrow portion is not zero copy. This library translates (copies) MongoDB's BSON document result structure into Arrow's data format. There are no native Arrow types in MongoDB. Alternatively, the app can store Arrow data as opaque BSON binary data in MongoDB but then the app looses the ability to query the data.

Thanks a lot! Do you know if there are plans to make something like Arrow backend in MongoDB to store all documents internally in Arrow structures as it was made in pandas 2.x and in Spark?

PS. My original question was caused by use case of batch processing of data stored in MongoDB by logic expressed in pandas. Try to find some light-weight alternative to popular solution like Spark+pandas_udf (which uses Arrow end-to-end: Spark->Arrow->pandas).

[blink1073]

Hi @sergun, following up, no, there are no plans to change the storage format on MongoDB for alternate formats.