verified-sources
verified-sources copied to clipboard
mongodb: support end_date, row_order, limit and arrow backend
Source name
mongo
Describe the data you'd like to see
I'd like a mongodb
features on par with sql_database
features:
- backfill with incremental (start / end ranges) - we need to support end_value
- explicit ordering of results (support
row_order
- allow to set a
limit
on a source/resource level to load just N first results. will help with microbatching -
pymongo
got arrow support: https://github.com/mongodb-labs/mongo-arrow and we should use it
Are you a dlt user?
Yes, I'm already a dlt user.
Do you ready to contribute this extension?
Yes, I'm ready.
dlt destination
any
Additional information
ad limit
: limit works well with ordered results. if results are not ordered (lack of incremental field and declared order) - issue a warning!
Please split ticket into two PRs
- requirements 1-3
- pyarrow backend - a separate one