DAGWorks-Inc
[good first issue - intermediate/advanced] Moar data adapters (i.e. materializers)!
Is your feature request related to a problem? Please describe. We have a few default ones, and could use a ton of extensions.
Describe the solution you'd like Moar data adapters!
Describe alternatives you've considered These should be small/lightweight extensions. Similar to the rest of the extension framework.
Some ideas:
- postgresql (needs some SQL stuff, might be complicated. Also connection management...)
- duckdb (ditto)
- arrow integrations
- s3-specific ones
- big query
- image
- image groups
- geopandas shape file
- Pandas on spark for ^^^ + the defaults
- Polars for ^^^ + the defaults
- openml data sets -- making it easy to load them
- huggingface -- making it easy to pull things from that ecosystem
Additional context
Will need extension help -- see, for example, https://github.com/DAGWorks-Inc/hamilton/blob/main/hamilton/plugins/pandas_extensions.py to get started. The TL;DR is that the extensions are responsible for registering, and raise a NotImplementedError if the required dependency isn't present (which then gets caught downstream). They're all loaded here: https://github.com/DAGWorks-Inc/hamilton/blob/bfc2300bf06cd83ab28b514474732c7a27698dd3/hamilton/function_modifiers/base.py#L26. That way you don't have to have extra setup.py, etc... for each one.
postgresql (needs some SQL stuff, might be complicated. Also connection management...)
How about tackling SQLite first to side-step the connection management complexity?
postgresql (needs some SQL stuff, might be complicated. Also connection management...)
How about tackling SQLite first to side-step the connection management complexity?
@Scorpil Yep that would simplify things, but to narrow the issue down, it's also a question of what the object type is being serialized/deserialized? We have Pandas SQL (#355 ) that covers this for Pandas dataframes, but not for other object types. Did you have an object type in mind to serialize to/from SQLite?
