langchain
langchain copied to clipboard
Hugging Face Loader: Add lazy load
Add lazy load to HF datasets loader
Unfortunately, there are no tests as far as i can tell. Verified code manually.
Same feature as this: https://github.com/hwchase17/langchain/pull/3501/files#diff-257a9ed31010c03aefa6e28d08934579d4b596d5e61090ad6d6d6583c5f8f317
@vowelparrot I'm going to merge as is. My understanding is that adding a default streaming=True may change behavior in an undesired way (data isn't downloaded locally -- which I think is often desirable). Instead we can follow up with a PR to expose more parameters in the init that will be passed to the load datasets function.