langchain icon indicating copy to clipboard operation
langchain copied to clipboard

Hugging Face Loader: Add lazy load

Open eyurtsev opened this issue 2 years ago • 1 comments

Add lazy load to HF datasets loader

Unfortunately, there are no tests as far as i can tell. Verified code manually.

eyurtsev avatar May 16 '23 17:05 eyurtsev

Same feature as this: https://github.com/hwchase17/langchain/pull/3501/files#diff-257a9ed31010c03aefa6e28d08934579d4b596d5e61090ad6d6d6583c5f8f317

eyurtsev avatar May 16 '23 17:05 eyurtsev

@vowelparrot I'm going to merge as is. My understanding is that adding a default streaming=True may change behavior in an undesired way (data isn't downloaded locally -- which I think is often desirable). Instead we can follow up with a PR to expose more parameters in the init that will be passed to the load datasets function.

eyurtsev avatar May 17 '23 16:05 eyurtsev