indexify icon indicating copy to clipboard operation
indexify copied to clipboard

Feat: Implement async sync_files method in data loaders in python SDK

Open Default2882 opened this issue 1 year ago • 1 comments

Problem?

In python-sdk/data_loaders, we have url_loader.py and local_directory_loader.py. We need to add another method sync_files to the existing data loaders which will return a Generator of the file. The requirement is to write it in an async way so that indexify can poll it internally.

User DX -

The function should look like -

def sync_files(s3_bucket: str) -> Generator[File]:
  ...
  yield File(...)

Testing -

TBD

Default2882 avatar Nov 06 '24 12:11 Default2882

@diptanu Please take a look at this issue's description, and also assign it to me. Thanks!

Default2882 avatar Nov 06 '24 12:11 Default2882