indexify
indexify copied to clipboard
Feat: Implement async sync_files method in data loaders in python SDK
Problem?
In python-sdk/data_loaders, we have url_loader.py and local_directory_loader.py. We need to add another method sync_files to the existing data loaders which will return a Generator of the file. The requirement is to write it in an async way so that indexify can poll it internally.
User DX -
The function should look like -
def sync_files(s3_bucket: str) -> Generator[File]:
...
yield File(...)
Testing -
TBD
@diptanu Please take a look at this issue's description, and also assign it to me. Thanks!