Handling Dynamic File Uploads in LlamaIndex RAG Pipeline

[jeekdata]

hello , I’m using the LlamaIndex RAG pipeline in a web UI and following the llamaindex_pipeline example. In the example, the document path is hardcoded like this:

python

self.documents = SimpleDirectoryReader("./data").load_data()
self.index = VectorStoreIndex.from_documents(self.documents)

However, in my case, users will upload files dynamically, so I don’t know in advance where the files will be stored. Because of that, I’m not sure how to reference the uploaded file path in the pipeline to load the document properly.

Has anyone dealt with this before or have any suggestions on how to handle dynamic file uploads with LlamaIndex? Any pointers would be appreciated!

Thanks!

[Saber-xxf]

same question as i

[torisetxd]

If this is relevant, files are stored within "UPLOAD_DIR" from open_webui.config

How i do it:

from open_webui.config import UPLOAD_DIR
import os 

for file in body['files']:
  for f in file['files']:
    if 'meta' in f:
      meta = f['meta']
      path = os.path.join(UPLOAD_DIR, f"{f['id']}_{meta['name']}")

And then you read them from there directly.

[torisetxd]

There might be a better way to do it but thats how i do it.

[arunbugkiller]

@jeekdata could you please provide details of the script you are using for LLamaIndex RAG

Another thing I want to know whether we can completely externalise the RAG pipeline instead of using OpenWebUIs backend RAG