[Upload] Enhancing Document Upload Experience and Performance

[DanielRSnell]

Enhancing Document Upload Experience and Performance

Overview

Currently, the document upload process in Quivr involves a series of sequences that can lead to a suboptimal user experience due to extended waiting times. This issue aims to propose and implement enhancements to reduce the perceived waiting time and improve the performance of document uploads.

Proposed Options

Option 1: Introduction of Serverless Function on Supabase

This option involves integrating a serverless function provided by Supabase that will handle actions on documents being added to the storage bucket. When a user uploads a document, the serverless function will be triggered, allowing the document processing to occur in the background while the user can continue with their workflow. Any failures or notifications related to document processing will be communicated to the user through SSE (Server-Sent Events) or appropriate notifications.

graph LR
A[User] -- Uploads document --> B(Quivr - NextJS)
B -- Sends document to serverless function --> C[Supabase - Serverless Function]
C -- Processes document in the background --> D((Supabase Storage Bucket))
D -- Notifies user of processing status --> A

Option 2: Offload to IndexedDB with Synchronization

The second option suggests offloading the document upload process to IndexedDB, a client-side database supported by modern web browsers. Upon document upload, the file will be stored locally in IndexedDB and synchronized with the server over time. This approach provides a seamless experience for the user, allowing them to proceed with their tasks while the document processing takes place in the background. SSE or notifications can be utilized to communicate any document processing failures to the user.

Suggested Improvements

In addition to the proposed options, the following suggestions can be considered to further enhance the document upload experience:

1. Progressive Upload: Implement a progressive upload mechanism that allows users to see the upload progress in real-time. This gives them a sense of the process and reduces the perception of waiting.

2. Chunked Uploads: Break large documents into smaller chunks and upload them incrementally. This approach improves reliability and allows resumable uploads in case of interruptions.

3. Parallel Processing: Utilize background threads or serverless functions to process documents concurrently, reducing the overall processing time.

4. Client-side Preprocessing: Offload computationally expensive preprocessing tasks, such as file format conversion or data extraction, to the client-side before uploading the document. This reduces the processing load on the server and provides faster feedback to the user.

[StanGirard]

Awesome !

I've added that to the roadmap. Please join ou discord.

I'd love to use the lambda from Supabase. However, that would tie use undeniably to Supabase, not a bad thing, but we might be able to avoid it.

I'd rather use Client-Sider preprocessing then. The langchain framework is compatible with JS :)

[sidhartha426]

Hello @gozineb @StanGirard I want to contribute can you guide me.Suggest me some packages and how to move forward.

[github-actions[bot]]

Thanks for your contributions, we'll be closing this issue as it has gone stale. Feel free to reopen if you'd like to continue the discussion.

[StanGirard]

Ping me on the discord :)

On Wed, Aug 23 2023 at 10:06 AM, github-actions[bot] < @.*** > wrote:

Thanks for your contributions, we'll be closing this issue as it has gone stale. Feel free to reopen if you'd like to continue the discussion.

— Reply to this email directly, view it on GitHub ( https://github.com/StanGirard/quivr/issues/138#issuecomment-1689483792 ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AEVUW3A6KDTW2FTZECNM3C3XWW2XVANCNFSM6AAAAAAYM7GR5M ). You are receiving this because you were mentioned. Message ID: <StanGirard/quivr/issues/138/1689483792 @ github. com>

[github-actions[bot]]

Thanks for your contributions, we'll be closing this issue as it has gone stale. Feel free to reopen if you'd like to continue the discussion.