appwrite
🚀 Feature: Bulk Document Creation
🔖 Feature description
Create a "createDocuments" post endpoint that takes an array of documents.
🎤 Pitch
In my project, I am trying to insert 12,000 documents in one go. It is inefficient running 12,000 external createDocument API calls to achieve this.
To remedy this, there should be a "createDocuments" endpoint that allows you to pass an array of documents. That way you could easily break up the number of external calls required, and allow the stack to process the creation of the large amount of documents interally, quickly and efficiently.
A potential workaround would be creating a function that does it locally, but I don't believe this is a proper solution.
👀 Have you spent some time to check if this issue has been raised before?
- [X] I checked and didn't find similar issue
🏢 Have you read the Code of Conduct?
- [X] I have read the Code of Conduct
I have a achieved an okay workaround by creating a ".json" file inside a bucket that contains an array of documents, which is then read by a function that inserts the required documents.
I have a achieved an okay workaround by creating a ".json" file inside a bucket that contains an array of documents, which is then read by a function that inserts the required documents.
It's currently taking about 46 seconds to process 15,863 simple json objects.
I have a achieved an okay workaround by creating a ".json" file inside a bucket that contains an array of documents, which is then read by a function that inserts the required documents.
Hi, can you point me to a demo file please. NodeJS if possible.
I have a achieved an okay workaround by creating a ".json" file inside a bucket that contains an array of documents, which is then read by a function that inserts the required documents.
Hi, can you point me to a demo file please. NodeJS if possible.
No worries.
You can find it at the link below. It was thrown together pretty quickly.
It expects a JSON file to be created in a bucket with the following structure
{ 'collection': 'collection_id', 'data': [ 'object-array' ] }
https://gist.github.com/Shadowfita/b5ccd20f65566cb9f2b40d416c5201a2
Using functions and bucket is brilliant solution!
Another potential workaround is to use multithreaded client and upload documents in parallel; for instance, client application developed in Kotlin. And, perhaps Server SDK can allow to create server-side extension; or, trivial, server-side application (such as "function", or Kotlin-based standalone) can read batch JSON from local filesystem (from "bucket", FTP, S3, etc) and create records, so we avoid excessive HTTP traffic.
Or, another solution:
- use "Staging" instance in local network, load data, backup MariaDB, restore MariaDB in production system
- backup production MariaDB, analyze SQL dump file, programmatically generate additional 12,000 SQL statements, and restore it
The best approach would be if Appwrite API has such functions: export (permissions/objects/buckets/collections/functions) (JSON), import, etc.; in this case we "abstract" underlaying implementation details and can do more granular export/import. For example, right now we don't have implementation-independent backup/restore (except that executing sqldump locally; what about cluster then?)
It would be great if bulk update is also considered as it ll greatly reduce the number of requests I make in my application.
Hey @stnguyen90 , Can I work on this. I have gone through the requirements and tried to implement a basic endpoint to achieve this and was able to implement this on my local setup for Bulk create. I would be happy to contribute to this.
@singhbhaskar, thanks for your interest! 🙏 However, it would be best for the core team to figure out how it should work.
I need this also, need to create 18K document which are Pincode I should say...
So I am working on a scrapping project , there are almost 2K records , it takes almost 20 min to write !!!
{"ColumnRef":"Name","index":"0","value":"orange house"}
this is the size of each record !!!
is there any way to speed up the writes !!
So I am working on a scrapping project , there are almost 2K records , it takes almost 20 min to write !!!
{"ColumnRef":"Name","index":"0","value":"orange house"}this is the size of each record !!!
is there any way to speed up the writes !!
You should run all write requests asynchronously and wrap them in Promise.all . Appwrite is built to scale and will handle that many concurrent requests with no issue, and it will reduce your wait-time exponentionally.
@Shadowfita Is there a guarantee that all promises will resolve if we do it with Promise.all now?
Last time I checked like 2 months ago, I tried to do bulk document deletion with Promise.all and it seems some of them failed.
Is there a guarantee that all promises will resolve if we do it with
Promise.allnow?
Answer depends actually,
INstead you should delete in batches, meaning 10-50 per Promise instead of all at one. if you do all at one then it will likely fail because of limitation of the server.
