bases with multiple tables and a lot of records take a lot of time to sync

[mahmouds12]

I tried mongo-airtable to pull my base to mongoDB, my Airtable base has 12 tables and more than 6500 records... it took 190 seconds to finish pulling and inserting I tried tp modify some functions to make it work async, all tables start pulling at once but as soon as any table finish pulling it will start inserting into mongodb directly ... this improved the performance and took 92 seconds instead of 190 seconds this still too much as I am intending to fire this syncing process every 5 minutes intervals or less to always keep mongodb synchronized with airtable

here is my edits to reach 92 secs... added this function to airtable.js:

 export async function initialPullAndInsert(config) {
   for (const table_to_sync of config.sync) {
     pullTable({
       ...table_to_sync,
       auth_key: config.auth.airtable
     }).then(() => {
       console.log(`end sync of ${table_to_sync.primary}`)
       const last_pulled_filename = path.resolve(`${__dirname}/../build/last-pulled.txt`);
       writeFile(last_pulled_filename, moment().format('ddd MMM D YYYY h:mm A ZZ'), 'utf-8').then(() => {
         initialInsertTable(table_to_sync);
       })
     })
   }
 }

and added this to mongodb.js:

 export async function initialInsertTable(table_to_sync) {
   const { primary, collection, database } = table_to_sync;
     await putIntoDB({
       primary,
       collection,
       database
     })
   }

and in pull.js I called await initialPullAndInsert(config);

[mahmouds12]

I also did the same for the update function "seeWhatChanged(config)" and also it finishes in 25 secs instead of 75 secs

[pondpiu]

just sharing my thoughts for me (and my use case), its better to just modify initialPull to pull in parallel then only start inserting after pulling is finished as is.

// airtable.js
export async function initialPull(config) {
  await Promise.all(
    config.sync.map(async (table_to_sync) => {
      await pullTable({
        ...table_to_sync,
        auth_key: config.auth.airtable,
      })
      console.log(`end sync of ${table_to_sync.primary}`)
    })
  )

  const last_pulled_filename = path.resolve(
    `${__dirname}/../build/last-pulled.txt`
  )
  return await writeFile(
    last_pulled_filename,
    moment().format('ddd MMM D YYYY h:mm A ZZ'),
    'utf-8'
  )
}

[Arro]

Definitely some room for optimization here. I need to get this repo back into a maintained state. Will accept PRs as well. Thanks for alerting me.