firestore2json.js fetch next batch

[martin-g]

Improve documentation

Link

Add a link to the page which needs improvement (if relevant)

https://github.com/supabase-community/firebase-to-supabase/blob/main/firestore/README.md

Describe the problem

Is the documentation missing? Or is it confusing? Why is it confusing?

https://github.com/supabase-community/firebase-to-supabase/blob/main/firestore/README.md#dump-firestore-collection-to-json-file says:

node firestore2json.js <collectionName> [<batchSize>] [<limit>]

batchSize (optional) defaults to 1000
output filename is <collectionName>.json
limit (optional) defaults to 0 (no limit)
note: <collectionName> is just the name of the collection (do not add .json at the end of this on the command line

The usage of batchSize is unclear to me. If I run node firestore2json.js collectionName then it downloads exactly 1000 records, as the default value of batchSize. If I run node firestore2json.js collectionName 100 then it downloads exactly 100 records, as the provided value of batchSize.

Describe the improvement

Please explain how to get the next batch, i.e. the next 1000 records

Additional context

I even tried node firestore2json.js collectionName 100000 but then it downloads just 1141 records, while my collection has 9556 records...

[martin-g]

I just tried the TypeScript version but it errors with:

npx ts-node firestore2json.ts collectionName 10000
/home/martin/git/vanko/firebase-to-supabase/node_modules/ts-node/src/index.ts:859
    return new TSError(diagnosticText, diagnosticCodes, diagnostics);
           ^
TSError: ⨯ Unable to compile TypeScript:
firestore2json.ts:5:5 - error TS7034: Variable 'processDocument' implicitly has type 'any' in some locations where its type cannot be determined.

5 let processDocument;
      ~~~~~~~~~~~~~~~
firestore2json.ts:11:5 - error TS7034: Variable 'db' implicitly has type 'any' in some locations where its type cannot be determined.

11 let db;
       ~~
firestore2json.ts:41:28 - error TS7053: Element implicitly has an 'any' type because expression of type 'string' can't be used to index type '{}'.
  No index signature with a parameter of type 'string' was found on type '{}'.

41             console.log(`${recordCounters[key]} records written to ${key}.json`);
                              ~~~~~~~~~~~~~~~~~~~
firestore2json.ts:47:11 - error TS7034: Variable 'data' implicitly has type 'any[]' in some locations where its type cannot be determined.

47     const data = [];
             ~~~~
firestore2json.ts:49:9 - error TS7053: Element implicitly has an 'any' type because expression of type 'string' can't be used to index type '{}'.
  No index signature with a parameter of type 'string' was found on type '{}'.

49     if (recordCounters[collectionName] >= limit) {
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
firestore2json.ts:50:17 - error TS7005: Variable 'data' implicitly has an 'any[]' type.

50         return {data, error};
                   ~~~~
firestore2json.ts:52:16 - error TS7053: Element implicitly has an 'any' type because expression of type 'string' can't be used to index type '{}'.
  No index signature with a parameter of type 'string' was found on type '{}'.

52     if (typeof recordCounters[collectionName] === 'undefined') {
                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
firestore2json.ts:53:9 - error TS7053: Element implicitly has an 'any' type because expression of type 'string' can't be used to index type '{}'.
  No index signature with a parameter of type 'string' was found on type '{}'.

53         recordCounters[collectionName] = 0;
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
firestore2json.ts:56:49 - error TS7053: Element implicitly has an 'any' type because expression of type 'string' can't be used to index type '{}'.
  No index signature with a parameter of type 'string' was found on type '{}'.

56         batchSize = Math.min(batchSize, limit - recordCounters[collectionName]);
                                                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
firestore2json.ts:58:11 - error TS7005: Variable 'db' implicitly has an 'any' type.

58     await db.collection(collectionName)
             ~~
firestore2json.ts:62:11 - error TS7006: Parameter 'snapshot' implicitly has an 'any' type.

62     .then(snapshot => {
             ~~~~~~~~
firestore2json.ts:63:24 - error TS7006: Parameter 'fsdoc' implicitly has an 'any' type.

63       snapshot.forEach(fsdoc => {
                          ~~~~~
firestore2json.ts:69:47 - error TS7005: Variable 'processDocument' implicitly has an 'any' type.

69         console.log('processDocument', typeof processDocument);
                                                 ~~~~~~~~~~~~~~~
firestore2json.ts:70:13 - error TS7005: Variable 'processDocument' implicitly has an 'any' type.

70         if (processDocument) {
               ~~~~~~~~~~~~~~~
firestore2json.ts:77:12 - error TS7006: Parameter 'err' implicitly has an 'any' type.

77     .catch(err => {
              ~~~
firestore2json.ts:80:13 - error TS7005: Variable 'data' implicitly has an 'any[]' type.

80     return {data, error};
               ~~~~

    at createTSError (/home/martin/git/vanko/firebase-to-supabase/node_modules/ts-node/src/index.ts:859:12)
    at reportTSError (/home/martin/git/vanko/firebase-to-supabase/node_modules/ts-node/src/index.ts:863:19)
    at getOutput (/home/martin/git/vanko/firebase-to-supabase/node_modules/ts-node/src/index.ts:1077:36)
    at Object.compile (/home/martin/git/vanko/firebase-to-supabase/node_modules/ts-node/src/index.ts:1433:41)
    at Module.m._compile (/home/martin/git/vanko/firebase-to-supabase/node_modules/ts-node/src/index.ts:1617:30)
    at Module._extensions..js (node:internal/modules/cjs/loader:1435:10)
    at Object.require.extensions.<computed> [as .ts] (/home/martin/git/vanko/firebase-to-supabase/node_modules/ts-node/src/index.ts:1621:12)
    at Module.load (node:internal/modules/cjs/loader:1207:32)
    at Function.Module._load (node:internal/modules/cjs/loader:1023:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:135:12) {
  diagnosticCodes: [
    7034, 7034, 7053, 7034,
    7053, 7005, 7053, 7053,
    7053, 7005, 7006, 7006,
    7005, 7005, 7006, 7005
  ]
}

[martin-g]

node firestore2json.js collectionName 100 100000 seems to behave better - it again downloads 1141 records, i.e. it seems it uses 100 as a batchSize internally and fetches up-to 100000 (the limit) records.

It seems this is not an issue for Documentation improvement, but there are actual bugs in the JS implementation.