Improve handling of large amount of data on drives

[Deadolus]

I now have >100GB on my Google drive and gcsf is not able to handle such large drives in a practical manner. It seems that gcsf is indexing the whole drive on startup, which takes quite some time for me. google-drive-ocamlfuse does a on-needed based lookup of the files. It would be great if gcsf could also handle such large drives in the future.

$ gcsf mount /mymountpoint/ -s mysession
 INFO  gcsf > Creating and populating file system...
 INFO  gcsf::gcsf::drive_facade > Received page 1 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 2 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 3 containing 460 files
 ... about 10 minutes later ..
 INFO  gcsf::gcsf::drive_facade > Received page 219 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 220 containing 125 file
 INFO  gcsf::gcsf::drive_facade > Received page 1 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 2 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 3 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 4 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 5 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 6 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 7 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 8 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 9 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 10 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 11 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 12 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 13 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 14 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 15 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 16 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 17 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 18 containing 460 files
 INFO  gcsf::gcsf::drive_facade > Received page 19 containing 258 files
 INFO  gcsf                     > File sytem created.
 INFO  gcsf                     > Mounting to /mymountpoint/
 INFO  gcsf                     > Mounted to /mymountpoint/
 INFO  gcsf::gcsf::file_manager > Checking for changes and possibly applying them.
...another 30 minutes or so later ...

INFO  gcsf::gcsf::file_manager > Checking for changes and possibly applying them.
 ERROR gcsf::gcsf::drive_facade > Got error: ErrorMessage { msg: "BadRequest(\n    ErrorResponse {\n        error: ServerError {\n            errors: [\n                ServerMessage {\n                    domain: \"global\",\n                    reason: \"fileNotDownloadable\",\n                    message: \"Only files with binary content can be downloaded. Use Export with Google Docs files.\",\n                    location_type: Some(\n                        \"parameter\"\n                    ),\n                    location: Some(\n                        \"alt\"\n                    )\n                }\n            ],\n            code: 403,\n            message: \"Only files with binary content can be downloaded. Use Export with Google Docs files.\"\n        }\n    }\n)" }

At this point I can finally see my files in the mount location. This long start up duration is not practical for me and I hope that you find the time to improve the situation in the future.

Thanks for your efforts.

[zmwangx]

Wow, seeing that your 19 pages of files took 30 minutes to initialize, I'm glad I killed gcsf within the hour after it fetched 910 pages of files and just sat there churning away in the background without even the "File system created" message.

[harababurel]

Wrote about this briefly in https://github.com/harababurel/gcsf/issues/35#issuecomment-460017177. Will use #4 to keep track of this task as I make progress on it.