on importFiles() ‘count’ event is fired after first ‘put’ event

[e-e-e]

Is this intended behaviour?

Current behaviour means that we can’t get a proper progress reading on the first few ‘put’ events. This could be easily solved by waiting until the dry run - https://github.com/datproject/dat-node/blob/master/lib/import-files.js#L36-L49 - is completed until starting the mirroring process. But then this will also slow down the process of importing for larger dats.

[joehand]

Yes... but perhaps not optimal (or well documented). As you say, I did this so large dats can start importing while counting is still happening, since that can take awhile. Right now I've left it up to the user to manage the two events + progress.

How do you think we could make this more clear? Or we could add an option to make the count complete before importing starts.

Right now it gets a bit confusing, especially for resuming large dats. But my goal was to make importing faster with the tradeoff of lightly less accurate progress.

[e-e-e]

I have been thinking about this and I am not certain how to make this clearer. But perhaps it could be useful to implement some sort of approximation indicator for while the initial count is still being calculated, so it does not hold up importation, but also provides context. If you like I am happy to open a PR - implementing a possible version of this while also adding some documentation of the count feature.

[joehand]

Perhaps it could be useful to implement some sort of approximation indicator for while the initial count is still being calculated

Yea, this is how I started the implementation in CLI. It definitely works well for new dat imports. Where it gets strange is resuming old dats (especially big ones) and live imports. You still need to take into account if the count is finished or not for the UI to make sense, this is part of why I left it up to the implementation.

I think I'd be okay to expose something like:

importer.stats = {
  remaining: 10 // count
  imported: 5 // importer.on('put')
  liveImports: 9 // this isn't reflected in remaining
}

Where I really get stuck up is the live importing, since that isn't reflected in the dry run counts. And those imports happen immediately, skipping the line of remaining. Happy to review any PRs or docs =).

You probably have seen it, but if not, it may be worth going through the CLI import usage (though I think there are a few edge cases I still haven't caught). The part that is hard is if count isn't done.