dirkmc

https://github.com/raulk/whitenoise/pull/1 updates whitenoise to point at https://github.com/filecoin-project/go-data-transfer/pull/217 which eliminates the received CIDs bottleneck. It seems that the transfer is still relatively slow on my machine: 1m41s to transfer 4GB: 40MB/s...

@aarshkshah1992 your suspicions seem to be correct, updating to graphsync v0.8.0 doesn't seem to make much difference. I still get a transfer speed of about 40 MB/s. The CPU profile...

Still with graphsync v0.8 but now with an in-memory datastore: - The transfer is significantly faster: 4GB takes 45s: 95MB/s - The major bottleneck seems to be hashing: ...

Results for [graphsync v0.7 with an in-memory datastore](https://github.com/raulk/whitenoise/compare/master...dirkmc:exp/gs-0.7?expand=1): The transfer time is comparable to graphsync v0.8: 4GB takes 45s: 95MB/s The CPU profile also looks similar:  [gs-0.7-inmem-prof.zip](https://github.com/filecoin-project/go-data-transfer/files/6631206/gs-0.7-inmem-prof.zip)

To try to remove any noise from other processes from running on a laptop I ran the profiles on a cloud machine: AWS c5d.12xlarge: 48vCPU / 96 GB RAM I...

The underlying cause actually seems to be a bug in graphsync, as described in the comment: ``` // TODO: There seems to be a bug in graphsync. I believe it...

Thanks @willscott - is there a pressing need to update go-ipld-prime or is this just a courtesy advisory?

I think the solution here is that the data transfer should attempt to restart the stalled channel automatically. We have code to do automatic restarts but it's still a little...

Hey, thanks for your interest in the plugin. I've been trying to track down that rename bug for a while now, it seems so obvious now that you've pointed it...

Ah ok I think I understand what you're saying. The reason the request key is a hash, instead of the original list of filenames is because if you have a...