Occasional "Download was canceled" error when start/end range is specified.

[ikreymer]

The following snippet occasionally results in this error:

var ram = require('random-access-memory');
var hyperdrive = require('hyperdrive');
var discovery = require('hyperdiscovery');

var opts = {"latest": false,
            "sparse": true}

var drive = hyperdrive(ram, '778f8d955175c92e4ced5e4f5563f69bfec0c86cc6f670352c457943666fe639');

drive.on("ready", function() {
  var sw = discovery(drive);

  var readOpts = {"start": 13, "end": 36}

  var stream = drive.createReadStream("dat.json", readOpts);

  stream.pipe(process.stdout);

  stream.on("error", function(err) {
    console.log(err);
  });

  stream.on("end", function() {
    process.stdout.write("\ndone\n");
  });

});

Sometimes this works correctly, and the output should be:

"Dat command line demo",
done

but occasionally this results in an error like this:

Error: Download was cancelled
    at Feed.undownload (http-range-dat/node_modules/hypercore/index.js:332:30)
    at onend (http-range-dat/node_modules/hyperdrive/index.js:498:24)
    at done (http-range-dat/node_modules/hypercore/index.js:561:5)
    at http-range-dat/node_modules/hypercore/index.js:529:40
    at onroot (http-range-dat/node_modules/hypercore/index.js:575:30)
    at onleftchild (http-range-dat/node_modules/hypercore/index.js:594:7)
    at Storage.getNode (http-range-dat/node_modules/hypercore/lib/storage.js:112:24)
    at onroot (http-range-dat/node_modules/hypercore/index.js:583:19)
    at onleftchild (http-range-dat/node_modules/hypercore/index.js:591:7)
    at Storage.getNode (http-range-dat/node_modules/hypercore/lib/storage.js:112:24)

I'm using this in the context of an http request (https://github.com/ikreymer/http-range-dat/) where this seems to happen more frequently. Perhaps I'm not using this correctly?

[magland]

I may be experiencing this issue as well, using dat-node. "Download was cancelled" error, and I am specifying a start/end range.

[magland]

Some more details... I am doing:

var stream = dat.archive.createReadStream(file_path, {start:start,end:end-1}); ...

The first time I do it, no problem. The second time (once the file already partially exists on local machine), that's when I get the "Download was cancelled" error.

[magland]

I managed to do a workaround, by wrapping my createReadStream code in

setTimeout(function() { ..... },0);

not sure why that's needed, but it certainly is in my case.

Update: Actually, that only works sometimes. As the original post indicates... this is an unpredictable issue.

[magland]

I solved the problem by commenting out the following code in hyperdrive/index.js:

if (length > -1 && length < stat.size) {
            self.content.seek(byteOffset + length, {start: start, end: end}, onend)
          }

Not sure exactly what this line does, but without it my app works beautifully. Do you want me to create a pull request @mafintosh ?

[mafintosh]

@magland open a PR, we'll discuss it there. If you add a test that'd be amazing. otherwise we'll manage :)

[magland]

Will do this afternoon. Thx.

[ikreymer]

Thanks @magland I can also confirm that by commenting out those lines, I am also no longer seeing the "Download was canceled" error either! Nice find! Always interesting when commenting out code fixes things, but I suppose if download already takes a range, the seek may be unnecessary? Look forward to seeing this fixed.

[eu1444]

I believe I have found and fixed the problem. I will open a PR.

Simply removing the lines discussed above does not fix the issue; it just makes createReadStream ignore the fact that opts.end was specified. The issue is that createReadStream was not accounting for opts.start correctly when seeking.

The code in question:

if (length > -1 && length < stat.size) {
      self.content.seek(byteOffset + length, {start: start, end: end}, onend)
}

should be:

if (length > -1 && length < stat.size) {
      self.content.seek(byteOffset + length + (opts.start ? opts.start : 0), {start: start, end: end}, onend)
}

Also, onend needed to be changed from this:

range = self.content.download({start: start, end: index, linear: true}, ondownload)

To this:

range = self.content.download({start: start, end: index + 1, linear: true}, ondownload)

The change to index + 1 accounts for the fact that the index returned from seek is the index the data is contained in and download takes a non-inclusive end option.