RangeError [ERR_OUT_OF_RANGE]

[staltz]

I was doing some not-so-careful experiments, and this meant I don't remember the reproduction steps perfectly, but I do know that I killed the app and then resumed db2 migration a couple of times. This happened suddenly:

RangeError [ERR_OUT_OF_RANGE]: The value of "offset" is out of range. It must be >= 0 and <= 65534. Received 80228
    at boundsError (internal/buffer.js:81:9)
    at Buffer.readUInt16LE (internal/buffer.js:238:5)
    at Object.getDataNextOffset (/data/data/se.manyver/files/nodejs-project/index.js:25310:31)
    at Stream.SyceLWe5Nm2mmym/v2ltZ2hAhKqlddxQJQd6z2avDTE=.Stream._handleBlock (/data/data/se.manyver/files/nodejs-project/index.js:41959:40)
    at /data/data/se.manyver/files/nodejs-project/index.js:42027:26
    at Request._callback (/data/data/se.manyver/files/nodejs-project/index.js:25269:9)
    at Request.Trwq+KqeWzxsbUet+Omb+wC9uH2QMBcZNBUiXOCqCv4=.Request.callback (/data/data/se.manyver/files/nodejs-project/index.js:42692:8)
    at onread (/data/data/se.manyver/files/nodejs-project/index.js:27150:31)
    at FSReqCallback.wrapper [as oncomplete] (fs.js:520:5) {
  code: 'ERR_OUT_OF_RANGE'
}

[arj03]

Is the app broken now? Or did it manage to fix itself after you reloaded?

[staltz]

Sorry, I deleted the db2 folder and restarted. I can try hard to reproduce it again, next week, maybe

[staltz]

Even if it would have recovered itself, I think a crash is anyway the wrong behavior, there should be a fix for that.

[arj03]

Fully agree about now happening in the first place.

This seems to be the same bug that Jacob ran into where I added some self-healing so it at least would be able to continue.

[staltz]

Jacob sent some crash reports, I'll check what those are too.

[staltz]

Let's assume this one was fixed by #36

[staltz]

I encountered this issue again, but the details are interesting.

It happened in Manyverse Desktop. It was a persistent crash, in the sense that every time I restarted the app, it would crash upon startup. Then I realized that I was still using the old async-aol because I grep'ped for writeWithFSync and it wasn't there. So I then obviously just updated async-aol and other dependencies, and thought this issue would go away, BUT it continued to crash upon startup every time I opened the app, even though it had async-aol with writeWithFSync (I double checked).

The crash looks like this:

[arj03]

#36 is supposed to fix the bug before it happens. So in that regard we don't know if it would have fixed the bug or not. There is some code (#12) that should automatically fix the latest block if it happens, so for some reason it must have written another block and can't auto-fix.

[staltz]

I wonder why does deleting indexes folder fix the issue for me, if async-aol is only concerned with the log. Technically, is the log broken or are the indexes?

[arj03]

I can confirm that it is possible to reproduce this bug if the offset in an index is incorrect (pointing to a random point in the file). Not really sure how that could happen, but at least this means that the bug might not be in this module, but rather jitdb (maybe).

[arj03]

Instead of random point in file, it might be more correct to say that the index is pointing to somewhere in the log that is not an entry. So this could either be because the index offset is wrong, or that the log is wrong compared to what it told the index. In the case of indexes, we do have crc checks so a write corruption is very unlikely. It could potentially still be the bug that was fixed in #36. Meaning on the log side. I'll see if I can do some more checking in jitdb, it is just rather simple in that it gets an offset + data for a record and processes that, finally updating the offset for the index.