Open large binary file in kaitai struct for javascript?

[Meigyoku-Thmn]

The current javascript implementation requires reading a whole file into memory (ArrayBuffer). This is not good if I want to read a large file, for example a very big zip file. Can you add an option to use file descriptor (returned by fs.openSync) instead of ArrayBuffer?

[GreyCat]

That's actually a viable idea, which would hopefully require only adjustments to runtime. Any JavaScript volunteers?

[cherue]

This is even possible in browsers using <input type="file">:

const file = input.files[0]
const slice = file.slice(offset, offset + size)

slice.arrayBuffer().then(
    arrayBuffer => {
        const data = new Uint8Array(arrayBuffer)
    }
)

[GreyCat]

Wow, that's even better :)

I wonder what the overhead of using slice() + arrayBuffer() + new Uint8Array for every single read will be significant vs one ArrayBuffer + Uint8Array for whole file?

[cherue]

Yeah that'll probably be an issue, especially for small files.

Maybe something like Python's from_bytes and from_file could work, if it all goes through the KaitaiStream constructor it wouldn't even need compiler changes.

• ArrayBuffer would be the current implementation
• Blob (File's ancestor) would be reading from a file in the browser
• string (path) and number (file descriptor) would read from a file in Node