cryptoENgine.js/HexEncoder.stringify() throws RangeError: Invalid array length

[jimhark]

I'm trying to encrypt a large (140 MB) html page using this command (PowerShell 7):

node .\cli\index.js .\page.html --password test --short --remember false

I get this error:

Exception has occurred: RangeError: Invalid array length
  at Array.push (<anonymous>)
    at Object.stringify (c:\staticrypt\lib\cryptoEngine.js:45:22)
    at Object.encrypt (c:\staticrypt\lib\cryptoEngine.js:84:50)
    at async encodeWithHashedPassword (c:\staticrypt\lib\codec.js:42:27)
    at async encodeAndGenerateFile (c:\staticrypt\cli\index.js:209:26) 

This occurs while processing the 112,813,857th byte. Results of Google searches suggests this may be caused by a bug in the JavaScript V8 engine.

I expect this to encrypt this file because my machine has plenty of memory (64 GB, more than half free) and 140 MB doesn't approach any documented internal limits of V8 JavaScript.

My setup:

StatiCrypt version: 3.5.4 Node version: 22.17.0 Windows 11

Fix (workaround, really) is available on my fork: https://github.com/jimhark/staticrypt/tree/ImproveBigFileSupport

I will submit a PR shortly.

[robinmoisson]

Hey @jimhark , a quick word to let you know all these PRs and issues look like really great work. Thank you very much!

I want to check it out in more depth to merge it in the main project. Work and life are very busy at the moment but should clear up at some point, so my apologies if it takes a while to review but know I'll definitely look into it.

And be sure I appreciate your work in improving StatiCrypt. Thanks!

[jimhark]

Thanks for the positive feedback. You were very clear in the readme that a response may take a while, so as far as I'm concerned you're ahead of schedule. No need to apologize. But let me apologize for a few things:

• While I've used git for years, these are my first github pull requests. One mistake I made was assuming a PR was for a commit. Now I see it's for a branch and see the value of that, it allows feedback/update iterations. My first 2 PRs would have been different if I knew what I was doing.
• For each PR I made an issue (a habit from work), the idea is to separate higher level / end-user discussions (issues) with technical, programmer, implementation details (PRs). If I had it to do over again, I would have grouped some PR under a single issue.
• It's not clear to me exactly how I could test the backwards compatible password hashes. Additional testing is needed.
• I've saved the worst for last. The second PR, the hex to binary conversion, is too large. It hits about 300 lines of about 1,500 lines (yes, 20%) of JavaScript in the codebase. I made it worse by combining 2 (smallish) related changes (already documented). The common thread was memory efficiency, but these should have been 3 PRs.

My work was motivated by needing to encrypt a large (140 MB) html. The encrypted file doubled in size and could not be opened by current versions Chrome and Android. With the updates (Base64 + compression) the encrypted file is smaller (134 MB) than but comparable the original file size and can be opened by Chrome on Android. Encrypt and decrypt performance has noticeably improved on my PC.

I have since selected large files from Project Gutenberg to use for performance testing, but have not had time to perform tests. If you have time performance tests I'd be very interested in a before / after comparison. I think memory requirements (for large files) has decreased by half or a third. In a memory constricted environment, I expect HTML file size limits should be 2x or 3x larger. If you already have tests for this, I'd love to see the results.

Thanks for creating staticrypt. I'm glad I've been able to contribute my time to such a valuable project.