Heavy RAM usage when encrypting large file

[shujaatak]

I was testing rage, I noticed that when encrypting a large video file of size like 4GB consumed more than 7GB of RAM.

It would be nice if the data is fetched, encrypted, and saved in the output file chunk by chunk so this way the RAM usage will be limited to max 2 chunks.

[str4d]

In the age library crate, that is indeed what we already do. However, in rage if both the input and output come from a TTY, we buffer the entire output before writing so that the output doesn't get in the way of typing the input: https://github.com/str4d/rage/blob/d7c727aef96cc007e142f5b21c0d19210154b3c7/age/src/cli_common/file_io.rs#L189-L195

How are you using rage in your workflow?

[shujaatak]

For example, there is an iso file of 4GB named a.iso and I want to encrypt it to a.o, so I use the following command in windows command prompt: rage -e -i key.txt -o a.o a.iso

But I notice heavy usage of RAM while doing this.

However, in rage if both the input and output come from a TTY, we buffer the entire output before writing so that the output doesn't get in the way of typing the input

This makes sense. So what I understand is to learn and try to use age library directly to test how it behaves in terms of memory.

[str4d]

For example, there is an iso file of 4GB named a.iso and I want to encrypt it to a.o, so I use the following command in windows command prompt: rage -e -i key.txt -o a.o a.iso

But I notice heavy usage of RAM while doing this.

Hmm, this specific prompt should not be triggering the "buffer output" logic, as neither the input nor the output involves a TTY. Maybe the TTY detection logic is not working correctly on Windows? What version of Windows are you on, and what specific command prompt (and version of it) are you using? (Command Prompt? PowerShell? Something else?)

[shujaatak]

@str4d

I am using Command Prompt on Windows 11 24H2.