Reduce memory pressure when decrypting large binary size

[wingyplus]

Hi,

I found memory pressure issues when encrypting ~200 MB bytes and decrypting it. Let me show reproduce steps:

First, setup vault in project:

defmodule CloakBug.Vault do
  use Cloak.Vault, otp_app: :cloak_bug
end

import Config

config :cloak_bug, CloakBug.Vault,
  ciphers: [
    default:
      {Cloak.Ciphers.AES.GCM,
       tag: "AES.GCM.V1",
       key: Base.decode64!("nZqBx2TpKXU/c+SX54Ki65igNzqPO8JT8iUfUSkKV7g="),
       iv_length: 12}
  ]

Then open IEx shell and prepare encryption:

iex(1)> pid = CloakBug.Vault.start_link()      
iex(2)> b = :rand.bytes(209_715_200) # ~200MB
iex(3)> ciphertext = CloakBug.Vault.encrypt!(b)

Open observer before decrypting ciphertext and decrypting it!:

iex(4)> :observer.start
iex(5)> CloakBug.Vault.decrypt!(ciphertext)  

Here is screen captures from observer:

Memory Usage:

Carrier Size:

All Allocator Types:

After tweak with this PR:

Memory Usage:

Carrier Size:

All Allocator Types:

The issue that I found is copy message that is converted to list during decoding tag. The code is not perfect, feel free to discuss.

[coveralls]

Coverage: 98.98%. Remained the same when pulling d9399116f0789bcf8738c64b021d26032286ef50 on wingyplus:optimize-cloak-tags-decoder into 404727b1b81b60cbbbb5441e72d45ca391e365a0 on danielberkompas:master.

[wingyplus]

ping @danielberkompas 🙇

[danielberkompas]

@wingyplus Sorry for the long delay here. My prior role at my job didn't leave me with much energy for open source maintenance.

I really appreciate all the charts and graphs showing the performance improvement here. Can you elaborate on the exact problem though? I know it's been awhile, so you may not remember the details at this point.

The issue that I found is copy message that is converted to list during decoding tag. The code is not perfect, feel free to discuss.

[wingyplus]

@danielberkompas Let me recall the situation ...😂

In my work, I used this library to encrypt the zip file that's contain a bunch of XML file.

One day our production got crash, I start investigating, and found that it crash because of OOM while decrypting the zip file, the file size is ~115MB.

[wingyplus]

Hi @danielberkompas, what else I need to do to get it merge?

[logandk]

We've encountered memory exhaustion for large files (~100 MB file consuming >4 GB memory) during decryption. I tested this PR and confirmed that memory consumption is significantly reduced.