Inappropriate use of persistent_term may be affecting the entire VM

[alco]

persistent_term is a specialized hash table inside BEAM VM that is optimize for reading: looking up values by key doesn't involve any copying and is an O(1) operation.

That read performance comes at a cost though:

• putting a new value (even for an already existing key) copies the whole hash table every time
• if the value is complex (not a number or an atom), replacing it with a new value causes the VM to schedule a full heap scan for every single process
• if the old value is still used by a process, that process will do a major (fullsweep) garbage collection and copy the term into the process.

In our own code we use persistent_term in two places:

• to store the generation key for metric exports in elixir-otel-metric-exporter. The value is an atom, so updating it doesn't cause a global process heap scan but the underlying hash table is still copied on every put.
• to store span attributes per StackSupervisor. The new value is only written at StackSupervisor initialisation. When the stack is restarted, it puts the same value into persistent_term as was there previously. In this case, the put operation is optimized to be a noop, so there isn't even copying of the table.

Some of our deps use persistent_term: protobuf, finch, mint, opentelemetry, opentelemetry_api and others. It's best to audit their use of it for our peace of mind.