[Perf] Don't recreate ReadOptions when querying the database

[ljedrz]

Heap profiling indicates that plenty of allocations currently come from the use of rocksdb_readoptions_create, which is an FFI function called internally from get_pinned. This can easily be avoided by creating an instance of ReadOptions just once and reusing it via get_pinned_opt, which is the equivalent of get_pinned with a &ReadOptions parameter.

This change reduces (in a 15-minute run consisting of 4 --dev validators) the average number of allocations per second in a dev node that doesn't produce txs by roughly 30%, which is beneficial for the performance of any heap allocator.

[ljedrz]

under what conditions do you see the reduction? Is it just normal operation of a node minus any load?

The difference was measured for a --dev 2 node which doesn't produce txs, but participates in consensus.

Would it be possible to profile this under heavy read load?

I requested a devnet run to see the impact with more transactions.

[kpandl]

Performed some memory profiling with ~2.5 hours of 25 TPS here. After the test, snarkOS maximum memory usage was reduced by 8% and average memory usage was reduced by 4.4% through this PR (maximum/average of all 10 nodes).

[iamalwaysuncomfortable]

A devnet test of this to profile has been kicked off

[ljedrz]

Observations based on devnet metrics:

• memory use is a bit lower, but there is still growth
• log frequency is up, suggesting higher average throughput
• average block lag is down, and its spikes are much lower
• number of threads appears to be a bit down
• minor page faults are reduced slightly

Overall, this change results in some small performance wins, and its effects are more significant with workloads that involve many database reads.

[howardwu]

I will await for approvals from reviewers prior to merging this one.

[ljedrz]

Update: the same can be done with WriteOptions; the win is smaller, as we write less frequently.

[howardwu]

@joske can you also review this PR?