Syscall to get the remaining log units

[armaniferrante]

Problem

When a program runs out of log units, newly emitted msg! (and others like sol_log_data) logs will silently fail while the transaction succeeds. This is problematic for the fidelity of off chain indexers that are reading these logs for certain events and other information.

Proposed Solution

Add a syscall to get the remaining log units, so that programs can determine if a log succeeded or not, and if not, abort the transaction.

[mvines]

The trick here is the allowed log units is not a part of consensus, for example RPC nodes may wish to configure this value higher than a staked validator.

Perhaps additional transaction metadata could be added to determine if log truncation actually occurred for a given RPC endpoint/transaction

[t-nelson]

It sounds like the goal is to fail the transaction if it cannot be guaranteed that an event was emitted. The indexer being able to notice after the fact is secondary as the transaction succeeding is undesirable.

I guess this is the first example of aggregating all costs into a nebulous "compute unit" being a deficiency

[armaniferrante]

It sounds like the goal is to fail the transaction if it cannot be guaranteed that an event was emitted. The indexer being able to notice after the fact is secondary as the transaction succeeding is undesirable.

The goal is not necessarily to fail the transaction, but to simply ensure that the indexer can indeed read all the logs for a program it cares about. For example, setting the log units to be effectively infinity for a particular program for an indexer's read only validator would be a solution (if that were possible).

[tlambertz]

The trick here is the allowed log units is not a part of consensus, for example RPC nodes may wish to configure this value higher than a staked validator.

Currently, the log size is for a whole TX. An attacker will always be able to put a junk IX in a TX together with his target IX. To be 100% sure not to miss events, the log size would have to be infinite, which you don't want, since then anyone spamming logs will take down all RPC nodes which use increased size. Easily could introduce DOS vulnerabilities that way. (not for Solana itself, but for the projects using the increased log sizes)

To hack around this, you'd have to patch the validator to have a per-IX limit (maybe only for the program you are interested in), but that's a bit annoying since the current LogCollector has no information about its context. You'd have to patch around in SyscallLog.

[mvines]

I do like the idea of making the log limit per-instruction!

[jarry-xiao]

This is becoming increasingly relevant to a project that I've been working on that uses emitted logs to index data off-chain. I think a log_bytes_used syscall seems pretty reasonable as that shouldn't be validator dependent and this can also solve the attack vector of a malicious instruction spamming logs before. I actually don't care if the logs get spammed after because to my knowledge the log buffer is LIFO

[askibin]

it would make sense to me if all limits behaved similarly - if you hit CU, invoke depth, stack or heap size, etc., as well as log units, your transaction should be aborted. Why would anyone want side effects to be truncated instead? Especially if the limit applies to the entire transaction, so what your instruction prints may vary depending on previous IXs.

[dnut]

log_bytes_used syscall

per-IX limit

The problem still remains: we don't have a guarantee about whether messages will be logged. But these are good ideas regardless - they are actionable and would prevent some cases of extreme abuse.

Add a syscall to get the remaining log units,

if you hit CU, invoke depth, stack or heap size, etc., as well as log units, your transaction should be aborted

the allowed log units is not a part of consensus

My understanding is that we need to be able to deterministically judge whether or not a transaction should succeed. With the current consensus mechanism, if any instruction's success is contingent upon its messages being logged, this gives RPC nodes arbitrary veto power over those instructions, which compromises consensus. This is true with either a hard limit, or even if all you do is expose the information with a syscall.

So, why not add log limits to consensus? Would one of these work?

• setting a constant value, like compute
• each leader selects a value that applies to the duration of their leadership
• a value is selected for each slot, like the clock

RPC nodes may wish to configure this value higher than a staked validator.

Could this be a way to get guaranteed logs? Run your own node, or convince an operator to log sufficient data for your purposes?

[jarry-xiao]

@armaniferrante the way I addressed the log buffer overflow problem is by CPIing the event data into a noop program and using that to index. Would be cool if anchor events had the option to do this as well. AFAIK inner instruction data is less likely to get dropped. Admittedly, it makes composability a little more constrained because you’re getting closer to the CPI stack depth.

[SphereLaboratories]

Any updates on this one? Would be great if we could guarantee fidelity for offchain data consumption.

[jarry-xiao]

It would somewhat make sense to me if the program overflows on log buffer overflows. Or at the very least, there should be a special log buffer for storing data that fails on overflow

[timaiv]

Hi! For our product (logs indexer), it makes sense to disable the logs only for foreign programs (inner instructions). Since the creation of 1 nft with edition, metadata and verified collection takes around 7kb.

Examples of too long strings:

• "Please upgrade to SPL Token 2022 for immutable owner support",
• "Verifying sized collection item",
• "Assign the account to the owning program",
• "Clean writing collection parent metadata".

So the creation of 2 nfts exceed limit.

[0xdeepmehta]

Aside from some hacks, are there any updates?