Lazy verification

[CasperN]

Discussed in https://github.com/google/flatbuffers/discussions/7125

(repost for visibility)

^{Originally posted by CasperN February 22, 2022} Our current verifiers were implemented eagerly, where we read and verify every single field/element in the buffer to the root table. We would be verifying lazily if, instead, we verified fields only when accessed. This idea came up when implementing the Rust verifier and I decided against it, in part because its harder, but also to present a similar interface to C++. That said, it might be worth thinking about this for every language even if we don't currently have the bandwidth to implement it.

Pros of lazy verification:

• Pay for what you use: Flatbuffers was originally built for fast random access, verifying every single field kind of defeats the point of flatbuffers for users who aren't going also read every single field / vector element.

Cons:

• Result types / exceptions will propagate into users' code which might be annoying
  • (though correctness oriented languages like Rust are into that stuff)
• Pay repeatedly for what you use (with checks on each field access)

[dbaileychess]

I agree, it would be nice to have a pay for what you use model.

What if this verifier is just a thin wrapper on top of the base API, that can memorize the result, so the verification only happens on the first access? That would sort of nullify the 2nd con you listed, with only a slight increase in memory.

To your first con, I think as long as this lazy verifier is opt-in, it doesn't really matter. People could always use the standard verifier if they don't want to handle exceptions/errors in code.

[bsilver8192]

I think Rust is uniquely well suited to this. I don't have a use case for it, but I think it's worth laying out a bit more about how this could work nicely in Rust for people less familiar with the language.

Propagating errors from functions that return Result is very easy. Given a function like this:

pub fn value(&self) -> Result<i32, VerificationFailed>;

you can just do this when calling it:

println!("look it's {}", something.value()? + 3);

The ? will return from the function if used on Result::Err, or return the value from Result::Ok. This only compiles if the current function returns a compatible Result type (each error type can define conversions with Into).

thiserror is a common library for making error types that roll up errors from various causes. Errors written this way are usable without an allocator, and make it easy to extract the root causes. anyhow is a common way to type-erase errors (aka put the concrete type in a malloced buffer).

[github-actions[bot]]

This issue is stale because it has been open 6 months with no activity. Please comment or label not-stale, or this will be closed in 14 days.

[github-actions[bot]]

This issue was automatically closed due to no activity for 6 months plus the 14 day notice period.