Update Persisted LSN reporting

[pcholakov]

Implement a more efficient reporting mechanism for reporting Persisted LSNs. Trimming will benefit from knowing much more precisely when data is flushed to disk. This should eliminate the need for manual flushing performed by PersistedLogLsnWatchdog.

Related: #3044. Depending on how we implement this, it might automatically resolve that issue.

• [ ] Implement EventListener support in RocksDB FFI
• [ ] Implement EventListener support in Rust RocksDB bindings
• [ ] Implement TablePropertiesCollector support in RocksDB FFI
• [ ] Implement TablePropertiesCollector support in Rust RocksDB bindings
• [ ] Update PartitionStore persisted LSN reporting in Restate

Design sketch

We will combine two separate RocksDB features to efficiently track persisted LSNs:

• A custom TablePropertiesListener to track the highest applied LSN FSM variable key per partition, for a given SST file
• An EventListener is notified of OnFlushCompleted events; this uses the TableProperties provided in the event to efficiently retrieve the maximum applied LSN which has been flushed to disk

sequenceDiagram
  participant PS as PartitionStore
  participant RDB as RocksDB

  Note over PS, RDB: Setup
  create participant EL as EventListener
  PS ->> EL: Create
  PS ->> RDB: Add EventListener

  create participant TPCF as TablePropertiesCollectorFactory
  PS ->> TPCF: Create
  PS ->> RDB: Add TablePropertiesCollectorFactory

  PS ->> RDB: Get Applied LSN
  PS ->> PS: Initial Persisted LSN

  Note over PS, RDB: Normal operation

  par Write path
    loop
        PS ->> RDB: WriteBatch
        RDB ->> RDB: Commit
    end
  
  and Async flush
    RDB ->> RDB: Write SST
    RDB ->> TPCF: Create collector
    activate RDB
    create participant TPL as TablePropertiesCollector
    TPCF ->> TPL: New
    loop For each key-value pair in file
        RDB ->> TPL: Add(key, value)
        TPL ->> TPL: If key is applied LSN, track max value
    end
    RDB ->> TPL: Finish() - finalize properties
    TPL ->> TPL: Add custom properties with max LSN info
    TPL -->> RDB: User properties
    destroy TPL
    RDB ->> TPL: Destroy
    create participant SST as SST File
    RDB ->> SST: Write SST file with properties

    RDB ->> EL: OnFlushCompleted event    
    Note right of EL: Event includes TableProperties for new SST
    EL ->> EL: Extract max applied LSN from properties
    EL -) PS: Update Persisted LSN
    
    deactivate RDB
  end

[pcholakov]

First set of PRs is ready for review.

EventListener:

• FFI support: https://github.com/restatedev/rocksdb/pull/7
• Rust bindings: https://github.com/restatedev/rust-rocksdb/pull/8

TablePropertiesCollector

• FFI support: https://github.com/restatedev/rocksdb/pull/8
• Rust bindings: https://github.com/restatedev/rust-rocksdb/pull/9