Live indexing "only" flag

[jaggad]

Add Live indexing "only" flag

We need to be able to discriminate on event batches that are from live indexing, or those from historical indexing. Say there is a use-case where we want to index native trasnfers and drive notifications from it.

Well we very likely do not want to notify on events which are historical. It would be better to only notify on truly live events, or drop them if too old.

Whilst we could handle this at the notification consumer side (assuming there is a queue/streaming solution) it would be more cost and resource efficient to simply not push every single historical event

See example below

pub async fn register_native_transfer_event(
    manifest_path: &PathBuf,
    ctx: Arc<IndexerContext>,
    registry: &mut TraceCallbackRegistry,
) {
    EvmTracesEventType::NativeTransfer(
        NativeTransferEvent::handler(
            |results, context| async move {
                // Example only... could be a method on a newtype wrapping the vec, could be a separate field in the closure params, etc
                if results.is_live_indexed() {
                    // Push to queue
                }

                process_events(
                    results,
                    context.extensions.clone(),
                    config,
                    |t, network, timestamp| {
                        Ok(vec![/* ... */])
                    },
                )
                .await
                .map_err(chain_str_error)
            },
            ctx.to_extensions(),
        )
        .await,
    )
    .register(manifest_path, registry)
    .await;
}

Ideally this should be done on the whole batch of results, to prevent us from having to map over each individual Tx and determine if it is live. This should also fit well with the rindexer model and separate indexing flows for historical vs live.

No code

It may also be desirable for this to be available in the no-code solution.

I would consider an option live_indexed_only: bool either on the "streams" or specific "stream" (i.e sns) option itself to control this

streams:
  sns:
    live_indexed_only: # true <------ New option, defaults to `false`, can be used to only index to this stream live events
    aws_config:
      region: {{ .Values.globals.aws_region }}
      access_key: ${AWS_ACCESS_KEY_ID}
      secret_key: ${AWS_SECRET_ACCESS_KEY}
      endpoint_url: ${AWS_ENDPOINT_URL}
    topics:
      - topic_arn: arn:aws:sns:{{ .Values.globals.aws_region }}:{{ .Values.globals.aws_account_id }}:{{ .Values.metadata.env }}-{{ .Values.globals.aws_region }}-deposit.fifo
        networks:
          - ethereum
        events:
          - event_name: Deposit
      - topic_arn: arn:aws:sns:{{ .Values.globals.aws_region }}:{{ .Values.globals.aws_account_id }}:{{ .Values.metadata.env }}-{{ .Values.globals.aws_region }}-withdrawal.fifo
        networks:
          - ethereum
        events:
          - event_name: Withdrawal

[joshstevens19]

this all makes sense to me completely - interface is nice and good spec