go icon indicating copy to clipboard operation
go copied to clipboard

Ingestion Filtering, advanced rule, include activity of assets issued by configured accounts

Open sreuland opened this issue 2 years ago • 0 comments

What problem does your feature solve?

Capturing asset related tx/ops data during filtering which is normally one-hop away and not dynamically captured during normal account whitelist rule processing.

What would you like to see?

As a service provider with my own Horizon installation, I want to configure a new filter rule to trigger tracking transactions and operations related to any assets issued by specific Accounts.

I want to configure which account ids trigger the ingestion filter of my Horizon installation to retain transaction and operational info related to any assets issued by the account.

I don't want my horizon historical database to contain more data related to other assets not related, so, I expect to see my horizon db size on disk still remain relatively small with a slower growth rate compared to no-filtering(i.e. data for entire ledger was saved)

Acceptance Criteria:

  • Horizon provides additional endpoint support on the Filter Configuration API on the admin port to perform CRUD on the 'issuer account' filter rule model to allow defining the whitelisted account ids that filter checks to determine whether to include tx's/ops related to any issued assets of the whitelisted account.
  • Design the config rule model, it could be a new rule such as asset_filter_by_issuer_accounts that uses the standard whitelist model of a list of strings(account ids), or it could be a new model such as a flag account_filter_traverse_issued_assets=[true|false], and the filter applies the flag in conjunction with existing account_filter_by_whitelist=["acct1","acct2",...] rule from https://github.com/stellar/go/issues/4221 .
  • All Horizon ingestion processes(current-forward/reingest) will apply the same filtering that applies this rule when persisting historical data.
  • The filtering on this new rule, effectively compiles the state of assets issued from a whitelisted account from genesis up to present ledger, and detects when an asset is issued from a whitelisted account, at which point the filtering effectively includes the asset in it's internal compiled 'issued assets whitelist' ruleset for filtering to apply.
  • This advanced filter feature should be provided as early/alpha drop builds to some willing early-adopter partners for trial usage.
  • This advanced filter feature is layered on top of existing filter infrastructure established on prior #4221 and #4222.

Pre-requisites:

This feature is dependent on #4221 and #4222 being implemented first to establish the initial filtration framework within ingestion.

sreuland avatar Feb 09 '22 23:02 sreuland