Fetch AO messages

[Ekzer]

Hi guys, I spent the past hours to try to run ar-io node to index AO messages, but unsuccessful still. I just need a read-only endpoint. I tried with these env variables :

ANS104_INDEX_FILTER='{"tags":[{"name":"Bundler-App-Name", "value":"AO"}]}'
ANS104_UNBUNDLE_FILTER='{"tags":[{"name":"Bundler-App-Name", "value":"AO"}]}'

I also tried with always:true but doesn't work still.

Unless I misunderstood the doc, would you mind help me please? Thank you in advance

[djwhitt]

Hi @Ekzer,

These are the filters you want:

ANS104_UNBUNDLE_FILTER={ "or": [ { "tags": [ { "name": "Bundler-App-Name", "value": "AO" } ] }, { "isNestedBundle": true } ] }
ANS104_INDEX_FILTER={"always": true}

The unbundle filter tells the node to unbundle all AO bundles and nested bundles in them. The indexing filter tells the node to index all the data items in those bundles.

That said, if you run a node with this filter you will be attempting to index a lot of bundles and data items (you can get feel for the volume by looking at the stats on ao.link). I think it's likely still doable on a single node, but it's not easy. We have plans to support parallel indexing across multiple nodes in the not too distant future though, so keep an eye on the change log. Let us know how things go as you continue to test too.

[Ekzer]

Hi @djwhitt Thank you very much for your quick answer. So I tried again but it is unsuccessful :( I try to run it on my Mac first (M2 MAX, 32gb), before deploying to a server. Here is my docker-compose and .env :

services:
  io-core:
    image: ar-io-core:latest
    restart: unless-stopped
    ports:
      - "4000:4000"
    volumes:
      - ./ar-io-data:/app/data
    environment:
    - ANS104_UNBUNDLE_FILTER=${ANS104_UNBUNDLE_FILTER:-}
    - ANS104_INDEX_FILTER=${ANS104_INDEX_FILTER:-}
    - ANS104_UNBUNDLE_WORKERS=${ANS104_UNBUNDLE_WORKERS:-}
    - START_HEIGHT=${START_HEIGHT:-}
    - TRUSTED_GATEWAY_URL=https://${TRUSTED_GATEWAY_HOST:-arweave.net}
    - START_WRITERS=${START_WRITERS:-true}
    - LOG_LEVEL=${LOG_LEVEL:-debug}
 

.env

ANS104_UNBUNDLE_FILTER='{ "or": [ { "tags": [ { "name": "Bundler-App-Name", "value": "AO" } ] }, { "isNestedBundle": true } ] }'
ANS104_INDEX_FILTER='{"always":true}'
START_HEIGHT=1603940
START_WRITERS=true
LOG_LEVEL=debug
ANS104_UNBUNDLE_WORKERS=4

For some reasons, I can't see the unbundling process nor indexing on work in logs.

[djwhitt]

I'm pretty swamped for the next couple days, but I'll test this myself in the not too distant future. We were running a node with a very similar configuration for a long time though and it was successfully indexing AO data items. You are likely already doing this, but make sure you're recreating containers by doing a down and up when changing environment variables. You can always docker inspect to sanity check them too.

[djwhitt]

@Ekzer I thought of another thing you should try adjusting:

BACKGROUND_RETRIEVAL_ORDER=trusted-gateways

That will set the default for unbundling to the trusted gateway. That makes bundle data retrieval less verifiable, but it should improve download performance.

[Ekzer]

@djwhitt I will try this out. Thank you for your support. To let you know my end goal : I want to query these messages through the GraphQL interface, thanks to the tags Data-Protocol=ao, Action=Transfer, etc... Ideally, I would like to only index the messages with Action=Transfer. I will keep trying hard and come back with results

[Ekzer]

Hi @djwhitt Great news ! I was able to index AO messages eventually 😃 thanks to your configuration! However, I realized there is a big problem still. For some unknown reasons, only the gateway "goldsky" is able to sync AO messages with Action=Transfer (token transfers) in real time. How is that possible ? It seems like they are not bundled by ArDriveTurbo nor AO Authority, hence nothing was found and indexed.

[djwhitt]

@Ekzer Excellent! I'm glad that worked.

For some unknown reasons, only the gateway "goldsky" is able to sync AO messages with Action=Transfer (token transfers) in real time.

This is because Arweave itself doesn't provide any realtime messaging capabilities. Turbo sends AO data items directly to Goldsky before they're settled on Arweave. That said you can improve latency compared to the default configuration by setting ENABLE_MEMPOOL_WATCHER=true. That will attempt to index bundles as soon as they appear in the mempool. Please let us know how it goes if you try. That functionality is still somewhat experimental.

[xinzhongyoumeng]

@Ekzer太棒了！我很高兴这个方法奏效了。

由于某些未知的原因，只有网关“goldsky”能够实时同步 Action=Transfer（令牌传输）的 AO 消息。

这是因为 Arweave 本身不提供任何实时消息传递功能。Turbo 在 AO 数据项结算到 Arweave 之前将其直接发送到 Goldsky。也就是说，您可以通过设置来改善与默认配置相比的延迟ENABLE_MEMPOOL_WATCHER=true。这将尝试在捆绑包出现在内存池中时立即对其进行索引。如果您尝试，请告诉我们进展如何。该功能仍处于实验阶段。

I also meet same problem. I run a container with below command:

docker run -it -u root --name ario -d -e ENABLE_MEMPOOL_WATCHER=true -e BACKGROUND_RETRIEVAL_ORDER='trusted-gateways' -e START_WRITERS=true -e ANS104_UNBUNDLE_WORKERS=16 -e SKIP_CACHE=false -e BACKFILL_BUNDLE_RECORDS=true -e FILTER_CHANGE_REPROCESS=true -e TRUSTED_NODE_URL="https://arweave.net" -e TRUSTED_GATEWAY_URL="https://arweave-search.goldsky.com" -e START_HEIGHT=1620980 -e ANS104_INDEX_FILTER='{ "always": true }' -e ANS104_UNBUNDLE_FILTER='{"or":[{"tags":[{"name":"Data-Protocol","value":"ao"},{"name": "Variant","value":"ao.TN.1"}]},{"isNestedBundle":true}]}' -v /mnt/data/:/app/data ghcr.io/ar-io/ar-io-core:0bb7270a43b41a5fa8647b699b4d72abafdc4c4a

I can't get transaction real time. such as:

curl 'http://172.17.0.13:4000/graphql' -H 'Accept-Encoding: text' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'Connection: keep-alive' --data-binary '{"query":"       {        transaction(id:\"KGK09rjgqLtdBUyVKx8cIJ-Bzu3jM2EqqWI1AtkJPX8\") {            id            owner {                address                key            }            recipient            fee {                winston                ar            }            tags {                name                value            }            quantity {                winston                ar            }            block {                height                timestamp                id            }        }        }  "}'
{"data":{"transaction":null}}