feat: added packet count telemetry metrics

[JoblersTune]

Changes proposed in this pull request

• Added packet counting metrics for telemetry v2
• Updated the transactions_total metric to packet_amount_fulfill and included source metadata with the amount metric collection.
• Moved the collection of all packet data over to the receiving node so that it can log a consistent match of incoming prepares to outgoing responses. This way, since telemetry is optional, we'll get a coherent reporting structure since one node can send coherent results in isolation.
• Moved packet count and amount metrics out of middleware (since metrics should only be collected based on finalised values after all other middleware has run)
• Cleaned up old telemetry code that is no longer needed after changes were made

Outside of this PR but related:

• Changes to the telemetry dashboard to be reviewed as well. I've created a test dashboard called TEST packet count which is viewable on Grafana.

Context

Working towards expanding telemetry for telemetry v2

• fixes #2734

TODO:

• add tests
• Update docs
• Consider transaction count being on sending node - planning to track on outgoing and incoming

Checklist

• [x] Related issues linked using fixes #number
• [ ] Tests added/updated
• [ ] Documentation added
• [ ] Make sure that all checks pass
• [ ] Bruno collection updated

[netlify[bot]]

✅ Deploy Preview for brilliant-pasca-3e80ec ready!

Name	Link
🔨 Latest commit	1c29f9d9e4273facaf51f4989f9c45ac7715e561
🔍 Latest deploy log	https://app.netlify.com/sites/brilliant-pasca-3e80ec/deploys/66c82350308bd900086e7dd7
😎 Deploy Preview	https://deploy-preview-2797--brilliant-pasca-3e80ec.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[JoblersTune]

Last outstanding task is updating the tests

[JoblersTune]

Are there any lifecycle tests for outgoing payments somewhere? I wanted to add in telemetry tests for the outgoing payment count but couldn't see where to put them?

[JoblersTune]

Please don't forget to look at the new dashboard additions as well please: https://rafikitelemetry.grafana.net/d/cdq2hn0w6frpcr/test-packet-count?orgId=1

[BlairCurrey]

Regarding the Packet count (by source) viz, should we either:

• use the prepare amount for the total OR
• sum the fulfill/reject amounts for the total

I think probably just the prepare amount? Currently we are summing fulfill/reject/prepare but the prepare include fulfill and reject already it looks like. So I think we can just remove the last one here (and related graphs):

[BlairCurrey]

How will the existing visualizaitons be impacted? It looks like maybe the new Packet count (by source) replace the Transaction Count (by source) ones but then the Value Sent Through network line graphs will be updated to use the new (and Total Transactions Per Second would need the same)? I like the new stat visualizations but think the existing line graphs are still useful.

[JoblersTune]

Regarding the Packet count (by source) viz, should we either:

• use the prepare amount for the total OR
• sum the fulfill/reject amounts for the total

I think probably just the prepare amount? Currently we are summing fulfill/reject/prepare but the prepare include fulfill and reject already it looks like. So I think we can just remove the last one here (and related graphs):

I don't have strong feelings about whichever approach.

I know Max was initially hoping we'd collect this to get a handle on packet loss through the network but this tricky when telemetry is optional.

Given that I assumed we would still be most interested in 1. the total actual number of packets moving through the network (prepare + reject + fulfill) and 2. the relationship between them (we expect prepare = fulfill + reject) and maybe good to see how many rejects we get vs fulfills as well. Of course some of the rejects are quote based as well.

I guess I'm not exactly sure.

Maybe we can just graph fulfill and reject and leave the prepare and totals out?

[JoblersTune]

How will the existing visualizaitons be impacted? It looks like maybe the new Packet count (by source) replace the Transaction Count (by source) ones but then the Value Sent Through network line graphs will be updated to use the new (and Total Transactions Per Second would need the same)? I like the new stat visualizations but think the existing line graphs are still useful.

Good question. This felt obvious when I started but less so now. I think we need a call about this. I'm gonna formulate some ideas and then let's rather discuss.

[BlairCurrey]

I don't have strong feelings about whichever approach.

I know Max was initially hoping we'd collect this to get a handle on packet loss through the network but this tricky when telemetry is optional.

Given that I assumed we would still be most interested in 1. the total actual number of packets moving through the network (prepare + reject + fulfill) and 2. the relationship between them (we expect prepare = fulfill + reject) and maybe good to see how many rejects we get vs fulfills as well. Of course some of the rejects are quote based as well.

I guess I'm not exactly sure.

Maybe we can just graph fulfill and reject and leave the prepare and totals out?

Given that I assumed we would still be most interested in 1. the total actual number of packets moving through the network (prepare + reject + fulfill) and 2.

Might need to take a step back and verify something... is this the correct way to count packets in ilp-packet.ts?

• always incrementing preprare AND
  • maybe incrementing fulfill OR
  • maybe incrementing reject

So prepare is actually equal to the total. But based off your comment it sounds like prepare/fulfill/reject should be mutually exclusive. Is that how it should be? I guess that would make sense. My original comment was taking the logic there at face value.

And just to elaborate on how its currently working, look at the graph. fulfill (yellow) reject (blue) almost equal the prepare (green). The difference is the loss. So that's everything we want. The total (orange) then sums these, which is equivalent to fulfill + reject + loss (the prepare) + fulfill + reject. Hence the double counting and recommendation to remove the Total (orange). But maybe we shouldnt be incrementing the prepare counter for every packet?

[JoblersTune]

@BlairCurrey I take your point.

... based off your comment it sounds like prepare/fulfill/reject should be mutually exclusive.

This came to be because if you look at the issue, the initial suggestion was to collect prepare packets at the receiver node, and fulfill/reject packets at the sending node. But, because telemetry is optional, I pointed out that we might get a distorted view of the situation. Because if only one of the nodes involved has enabled telemetry we could see a whole bunch of prepares but no responses or vice versa.

This is why I suggested collecting ALL the packet counts on the same node (the receiving node). However, maybe this makes collecting prepare packets redundant... Since we only send a response if we get a prepare, and I'm assuming we ALWAYS send a response when we get a prepare... Is that true?

In this case, maybe

1. Collecting prepare packet counts actually adds no value and we should drop it
2. We should collect prepare packet amounts because they are still part of the total number of packets flowing on the network (prepare + fulfill + reject = total)
3. We go back to the idea of collecting the number of prepare packets on the sending node and accept that these metrics (prepares vs. responses) will possibly be misaligned...

I feel this always goes back to asking what the purpose of these metrics is. Do we want to have a number for marketing that says x number of packets were sent over the Interledger network in the last year. Or do we want to estimate packet loss over the network - which is in a pure sense, impossible to do because telemetry is optional. But maybe having the asymmetric view still adds some other kind of value?

But I still want to make sure we're in alignment that the total number of ILP packets that traverse the Interledger network = prepare + reject + fulfill. Because these are all packets, they are all distinct packet types, and they all travel over the network.

[BlairCurrey]

Is this how we intend to count? Matches my understanding from our previous call but wanted to verify.

This is how its getting counted from my local testing.

sequenceDiagram
    Title: Happy Path
    participant A
    participant B

    note left of A: prepare=0
    note left of A: fulfill=0
    A->>A: prepare++
    A->>B: Send prepare.
    note right of A: prepare=1
    note right of A: fulfill=0
    B->>A: Send fulfill
    note right of A: prepare=1
    note right of A: fulfill=0

    A->>A: fulfill++
    note left of A: prepare=1
    note left of A: fulfill=1

[BlairCurrey]

And then I guess this would be multi hop:

sequenceDiagram
    Title: Happy Path Multi-Hop
    participant A
    participant B
    participant C

    note left of A: prepare=0
    note left of A: fulfill=0
    A->>A: prepare++  
    A->>B: Send prepare.
    note right of A: prepare=1
    note right of A: fulfill=0

    B->>B: prepare++
    B->>C: Send prepare
    note right of B: prepare=2
    note right of B: fulfill=0

    C->>B: Send fulfill
    note right of B: prepare=2
    note right of B: fulfill=0

    B->>B: fulfill++
    B->>A: Send fulfill
    note right of A: prepare=2
    note right of A: fulfill=1

    A->>A: fulfill++
    note left of A: prepare=2
    note left of A: fulfill=2

[BlairCurrey]

Max mentioned this for the loss which I think still seems right?

Number of packets - (2 * (rejected + fulfilled))

Which is just prepare - reject - fulfill. (Number of packets = prepare + fulfill + reject. Expand the terms then add). Makes sense I think - each prepare should have a fulfill or reject.

[JoblersTune]

Thanks for the diagrams @BlairCurrey they match my understanding as well.

[mkurapov]

@JoblersTune just need to fix the build and good to go

[JoblersTune]

@melissahenderson

This PR has been waiting a while, so I nearly forgot there was a doc change here too.

I'll just paste the text here for your reference and remove the file changes in this PR (telemetry overview page)

---
title: Overview
---

## Purpose

The objective of the telemetry feature is to gather metrics and establish an infrastructure for visualizing valuable network insights. The metrics we at the Interledger Foundation collect include:

- The total amount of money transferred via packet data within a specified time frame (daily, weekly, monthly).
- The number of transactions that have been at least partially successful.
- The number of ILP packets flowing through the network.
- The average amount of money held within the network per transaction.
- The average time it takes for an outgoing payment to complete

We aim to track the growth of the network in terms of transaction sizes and the number of transactions processed. Our goal is to use these data for our own insights and to enable [Account Servicing Entities](/reference/glossary#account-servicing-entity) (ASEs) to gain their own insights.

## Privacy and Optionality

Privacy is a paramount concern for us. Rafiki's telemetry feature is designed to provide valuable network insights without violating privacy or aiding malicious ASEs. For more information, please [read the privacy docs](/telemetry/privacy).

The telemetry functionality is currently enabled by default on test (non-livenet) environments, i.e. any environment that is not dealing with real money. When active, it transmits metrics to the "testnet" collector. In the future, those ASEs operating in a production "livenet" environment (real money) will be able to opt-in to sharing their metrics with a "livenet" collector. Regardless of environment, Account Servicing Entities (ASEs) can also opt-out of telemetry completely.

## Architecture

The architecture of the telemetry feature is illustrated below:

![Telemetry architecture](/img/telemetry-architecture.png)

## OpenTelemetry

We have adopted [OpenTelemetry](https://opentelemetry.io/) to ensure compliance with a standardized framework that is compatible with a variety of tool suites. This allows clients to use their preferred tools for data analysis, while Rafiki is instrumented and observable through a standardized metrics format.

## Telemetry ECS Cluster

The Telemetry Replica service is hosted on AWS ECS Fargate and is configured for availability and load balancing of custom ADOT (AWS Distro for Opentelemetry) Collector ECS tasks.

When ASEs opt for telemetry, metrics are sent to our Telemetry Service. To enable ASEs to build their own telemetry solutions, instrumented Rafiki can send data to multiple endpoints. This allows the integration of a local [Otel collector](https://opentelemetry.io/docs/collector/) container that can support custom requirements. Metrics communication is facilitated through [gRPC](https://grpc.io/).

## Otel SDK - Rafiki Instrumentation

The Opentelemetry SDK is integrated into Rafiki to create, collect, and export metrics. The SDK integrates seamlessly with the OTEL Collector.

## Prometheus - AMP

We use Amazon's managed service for Prometheus (AMP) to collect data from the Telemetry cluster.

**Note**: AMP offers limited configuration options and cannot crawl data outside of AWS. This limitation led us to adopt a push model, using prometheusRemoteWrite, instead of a pull model. For future development, we may consider hosting our own Prometheus.

## Grafana - Grafana Cloud

Grafana Cloud is used for data visualization dashboards. It offers multiple tools that extend Prometheus Promql.

**Note**: We initially used Amazon hosted Grafana, but it did not meet our needs for embedding dashboards. Grafana Cloud offers a feature called “Public dashboards”, which allows us to share dashboards. However, embedding may still pose a challenge.

## Exchange Rates

For telemetry purposes, all amounts collected by instrumented Rafiki should be converted to a base currency.

**Privacy Reasoning**: If only two ASEs are peered over a non-USD currency and we collect data in that currency, it would be easy to determine the volumes moved between those two ASEs. To maintain privacy, we convert all amounts to a base currency.

If an ASE does not provide the necessary exchange rate for a transaction, the telemetry solution will still convert the amount to the base currency using external exchange rates. A Lambda function on AWS retrieves and stores these external exchange rates. It is triggered by a daily CloudWatch event and stores the rates in a public S3 Bucket. The S3 Bucket does not have versioning, and the data is overwritten daily to further ensure privacy.

## Instrumentation

Data points for all metrics (e.g. counter increases) are exported to collection endpoints at a configurable interval (default recommended to 15s).

Currently collected metrics:

- `transactions_total` - Counter metric
  - Description: “Count of funded outgoing transactions”
  - This counter metric increases by 1 for each successfully funded outgoing payment resource.
- `packet_count_prepare` - Counter metric
  - Description: “Count of prepare packets that are sent”
  - This counter metric increases by 1 for each prepare packet that is sent.
- `packet_count_fulfill` - Counter metric
  - Description: “Count of fulfill packets”
  - This counter metric increases by 1 for each fulfill packet that is received.
- `packet_count_reject` - Counter metric
  - Description: “Count of reject packets”
  - This counter metric increases by 1 for each reject packet that is received.
- `packet_amount_fulfill` - Counter metric
  - Description: “Amount sent through the network”
  - This amount metric increases by the amount sent in each ILP packet.
- `transaction_fee_amounts` - Counter metric
  - Description: “Fee amount sent through the network”.
  - This fee amount metric increases by the (amount sent minus amount received) for an outgoing payment.
- `ilp_pay_time_ms` - Histogram metric
  - Description: “Time to complete an outgoing ILP payment”
  - This histogram metric records the time taken to make an ILP payment.

**Note**: The current implementation only collects metrics on the SENDING side of a transaction. Metrics for external open-payments transactions RECEIVED by a Rafiki instance in the network are not collected.

[melissahenderson]

@JoblersTune Thank you! I'll get the updates worked in to our other branch.