Replication tracking task

[richardhuaaa]

### Deployment and metrics
- [ ] https://github.com/ephemeraHQ/infrastructure/issues/389
- [ ] https://github.com/ephemeraHQ/infrastructure/issues/388
- [ ] https://github.com/xmtp/xmtpd/issues/137
- [ ] https://github.com/xmtp/xmtpd/issues/138
- [ ] https://github.com/xmtp/xmtpd/issues/141
- [ ] https://github.com/xmtp/xmtpd/issues/140
- [ ] https://github.com/xmtp/xmtpd/issues/139
- [ ] https://github.com/xmtp/xmtpd/issues/149
- [ ] https://github.com/xmtp/xmtpd/issues/151
- [ ] https://github.com/xmtp/xmtpd/issues/152
- [ ] Individual node E2E Testing / Probing
- [ ] Implement Rate Limiting
- [ ] Split a staging environment from prod and set up CI/CD accordingly
- [ ] https://github.com/xmtp/xmtpd/issues/191
- [ ] Partners: Write publicly available TF modules
- [ ] Partners: Write publicly available helm charts
- [ ] Collect metrics/tracing from other nodes for distributed tracing
- [ ] Spike: Migrate existing data to the new system?
- [ ] Alerting, Paging, Rotation, etc
- [ ] DDoS prevention, IP blacklisting, ...
- [ ] Database backup&recovery, snapshotted database for new nodes
- [ ] Data aging/expiry
- [ ] Spike: node catchup rate limit (prevent new nodes from slowing down the system)

### Blockchain
- [x] Utilities to publish GroupMessages to blockchain
- [ ] https://github.com/xmtp/xmtpd/issues/135
- [ ] https://github.com/xmtp/xmtpd/issues/136
- [ ] https://github.com/xmtp/xmtpd/issues/144
- [ ] https://github.com/xmtp/xmtpd/issues/155
- [x] CLI command for running administrative functions against deployed Nodes contract
- [ ] https://github.com/xmtp/xmtpd/issues/168
- [ ] Utilities to publish IdentityUpdates to blockchain
- [ ] Add metrics for indexing GroupMessages from blockchain
- [ ] Add metrics for indexing IdentityUpdates from blockchain
- [ ] Performance enhancements for chain indexing
- [ ] Investigate combining `getLogs` and log subscription RPCs
- [ ] Load test blockchain writes to understand real world TPS and max transaction size
- [ ] Detect and deal with blockchain forks
- [ ] E2E tests for modifying nodes contract and seeing results in the NodeRegistry

### Publish
- [ ] https://github.com/xmtp/xmtpd/issues/127
- [ ] Topic validation (validate payloads are published to the correct topic)
- [ ] Key package validation
- [ ] Add logic to determine if a given write should go to blockchain or broadcast network
- [ ] Causal ordering validation
- [ ] Identity update validation and computed data
- [ ] Payer validation
- [ ] Endpoint for publishing misbehavior report (pre-defined topic?)
- [ ] Create Payer API

### Query and Subscribe
- [ ] https://github.com/xmtp/xmtpd/issues/124
- [ ] https://github.com/xmtp/xmtpd/issues/132
- [ ] https://github.com/xmtp/xmtpd/issues/125
- [ ] https://github.com/xmtp/xmtpd/issues/167
- [ ] https://github.com/xmtp/xmtpd/issues/126
- [ ] JWT authorization for requests from other nodes
- [ ] Subscribe to misbehavior reports (pre-defined topic?)
- [ ] Node registry subgraph endpoint

### Sync
- [ ] https://github.com/xmtp/xmtpd/issues/181
- [ ] Failover when an originator becomes unresponsive, and return to originator once responsive
- [ ] Handling the case when the blockchain stream is unresponsive
- [ ] Respond to changes in the node registry
- [ ] Validation of originator payloads (sid, timestamp, signature), with misbehavior reports
- [ ] Set JWT for requests to other nodes
- [ ] Write a node2node test framework
- [ ] Spike: node2node versioning, upgrades, and version coexistance
- [ ] Spike: Node offboarding
- [ ] Spike: New node onboarding? Can we backup ship?

### Client
- [ ] https://github.com/xmtp/xmtpd/issues/150
- [ ] https://github.com/xmtp/xmtpd/issues/169
- [ ] Query node registry subgraph to get list of nodes, with ability to override it
- [ ] Implement preferred node algorithm, with ability to override it
- [ ] Implement node switching for unavailable nodes (likely on retries)
- [ ] Deduplicate payloads published to multiple originators (as well as when switching gateways)
- [ ] Track latest SIDs on receive, and set the vector clock on send
- [ ] Setting and validating authenticated data
- [ ] Implement retries for dependency errors on send
- [ ] Implement causal ordering algorithm and retries on receive
- [ ] (Optional) Simplify subscription logic (as subscriptions are reliable)
- [ ] (Optional) Multi-node publishes/queries for trust purposes
- [ ] (Optional) Stream identity updates

### Documentation and Training
- [ ] Finish XIP
- [ ] Rudimentary system documentation
- [ ] Emergency response guides
- [ ] Partner guides
- [ ] https://github.com/xmtp/xmtpd/issues/187