Sync worker reliability

[neekolas]

tl;dr

• Adds the concept of retryable and non-retryable errors to the SyncWorker
• Make the SyncWorker retry retryable errors
• Adds some tests to the SyncWorker
• Refactors some of our test helpers to make mocking the SyncWorker easier

Summary by CodeRabbit

• New Features

  • Introduced a new package for mock generation, enhancing testing capabilities.
  • Added a new field for target rate per minute in the fee structure, improving fee calculation functionalities.
  • Created functions for node and mock registry creation, facilitating easier testing setups.

• Bug Fixes

  • Enhanced processing reliability by implementing an exponential backoff retry mechanism for envelope handling, improving error management and system stability.

• Development Environment

  • Enabled language server support and specified formatting preferences for Go code in the development environment settings.

[neekolas]

• #694 👈 (View in Graphite)
• main

---

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

• Queue - adds this PR to the back of the merge queue
• Hotfix - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[macroscopeapp[bot]]

Extract originatorStream component from syncWorker to improve sync worker reliability

• Extracts originatorStream functionality into a dedicated component in originatorStream.go with methods for stream handling, envelope validation, fee calculation, and storage
• Refactors syncWorker in syncWorker.go to delegate stream handling to the new component
• Adds unit tests for the originatorStream component in originatorStream_test.go
• Introduces registry test utilities in registry.go for creating test nodes and mock registries
• Adds mock generation configuration for ReplicationApi_SubscribeEnvelopesClient interface

📍Where to Start

Start with the newOriginatorStream constructor and listen method in originatorStream.go, which contains the core stream handling logic extracted from the sync worker.

---

Macroscope summarized a7d3c8f.

[coderabbitai[bot]]

[!WARNING]

Rate limit exceeded

@neekolas has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 11 minutes and 52 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Reviewing files that changed from the base of the PR and between b1b14b203b0d9ba84e268c5dbc0c75fe5ba0a8aa and a7d3c8f61d8d7ed9043756c292b626fd7643be7d.

📒 Files selected for processing (8)

• .mockery.yaml (1 hunks)
• pkg/mocks/message_api/mock_ReplicationApi_SubscribeEnvelopesClient.go (1 hunks)
• pkg/server/server_test.go (3 hunks)
• pkg/sync/originatorStream.go (1 hunks)
• pkg/sync/originatorStream_test.go (1 hunks)
• pkg/sync/syncWorker.go (2 hunks)
• pkg/testutils/fees/rates.go (1 hunks)
• pkg/testutils/registry/registry.go (1 hunks)

## Walkthrough
This pull request introduces several updates across the codebase. A new interface entry is added in the mock configuration and its corresponding mock implementation is provided for the Replication API. Test code is refactored to use helper functions for node and registry creation. In addition, the envelope processing in the sync worker is enhanced with an exponential backoff retry mechanism and updated error handling. New tests for the sync worker have been added, and test utilities for fee rates and registry operations have been expanded.

## Changes

| File(s) | Change Summary |
|---------|----------------|
| `.mockery.yaml`<br>`pkg/mocks/message_api/mock_ReplicationApi_SubscribeEnvelopesClient.go` | Added a new package entry for the Replication API interface and implemented its corresponding mock with all required methods and helper functions. |
| `pkg/server/server_test.go` | Refactored test setup by replacing manual node initialization with utility functions (`CreateNode` and `CreateMockRegistry`) for cleaner registry mocking. |
| `pkg/sync/syncWorker.go`<br>`pkg/sync/syncWorker_test.go` | Introduced an exponential backoff retry mechanism in envelope processing, refined error handling (distinguishing transient from permanent errors), and added corresponding unit tests for various error scenarios. |
| `pkg/testutils/fees/rates.go` | Added a new `TargetRatePerMinute` field to test fee rates and introduced the `NewTestFeeCalculator` function to facilitate fee calculator creation based on these rates. |
| `pkg/testutils/registry/registry.go` | Added new helper functions (`CreateNode` and `CreateMockRegistry`) to simplify the creation of node instances and mock registries for testing purposes. |
| `.vscode/settings.json` | Moved the `"go.useLanguageServer": true` setting to a different position in the file without changing its value or adding new settings. |

## Suggested reviewers
- mkysel

✨ Finishing Touches

• [ ] 📝 Generate Docstrings

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[mkysel]

@neekolas can you give me a bit of context around this? What prompted this changeset. Did you see something broken? How does this fix the issue.

[neekolas]

I was working on the misbehavior detection, and realized that any database error when saving the message will throw the message out permanently (bad on its own)...and can make it seem like there is a gap in sequence_id when in fact there isn't.

[graphite-app[bot]]

Merge activity

• May 13, 2:18 PM PDT: neekolas added this pull request to the Graphite merge queue.
• May 13, 2:18 PM PDT: CI is running for this pull request on a draft pull request (#792) due to your merge queue CI optimization settings.
• May 13, 2:19 PM PDT: Merged by the Graphite merge queue via draft PR: #792.