[PAY-737] Backfill audio_tx_hist time desc order

[dharit-tan]

Description

Rewrote backfiller code to traverse txs in time descending order (opposite of how they usually are). This greatly simplifies the logic and improves performance from O(n^2) to O(n).

Note - lots of duplicate logic, would like to refactor at least some of the fns like check_if_backfilling_complete.

Tests

Testing on remote-dev with 3 DNs and prod sandbox.

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

• Added lots of logging.
• Checks in get_health.py that show progress:

    "transactions_history_backfill": {
      "rewards_manager_backfilling_complete": false,
      "spl_token_backfilling_complete": false,
      "user_bank_backfilling_complete": false
    },
    "transactions_history_backfill_progress": {
      "rewards_manager_backfilling_progress": {
        "earliest_processed_sig": "3xcR2snhMPHXwHgoE276eivPu4mDFYNk6wDg6QUqLNZU6CSR4p5Fpy6stqDJ6hgpnsivof5D5xXMG9vHfkLWAG5j",
        "earliest_processed_slot": 164029133,
        "earliest_program_sig": "PepcujHu5x1uGbSZRV26bVYh3k3tmYxFjiKhGUB15B8WkhD9JGicrgqMzHS2pNNQW6dAH1L94yowt5FZpFyMmCw",
        "earliest_program_slot": 115695415,
        "slots_left": 48333718,
        "txs_traversed": 0
      },
      "spl_token_backfilling_progress": {
        "earliest_processed_sig": "eKPztkTptAXMPccH8bLMVZD4RoyZhEwkJtW2dC8qrjN1tkpEVUfNU3qx2SFiG2uCi4hhEFKAYnKVsBH65wRp5VC",
        "earliest_processed_slot": 163369015,
        "earliest_program_sig": "2D5maP2eXVXcx2XvyNrxSyBbQkFXqVVdTnmSC7v7HBXorC2m8QbNNaFsCBiKGq4drBegL3UgnQmRRashQ7Nf2fmA",
        "earliest_program_slot": 96997537,
        "slots_left": 66371478,
        "txs_traversed": 1098
      },
      "user_bank_backfilling_progress": {
        "earliest_processed_sig": "84NEXEzJsEugRzWJbMEGtEepnZd53D8YzC1AMqPx29otn1HfHvs4Gtfsn9esvqrDM6ePCKj3FN35wzrhTdHmxLa",
        "earliest_processed_slot": 163657656,
        "earliest_program_sig": "crtpmGZKrQv7Xud46VASp4cBkQaAdZV1v1VXH1kBETpMwxZMqFoTwvBdcGuDsa7uHLRFawtmWvNXz64F2j1jt4X",
        "earliest_program_slot": 115673923,
        "slots_left": 47983733,
        "txs_traversed": 2000
      }
    },

[gitguardian[bot]]

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	Secret	Commit	Filename
5071455	Generic High Entropy Secret	7db7a0cfa8c5e15fe8a24880eaf4253cb0eed526	discovery-provider/default_config.ini	View secret
5232911	PostgreSQL Credentials	b515d404d8129be8189b4857b94af43e11782d11	discovery-provider/alembic/env.py	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secrets safely. Learn here the best practices.
3. Revoke and rotate these secrets.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

<sup>🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!</sup>

[dharit-tan]

We're not merging this anymore!