stellar-etl icon indicating copy to clipboard operation
stellar-etl copied to clipboard

[WIP] Refactor `stellar-etl` to use GCS backend instead of Captive Core

Open lucaszanotelli opened this issue 2 years ago • 1 comments

This PR implements the new libraries metaarchive and support/storage from stellar/go@lighthorizon to make use of GCS as a backend instead of the resource hungry Captive Core.

The following export commands were refactored:

  • export_transactions
  • export_effects
  • export_operations
  • export_assets
  • export_trades
  • export_ledger_entry_changes

There's still some work to do:

  • Remove hardcoded bucket (will it be filled with the --testnet logic?)
  • Update all tests to run against public ledgers (maybe after LedgerExporter is up and running?)

Note: the reason behind args "--gcs-bucket", "not" inside tests is to prevent cmd from deleting the file. Since I have valid credentials for the GCP project and the default bucket is a valid bucket, the cmd uploads the file to GCS and delete from my machine which ultimately causes the test to fail locally.

lucaszanotelli avatar Feb 24 '23 19:02 lucaszanotelli

looking forward for this PR to merge tho!

jensenity avatar Oct 29 '23 03:10 jensenity

Closed completed in #235

sydneynotthecity avatar May 16 '24 15:05 sydneynotthecity