stellar-etl
stellar-etl copied to clipboard
[WIP] Refactor `stellar-etl` to use GCS backend instead of Captive Core
This PR implements the new libraries metaarchive
and support/storage
from stellar/go@lighthorizon to make use of GCS as a backend instead of the resource hungry Captive Core.
The following export commands were refactored:
-
export_transactions
-
export_effects
-
export_operations
-
export_assets
-
export_trades
-
export_ledger_entry_changes
There's still some work to do:
- Remove hardcoded bucket (will it be filled with the
--testnet
logic?) - Update all tests to run against
public
ledgers (maybe after LedgerExporter is up and running?)
Note: the reason behind args "--gcs-bucket", "not"
inside tests is to prevent cmd from deleting the file. Since I have valid credentials for the GCP project and the default bucket is a valid bucket, the cmd uploads the file to GCS and delete from my machine which ultimately causes the test to fail locally.
looking forward for this PR to merge tho!
Closed completed in #235