Evgeny Medvedev
Evgeny Medvedev
SLOs might include: - Maximum data latency - Number of duplicate rows per year - Other consistency indicators - Availability
Using log filters instead of retrieving transaction receipts significantly improves performance of ERC20 retrieval https://github.com/ethereum/wiki/wiki/JSON-RPC#eth_getfilterlogs My tests show 100x improvement for some blocks. Here is an example implementation in python...
This will allow validating complex table definitions like these https://github.com/blockchain-etl/ethereum-etl-airflow/commit/ba6235d93c575e9d1958cd1d9c81da00f13cac29 in BigQuery, before a PR is created
ethers.js doesn't yet support parsing anonymous events: https://github.com/ethers-io/ethers.js/blob/master/packages/abi/lib.esm/interface.js#L466 Ask in their support group how it can be done with ethers.js.
The dag should run daily and query the last day's partition, save it to a temporary table and [extract](https://cloud.google.com/bigquery/docs/exporting-data) the result to a GCS bucket in gzipped csvs. The following...
Right now we have [~500 events](https://github.com/blockchain-etl/ethereum-etl-airflow/tree/master/dags/resources/stages/parse/table_definitions) that we parse. Every day ~500MB of log data is generated. This sums up to 250GB parsed in BigQuery daily, ~7.5 TB per month....
For some parsers contract_address is a static list of addresses which is duplicated in other definitions https://github.com/blockchain-etl/ethereum-etl-airflow/pull/94/files#diff-f0b9ae4effce038c22ca6abf7dfec316R28. Similar to dbt we can have a table definition with seed parser: ```...
https://cloud.google.com/bigquery/docs/clustered-tables. It can make some queries cheaper and faster.
Right now we support S3 and GCS as cloud storage providers and file upload/download is done directly in export DAG. Instead 2 classes S3CloudStorage and GCSCloudStorage can be created with...