The daily mainnet dumps stopped updating after Jan 16, 2024

[artem-staked]

Preliminary Checks

• [X] This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/MinaProtocol/mina/issues
• [X] This issue is not a question, feature request, RFC, or anything other than a bug report. Please post those things in GitHub Discussions: https://github.com/MinaProtocol/mina/discussions

Description

The latest mainnet daily dump is https://storage.googleapis.com/mina-archive-dumps/mainnet-archive-dump-2024-01-16_0000.sql.tar.gz There are no more files for the mainnet after this date.

Steps to Reproduce

Try to download these files

1. https://storage.googleapis.com/mina-archive-dumps/mainnet-archive-dump-2024-01-17_0000.sql.tar.gz
2. https://storage.googleapis.com/mina-archive-dumps/mainnet-archive-dump-2024-01-18_0000.sql.tar.gz
3. https://storage.googleapis.com/mina-archive-dumps/mainnet-archive-dump-2024-02-11_0000.sql.tar.gz ...

Expected Result

Listed files are downloaded

Actual Result

404 errors

Daemon version

it's not related to mina daemon

How frequently do you see this issue?

Always

What is the impact of this issue on your ability to run a node?

Low

Status

it's not related to mina client

Additional information

No response

[psteckler]

The last four digits are not necessarily 0000.

$ gsutil ls gs://mina-archive-dumps/mainnet* | grep 2024-01-17
gs://mina-archive-dumps/mainnet-archive-dump-2024-01-17_0001.sql.tar.gz
gs://mina-archive-dumps/mainnet-backup-archive-dump-2024-01-17_0001.sql.tar.gz

$ gsutil ls gs://mina-archive-dumps/mainnet* | grep 2024-01-18
gs://mina-archive-dumps/mainnet-archive-dump-2024-01-18_0001.sql.tar.gz
gs://mina-archive-dumps/mainnet-backup-archive-dump-2024-01-18_0001.sql.tar.gz

$ gsutil ls gs://mina-archive-dumps/mainnet* | grep 2024-02-11
gs://mina-archive-dumps/mainnet-archive-dump-2024-02-11_0002.sql.tar.gz
gs://mina-archive-dumps/mainnet-backup-archive-dump-2024-02-11_0002.sql.tar.gz
gs://mina-archive-dumps/mainnet-migrated-archive-dump-2024-02-11_0000.sql.tar.gz

[artem-staked]

@psteckler Hello! The last complete mainnet archive I was able to fetch is https://storage.googleapis.com/mina-archive-dumps/mainnet-archive-dump-2024-04-05_0002.sql.tar.gz, all the further dates return empty files, e.g. https://storage.googleapis.com/mina-archive-dumps/mainnet-archive-dump-2024-04-09_0002.sql.tar.gz Did we miss anything here? Are those dumps still expected to be there?

[psteckler]

Yes, looks like the dumps are complete through April 5, 2024.

There are dumps created after that date, but indeed, they are all empty, so something is broken.

[artem-staked]

@psteckler Is this something you're gonna fix, or is it a third-party thing we can't rely on anymore?

[dkijania]

@artem-staked we fixed dumps export starting from:

https://storage.googleapis.com/mina-archive-dumps/mainnet-archive-dump-2024-04-13_0001.sql.tar.gz

We still need to address hours and minutes suffix issue to make it consistent (always _0000 or remove it entirely )

[artem-staked]

@dkijania just in case, we tried to restore from a couple of dates but both of them have missed blocks in DB

We check it with this query :

SELECT
        (select count(*) from blocks where parent_id is null) as parent_null,
        count(*) as missed_blocks
        FROM (SELECT h::int FROM generate_series(1 , (select max(height) from blocks)) h
        LEFT JOIN blocks b
        ON h = b.height where b.height is null) as v;

We expect it returns { "parent_null": 1, "missed_blocks": 0 } if there are no missed blocks in DB, but now I see { "parent_null": 9, "missed_blocks": 1862 }

[dkijania]

@artem-staked Thanks for pointing this out. This is very valuable discussion as I learnt that we (o1Labs) are not the only "customers" of our dumps in gcloud. As a result i will keep more attention on their availability . I will patch mainnet db with missing blocks.

We also had a very productive discussion internally regarding naming convention of dumps and I will make attempts to remove this annoying issue with "random" hhmm suffix for dumps timestamp. The proposal is to remove _hhmm suffix for the first dump at particular date, so it will be super easy to find it. If we run additional one the same day, we will add _hhmm suffix Please do tell me if you are interested in getting dumps daily or just occasionally or more often ?

[artem-staked]

@dkijania thanks! Yeah, we use them on a daily basis. But to be honest it's a workaround because we had hard times with archival node running. It was constantly missing blocks, and at that point just using dumps looked ok.

I've just thought, maybe it would be more convenient if you host a replica of your archival DB publicly available somewhere? E.g. I know Solana has it's archival DB in BigQuery

[SanabriaRusso]

Archive dumps from mainnet are being uploaded again. Would you mind checking @artem-staked ?

e.g.: https://storage.googleapis.com/mina-archive-dumps/mainnet-archive-dump-2024-04-25_0600.sql.tar.gz

[artem-staked]

@SanabriaRusso Looks good now! Thanks!

[
  {
    "parent_null": 1,
    "missed_blocks": 0
  }
]