Make ethereumjs-blockchain compatible again with the latest Geth leveldb database format

[BigMurry]

I'm trying to use ethereumjs-blockchain to read the raw levedb data created by geth, but I failed to do so. And I also try to use the dbManager to read the levedb data, failed again. My steps are:

1. sync some the blockchain data by using geth --goerli --datadir <my data dir>
2. write code to read the level db data.

const level = require('level');
const DbManager = require('ethereumjs-blockchain/dist/dbManager').default;

async function run () {
  const db = level('path to the geth chaindata');
  const dbManager = new DbManager(db);
  const block = await dbManager.getBlock(5); // successful when blockNumber = 0
  return block;
 }
// get the error: Error [NotFoundError]: Key not found in database [hn]

My question is if this package is compatible with the leveldb data created by geth or I missed something?

[holgerd77]

Hi @BigMurry, there is an example stated on the ethereumjs-blockchain package README page for connecting to a Geth DB, did you try? This likely hasn't been run for a while though.

If you are do please post the version you are using (master or some release?) together with the output you are (or are not 😄) getting here.

[BigMurry]

Hi @holgerd77 , thanks for the quick reply. I just tried the example, and the out put is:

Done.

nothing else. I've run geth --goerli to get some .ldb files in the chaindata folder, but the example code didn't log any thing.

---

ethereumjs-blockchain: 4.0.3 geth: Geth/v1.9.14-unstable-82f9ed49-20200508/linux-amd64/go1.14.2

[BigMurry]

I think it may caused by the ancient blob storage feature introduced by Geth from 1.9.x.

[holgerd77]

Sounds like this would need some further investigation. I've changed the title to reflect this and added some fitting labels. This would also also be a good grasp for external help, so I added help wanted and good first issue labels as well.

[jochem-brouwer]

This is probably because Goerli is a PoA chain which we do not support (i.e. the blocks are invalid, because we think it is a PoW chain).

Linking #689

[holgerd77]

I am not sure what's the status here and if it's still realistic to read the raw Geth leveldb database or if the format has changed substantially (e.g. with the latest snap sync changes?) and we should change on the example instead (or just remove for now).

Needs some investigation.

[holgerd77]

This would still need some investigation.

[ryanio]

It may be unreasonable for us to stay compatible with the geth leveldb format depending on how often it is changed or updated, what happens with a flat db layout strategy, etc. I assume there would be many changes over time.

If this is the case we should reword blockchain/README.md so it doesn't read The following is an example to iterate through an existing Geth DB (needs `level` to be installed separately). but rather something else, or at least specify the geth versions we know it works.

Maybe we can ask for some thoughts from the geth team, if it doesn't change that often, then maybe it is worth it to do the work to remain compatible, if there is a strong use case of people using this feature to work on geth dbs.

[wong2]

Maybe there could be an abstracted storage layer, so people can build and share custom implementations.

[acolytec3]

We do have custom db wrappers implemented for v7. Methinks we can close this and people can implement their own wrappers to allow this sort of thing to work.