[Access] Test pebble execution data db on testnet

[peterargue]

Using an observer on testnet, test running execution sync using a pebble db datastore.

Monitor logs and metrics for any errors or halts while syncing, or any other signs of performance issues.

You can use the configs from localnet to setup a local version of grafana to view metrics if that's useful

[peterargue]

@UlyanaAndrukhiv can you add details about how to set this up

[UlyanaAndrukhiv]

The Pebble execution data DB setup requires the following flags:

--execution-data-db=pebble
--execution-data-dir=<path_to_pebble_db>
--execution-data-sync-enabled=true

[Guitarheroua]

@peterargue

To check the pebble on mainnet, the Observer Node was up and running with next arguments:

docker run --rm \
-v $PWD/flow_observer/bootstrap:/bootstrap:ro \
-v $PWD/flow_observer/data:/data:rw \
--name flow_observer \
-p 80:80 \
-p 8080:8080 \
-p 3569:3569 \
-p 9000:9000 \
-p 9001:9001 \
gcr.io/flow-container-registry/observer:v0.37.10 \
--bootstrapdir=/bootstrap \
--datadir=/data/protocol \
--bind 0.0.0.0:3569 \
--rest-addr=:80 \
--loglevel=error \
--secretsdir=/data/secrets  \
--upstream-node-addresses=access-008.mainnet25.nodes.onflow.org:9001 \
--upstream-node-public-keys=11742552d21ac93da37ccda09661792977e2ca548a3b26d05f22a51ae1d99b9b75c8a9b3b40b38206b38951e98e4d145f0010f8942fd82ddf0fb1d670202264a \
--bootstrap-node-addresses=access-008.mainnet25.nodes.onflow.org:3570 \
--bootstrap-node-public-keys=11742552d21ac93da37ccda09661792977e2ca548a3b26d05f22a51ae1d99b9b75c8a9b3b40b38206b38951e98e4d145f0010f8942fd82ddf0fb1d670202264a \
--observer-networking-key-path=/bootstrap/network.key \
--execution-data-db=pebble \
--execution-data-dir=/data/execution-data \
--execution-data-sync-enabled=true \
--execution-state-dir=/data/execution-state \
--execution-state-checkpoint=/bootstrap/execution-state

After syncing up data, I got the next content in my /data/execution-state/blobstore:

total 3.1G
-rw-r--r-- 1 root root 5.7M Sep 13 15:03 002377.sst
-rw-r--r-- 1 root root  16M Sep 13 15:30 002817.sst
-rw-r--r-- 1 root root 8.1M Sep 13 15:30 002879.sst

...

-rw-r--r-- 1 root root 5.5M Sep 13 16:56 005085.sst
-rw-r--r-- 1 root root 6.2M Sep 13 16:56 005086.sst
-rw-r--r-- 1 root root 1.5M Sep 13 16:56 005087.sst
-rw-r--r-- 1 root root   16 Sep 13 12:13 CURRENT
-rw-r--r-- 1 root root    0 Sep 13 12:13 LOCK
-rw-r--r-- 1 root root 866K Sep 13 16:56 MANIFEST-000001
-rw-r--r-- 1 root root 2.5K Sep 13 12:13 OPTIONS-000003
-rw-r--r-- 1 root root    0 Sep 13 12:13 marker.format-version.000015.016
-rw-r--r-- 1 root root    0 Sep 13 12:13 marker.manifest.000001.MANIFEST-000001

So the data is stored in pebble db and *.sst files store the data.

During the sync process, however, I got errors, that may be not related to pebble, and they do not crash the node, but I mention them here:

{
  "level": "error",
  "node_role": "observer",
  "node_id": "bfe8cfd3018871cc104a61b8c5971ba9b7518fcc8e4ec77792896de573ed121e",
  "engine": "follower_core",
  "origin_id": "1f46b4d23c7557e7b3c21f85aebddcb7bcda7cc4d589732ce53087cb1b5faf8e",
  "chain_id": "flow-mainnet",
  "first_block_height": 86932614,
  "first_block_view": 957352,
  "last_block_height": 86932614,
  "last_block_view": 957352,
  "last_block_id": "83ba52d0f9467266a4a698c9e0a955b72476b9f940de7a30873ee01cd24ad2c9",
  "range_length": 1,
  "error": "error verifying leader signature for block 83ba52d0f9467266a4a698c9e0a955b72476b9f940de7a30873ee01cd24ad2c9: error retrieving voter Identity at view 957352: by-view query for unknown epoch",
  "time": "2024-09-13T15:44:41.779121803Z",
  "message": "unable to validate proposal with view from unknown epoch"
}

This happens for blocks from range 86932614 to 86932724.

[peterargue]

I think the way to verify the data in the db is correct is to do something like this:

1. Get the node's root block using the NodeVersionInfo endpoint
2. Setup a stream of execution data starting from the node's root block.
3. Use execution_data.CalculateID to calculate the root ID for the BlockExecutionData object.
4. Use GetExecutionResultByBlockID to get the execution result for the block, and compare the ID calculated in 3 with the ExecutionDataID field. If they match, then the data is identical

[Guitarheroua]

1. To test the database data, I added an extra argument --state-stream-addr=127.0.0.1:9003 to my node, enabling event streaming.
2. I retrieved the root block by running curl localhost:80/v1/node_version_info and received a root block height of 88226267.
3. I subscribed to events using wscat -c ws://localhost:80/v1/subscribe_events?start_height=88226267 and received all events starting from the root block.

[peterargue]

as discussed in standup, please test using BlockExecutionData instead of events.

[Guitarheroua]

Hey, @peterargue! Even after checking all the steps described above, I got the wrong results from CalculateID. I will attached my data as files. There are results from BlockExecutionData subscription CCF encoded, GetExecutionResultByBlockID and generated ID by CalculateID. The name of the files contains block height:

block_88226308_ccf.txt block_88226333_ccf.txt block_88226617_ccf.txt block_88226667_ccf.txt

Here is the code, that I used to convert the block_execution_data object:

func TestCalculateID(t *testing.T) {

// object shorten for this comment
	jsonData := `{"chunk_execution_data":[{"events":[{"type":"A.e467b9dd11fa00df.EVM.BlockExecuted","transaction_id":..,"block_id":"XxRzxnsnB+E3d54YiGuTFDl7vs1tHV1QOvcsBHldwFg="}`
	var blockExecutionData entities.BlockExecutionData
	err := json.Unmarshal([]byte(jsonData), &blockExecutionData)
	require.NoError(t, err)

	convertedExecData, err := accessConvert.MessageToBlockExecutionData(&blockExecutionData, flow.Mainnet.Chain())
	require.NoError(t, err)

	executionDataID, err := execution_data.CalculateID(context.Background(), convertedExecData, execution_data.DefaultSerializer)
	require.NoError(t, err)
	fmt.Printf("blockID: %s", base64.StdEncoding.EncodeToString(convertedExecData.BlockID[:]))
	fmt.Println("")
	fmt.Printf("executionDataID: %s", base64.StdEncoding.EncodeToString(executionDataID[:]))
	fmt.Println("")
}

[peterargue]

I don't think this is going to work using json. you will need to use grpc to get the binary data from the GetExecutionDataByBlockID (or streaming version) with CCF encoded events, then use the code you have above.

executionDataID won't equal the block ID, it should match the ExecutionDataID from the ExecutionResult for the block. You can use GetExecutionResultByBlockID to get the correct result.

[Guitarheroua]

I tested this with a small utility, and the results are the same: execution_data_id does not match the result of the CalculateID function. I tested it on my observer node with pebble and on the mainnet. Here is my code:

https://gist.github.com/Guitarheroua/5c901650fe809f9f31ed4289a07c61a1

The example of results for the same block:

> blockID: MKvPQvtljvuWht71odwUVGhy/SlytRH6fndScPF5Ezo=
> Calculated executionDataID: Vhdt1OeVAc97RizHEs3kOtVuAe+sTE7fZZut4uscLyU= 
> GetExecutionResultForBlockID blockID: MKvPQvtljvuWht71odwUVGhy/SlytRH6fndScPF5Ezo= 
> GetExecutionResultForBlockID executionDataID: N5hmea+3Fgohbp8ckV1W3+zihF3i8myBxmV7YruTWGw=
> 
>
> blockID: YG2FOBE4XAMygM5dYh32AFg7tNkhoNf1WInOlI5PXaw=
> Calculated executionDataID: 1MPekpu9PBjZz+kHUkNwx6FZh0OI/whmUL02QeD0mhI= 
> GetExecutionResultForBlockID blockID: YG2FOBE4XAMygM5dYh32AFg7tNkhoNf1WInOlI5PXaw=
> GetExecutionResultForBlockID executionDataID: Iy2PyiYnbUuy95FvI44m1Y8Sa9bbyWppdlS9ntLrSAI=

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.