cardano-db-sync
cardano-db-sync copied to clipboard
Published
20 hours ago •
input-output-hk
input-output-hk
Extendend whitelist options
Description
This PR fixes #1600 adding more whitelists to the configs
The most complex whitelist was the Shelley Stake Address as it stored in the table stake_address.
The functionality is if the stake address doesn't match in the whitelist on any of the inserts into the following tables, then we don't insert at all:
+-----------------------+
| table |
|-----------------------+
| collateral_tx_out |
| delegation |
| delegation_vote |
| epoch_stake |
| gov_action_proposal |
| instant_reward |
| pool_owner |
| pool_update |
| reserve |
| reward |
| stake_deregistration |
| stake_registration |
| treasury |
| treasury_withdrawal |
| tx_out |
| withdrawal |
+-----------------------+
Checklist
- [x] Commit sequence broadly makes sense
- [x] Commits have useful messages
- [X] New tests are added if needed and existing tests are updated
- [ ] Any changes are noted in the changelog
- [x] Code is formatted with
fourmoluon version 0.10.1.0 (which can be run withscripts/fourmolize.sh) - [x] Self-reviewed the diff
Migrations
- [ ] The pr causes a breaking change of type a,b or c
- [ ] If there is a breaking change, the pr includes a database migration and/or a fix process for old values, so that upgrade is possible
- [ ] Resyncing and running the migrations provided will result in the same database semantically
If there is a breaking change, especially a big one, please add a justification here. Please elaborate more what the migration achieves, what it cannot achieve or why a migration is not possible.
Run into following error message with executable built from this branch, on sanchonet:
[db-sync-node:Error:68] [2024-04-09 19:18:08.57 UTC] Error SNErrDefault: "resolveGovActionProposal.queryGovActionProposalId missing GovAction (TxKey {unTxKey = SqlBackendKey {unSqlBackendKey = 1878}}, 0)" [db-sync-node:Info:68] [2024-04-09 19:18:08.57 UTC] Shutting down DB thread
and it is hanging like that for quite some time, even though I used export DbSyncAbortOnPanic=1;
Longer log:
[db-sync-node:Info:68] [2024-04-09 19:18:08.15 UTC] Insert Conway Block: epoch 241, slot 20899080, block 1045000, hash 63f067e9e3ea707046130d460f1ab5fa9b66a6e28a2b00db937b7919d92db6c4
[db-sync-node:Info:68] [2024-04-09 19:18:08.16 UTC] queryOrInsertRewardAccount: "e0e23ef55a0099f4f3c0bf172828424d91ae0d61ef2d1456ba59117f59"
[db-sync-node:Info:68] [2024-04-09 19:18:08.17 UTC] queryOrInsertRewardAccount: "e0e23ef55a0099f4f3c0bf172828424d91ae0d61ef2d1456ba59117f59"
[db-sync-node:Info:68] [2024-04-09 19:18:08.17 UTC] queryOrInsertRewardAccount: "e0e23ef55a0099f4f3c0bf172828424d91ae0d61ef2d1456ba59117f59"
[db-sync-node:Info:68] [2024-04-09 19:18:08.17 UTC] queryOrInsertRewardAccount: "e0e23ef55a0099f4f3c0bf172828424d91ae0d61ef2d1456ba59117f59"
[db-sync-node:Info:68] [2024-04-09 19:18:08.21 UTC] queryOrInsertRewardAccount: "e05d7925e67a3c06e73529044636347d622f825cfbf8f2901bce2a4076"
[db-sync-node:Info:68] [2024-04-09 19:18:08.23 UTC] queryOrInsertRewardAccount: "e0d7b6df431bcd4f6547370dd91a84b9f19baa531fed0050d06d3eccbf"
[db-sync-node:Info:68] [2024-04-09 19:18:08.23 UTC] queryOrInsertRewardAccount: "e0d7b6df431bcd4f6547370dd91a84b9f19baa531fed0050d06d3eccbf"
[db-sync-node:Info:68] [2024-04-09 19:18:08.23 UTC] queryOrInsertRewardAccount: "e0d7b6df431bcd4f6547370dd91a84b9f19baa531fed0050d06d3eccbf"
[db-sync-node:Info:68] [2024-04-09 19:18:08.23 UTC] queryOrInsertRewardAccount: "e0d7b6df431bcd4f6547370dd91a84b9f19baa531fed0050d06d3eccbf"
[db-sync-node:Info:68] [2024-04-09 19:18:08.43 UTC] queryOrInsertRewardAccount: "e08411259f67557f242a8a3ce67ad49533861d88619493b0d2909a85ce"
[db-sync-node:Info:68] [2024-04-09 19:18:08.44 UTC] queryOrInsertRewardAccount: "e08411259f67557f242a8a3ce67ad49533861d88619493b0d2909a85ce"
[db-sync-node:Info:68] [2024-04-09 19:18:08.44 UTC] queryOrInsertRewardAccount: "e08411259f67557f242a8a3ce67ad49533861d88619493b0d2909a85ce"
[db-sync-node:Info:68] [2024-04-09 19:18:08.44 UTC] queryOrInsertRewardAccount: "e08411259f67557f242a8a3ce67ad49533861d88619493b0d2909a85ce"
[db-sync-node:Info:68] [2024-04-09 19:18:08.46 UTC] queryOrInsertRewardAccount: "e0fbfde89e03ec91335ea6a634f5c0c25b23fc0ea5bf12b6ad38b5e4dd"
[db-sync-node:Info:68] [2024-04-09 19:18:08.46 UTC] queryOrInsertRewardAccount: "e0fbfde89e03ec91335ea6a634f5c0c25b23fc0ea5bf12b6ad38b5e4dd"
[db-sync-node:Info:68] [2024-04-09 19:18:08.47 UTC] queryOrInsertRewardAccount: "e0fbfde89e03ec91335ea6a634f5c0c25b23fc0ea5bf12b6ad38b5e4dd"
[db-sync-node:Info:68] [2024-04-09 19:18:08.47 UTC] queryOrInsertRewardAccount: "e0fbfde89e03ec91335ea6a634f5c0c25b23fc0ea5bf12b6ad38b5e4dd"
[db-sync-node:Info:68] [2024-04-09 19:18:08.48 UTC] queryOrInsertRewardAccount: "e0608b8ec1b50edf6973e68b882c0e316eb0aa003f26be94d8cfd29b22"
[db-sync-node:Info:68] [2024-04-09 19:18:08.49 UTC] queryOrInsertRewardAccount: "e0892f36af0ddfd1ab639143893140b513f96ed48f7759ee6e4a652818"
[db-sync-node:Info:68] [2024-04-09 19:18:08.49 UTC] queryOrInsertRewardAccount: "e0892f36af0ddfd1ab639143893140b513f96ed48f7759ee6e4a652818"
[db-sync-node:Info:68] [2024-04-09 19:18:08.49 UTC] queryOrInsertRewardAccount: "e0892f36af0ddfd1ab639143893140b513f96ed48f7759ee6e4a652818"
[db-sync-node:Info:68] [2024-04-09 19:18:08.49 UTC] queryOrInsertRewardAccount: "e0892f36af0ddfd1ab639143893140b513f96ed48f7759ee6e4a652818"
[db-sync-node:Info:68] [2024-04-09 19:18:08.51 UTC] queryOrInsertRewardAccount: "e01d036f6ffdaf8341c63e4cf02ce788f5bcecd9b517cae17d22a08b53"
[db-sync-node:Info:68] [2024-04-09 19:18:08.52 UTC] queryOrInsertRewardAccount: "e01d036f6ffdaf8341c63e4cf02ce788f5bcecd9b517cae17d22a08b53"
[db-sync-node:Info:68] [2024-04-09 19:18:08.53 UTC] queryOrInsertRewardAccount: "e01d036f6ffdaf8341c63e4cf02ce788f5bcecd9b517cae17d22a08b53"
[db-sync-node:Info:68] [2024-04-09 19:18:08.53 UTC] queryOrInsertRewardAccount: "e01d036f6ffdaf8341c63e4cf02ce788f5bcecd9b517cae17d22a08b53"
[db-sync-node:Info:68] [2024-04-09 19:18:08.56 UTC] queryOrInsertRewardAccount: "e0192a8eb77a6f405d8196dd96db13950c48300a0ff8b6f42e7901a2ef"
[db-sync-node:Info:68] [2024-04-09 19:18:08.56 UTC] queryOrInsertRewardAccount: "e0192a8eb77a6f405d8196dd96db13950c48300a0ff8b6f42e7901a2ef"
[db-sync-node:Info:68] [2024-04-09 19:18:08.56 UTC] queryOrInsertRewardAccount: "e0192a8eb77a6f405d8196dd96db13950c48300a0ff8b6f42e7901a2ef"
[db-sync-node:Error:68] [2024-04-09 19:18:08.57 UTC] Error SNErrDefault: "resolveGovActionProposal.queryGovActionProposalId missing GovAction (TxKey {unTxKey = SqlBackendKey {unSqlBackendKey = 1878}}, 0)"
[db-sync-node:Info:68] [2024-04-09 19:18:08.57 UTC] Shutting down DB thread
Another question would be - should we have validator for stake addresses in config whitelisting ?
"shelley": {
"enable": true,
"stake_addresses": ["ABCDEF248f556b733c3ef24899ae0609d3796198d5470192304c4894dd85cb"]
},
I used this and no errors on cardano-db-sync side.
@ArturWieczorek
run into following error message with executable built from this branch, on sanchonet
what version of cardano-node are you using at the moment? I can't even get things to start syncing locally. I can see in the code where the issue is happening. Will try to get sanchonet working today 👍
Another question would be - should we have validator for stake addresses in config whitelisting
The problem with that is if you add the stake address before starting a fresh sync, there is no way to validate if it's existence as it's not been yet created.
@ArturWieczorek what version of cardano-node are you using at the moment?
@Cmdv For checking it I used:
cardano-node 8.9.0 - linux-x86_64 - ghc-9.6
git rev f7900cd232b33aa96e18e7a533454ce077d299c1
not sure if this is right - it seemed to work with this version.
I am using...
node:
./cardano-node version
cardano-node 8.9.1 - linux-x86_64 - ghc-8.10
git rev da945ea983d4722a9ffe54250edba9a193a57cf0
db-sync:
db-sync-node/bin/cardano-db-sync version
cardano-db-sync 13.2.0.1 - linux-x86_64 - ghc-8.10
git revision efa38c28ce7ebe666d2649299e38c4701dbca495
Network: sanchonet
Network Configs:
wget https://book.play.dev.cardano.org/environments/sanchonet/config.json \
https://book.play.dev.cardano.org/environments/sanchonet/db-sync-config.json \
https://book.play.dev.cardano.org/environments/sanchonet/submit-api-config.json \
https://book.play.dev.cardano.org/environments/sanchonet/topology.json \
https://book.play.dev.cardano.org/environments/sanchonet/byron-genesis.json \
https://book.play.dev.cardano.org/environments/sanchonet/shelley-genesis.json \
https://book.play.dev.cardano.org/environments/sanchonet/alonzo-genesis.json \
https://book.play.dev.cardano.org/environments/sanchonet/conway-genesis.json
db-sync config:
"insert_options": {
"tx_out": {
"value": "enable"
},
"ledger": "enable",
"shelley": {
"enable": true,
"stake_addresses": ["e083d27f03b197750ee41b7dca02c806718164b934b0fbfa747349e8f2"]
},
"multi_asset": {
"enable": true
},
"metadata": {
"enable": true
},
"plutus": {
"enable": true
},
"governance": "enable",
"offchain_pool_data": "enable",
"json_type": "text"
},
Logs:
So If I am reading it correctly - it breaks on querying / inserting first record in gov_action_proposal table:
Error SNErrDefault: "resolveGovActionProposal.queryGovActionProposalId missing GovAction (TxKey {unTxKey = SqlBackendKey {unSqlBackendKey = 1878}}, 0)"
Query on database synced without whitelisting:
sancho=# select * from gov_action_proposal limit 3;
id | tx_id | index | prev_gov_action_proposal | deposit | return_address | expiration | voting_anchor_id | type | description | param_proposal | ratified_epoch | enacted_epoch | dropped_epoch | expired_epoch
----+-------+-------+--------------------------+------------+----------------+------------+------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+----------------+---------------+---------------+---------------
1 | 1878 | 0 | | 1000000000 | 1024 | 251 | 1 | TreasuryWithdrawals | {"tag": "TreasuryWithdrawals", "contents": [[[{"network": "Testnet", "credential": {"keyHash": "248f556b733c3ef24899ae0609d3796198d5470192304c4894dd85cb"}}, 10000000000000]], null]} | | | | |
2 | 1948 | 0 | | 1000000000 | 1024 | 252 | 2 | InfoAction | {"tag": "InfoAction"} | | | | |
3 | 4116 | 0 | | 1000000000 | 1044 | 253 | 1 | TreasuryWithdrawals | {"tag": "TreasuryWithdrawals", "contents": [[[{"network": "Testnet", "credential": {"keyHash": "248f556b733c3ef24899ae0609d3796198d5470192304c4894dd85cb"}}, 51000000]], null]} | | | | |
(3 rows)
OK, I think I got it something more.
So basically adding return address
sancho=# select * from gov_action_proposal limit 3;
id | tx_id | index | prev_gov_action_proposal | deposit | return_address | expiration | voting_anchor_id | type | description | param_proposal | ratified_epoch | enacted_epoch | dropped_epoch | expired_epoch
----+-------+-------+--------------------------+------------+----------------+------------+------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+----------------+---------------+---------------+---------------
1 | 1878 | 0 | | 1000000000 | 1024 | 251 | 1 | TreasuryWithdrawals | {"tag": "TreasuryWithdrawals", "contents": [[[{"network": "Testnet", "credential": {"keyHash": "248f556b733c3ef24899ae0609d3796198d5470192304c4894dd85cb"}}, 10000000000000]], null]} | | | | |
in this case id=1024:
sancho=# select * from stake_address where id=1024;
id | hash_raw | view | script_hash
------+--------------------------------------------------------------+------------------------------------------------------------------+-------------
1024 | \xe0192a8eb77a6f405d8196dd96db13950c48300a0ff8b6f42e7901a2ef | stake_test1uqvj4r4h0fh5qhvpjmwedkcnj5xysvq2plutdapw0yq69mccrcnd7 |
(1 row)
to stake_addresses whitelist:
"stake_addresses": ["e083d27f03b197750ee41b7dca02c806718164b934b0fbfa747349e8f2","e0192a8eb77a6f405d8196dd96db13950c48300a0ff8b6f42e7901a2ef"]
Moved error past epoch 241:
[db-sync-node:Info:68] [2024-04-10 16:52:15.19 UTC] Starting epoch 243
[db-sync-node:Info:68] [2024-04-10 16:52:15.19 UTC] Insert Conway Block: epoch 243, slot 20995259, block 1048448, hash b1bbaf1289853643b930e143c20f88d02433804eaba473e071dff13f7aeaa617
[db-sync-node:Info:68] [2024-04-10 16:52:15.20 UTC]
Inserted epoch 242 from updateEpochWhenSyncing with Cache.
epoch: Epoch {epochOutSum = 12944517829466, epochFees = DbLovelace 280783544, epochTxCount = 1411, epochBlkCount = 2974, epochNo = 242, epochStartTime = 2024-02-12 00:30:50 UTC, epochEndTime = 2024-02-13 00:29:06 UTC}
[db-sync-node:Info:76] [2024-04-10 16:52:15.24 UTC] Asynchronously wrote a ledger snapshot to ledger-state/sancho_whitelist/20995146-631926b413-242.lstate in 0.058838804s.
[db-sync-node:Info:68] [2024-04-10 16:52:15.74 UTC] Inserted 1 EpochStake for EpochNo 244
[db-sync-node:Info:68] [2024-04-10 16:52:16.97 UTC] Insert Conway Block: epoch 243, slot 21043666, block 1050000, hash e4ed716d1560fc7d3051b4da271410f31a28e233bb0c3462142fcd2950470f87
[db-sync-node:Error:68] [2024-04-10 16:52:17.55 UTC] Error SNErrDefault: "resolveGovActionProposal.queryGovActionProposalId missing GovAction (TxKey {unTxKey = SqlBackendKey {unSqlBackendKey = 4222}}, 0)"
[db-sync-node:Info:68] [2024-04-10 16:52:17.55 UTC] Shutting down DB thread
So if stake address for return_address is filtered out by whitelisting it will cause this exception.
@Cmdv With latest version on this branch I am left stuck at Reached EpochNo 301 message.
I see that tables listed below are empty while the non whitelisted version of DB have records for them ( the common thing is that they have gov_action_proposal_id field ):
select * from constitution;
id | gov_action_proposal_id | voting_anchor_id | script_hash
----+------------------------+------------------+-------------
(0 rows)
select * from new_committee_info;
id | gov_action_proposal_id | quorum_numerator | quorum_denominator
----+------------------------+------------------+--------------------
(0 rows)
select * from new_committee;
id | gov_action_proposal_id | deleted_members | added_members | quorum_numerator | quorum_denominator
----+------------------------+-----------------+---------------+------------------+--------------------
(0 rows)
select * from new_committee_member;
id | gov_action_proposal_id | committee_hash_id | expiration_epoch
----+------------------------+-------------------+------------------
(0 rows)
select * from voting_procedure;
id | tx_id | index | gov_action_proposal_id | voter_role | drep_voter | pool_voter | vote | voting_anchor_id | committee_voter
----+-------+-------+------------------------+------------+------------+------------+------+------------------+-----------------
(0 rows)
Also do not see records in those tables ( contrary to non whitelisted DB version )
off_chain_pool_data
off_chain_pool_fetch_error
off_chain_vote_fetch_error
EDIT:
"shelley": {
"enable": true,
"stake_addresses": ["e083d27f03b197750ee41b7dca02c806718164b934b0fbfa747349e8f2"]
},
sancho_regular=# select * from stake_address where id=5;
id | hash_raw | view | script_hash
----+--------------------------------------------------------------+------------------------------------------------------------------+-------------
5 | \xe083d27f03b197750ee41b7dca02c806718164b934b0fbfa747349e8f2 | stake_test1uzpaylcrkxth2rhyrd7u5qkgqeccze9exjc0h7n5wdy73usd0pxpl |
(1 row)
It seems like there might be also issue with tx_out table - on non whitelisted database stake_address_id=5 has no entries:
sancho_regular=# select * from tx_out where stake_address_id=5;
id | tx_id | index | address | address_has_script | payment_cred | stake_address_id | value | data_hash | inline_datum_id | reference_script_id
----+-------+-------+---------+--------------------+--------------+------------------+-------+-----------+-----------------+---------------------
(0 rows)
sancho_regular=# select count(*) from tx_out where stake_address_id IS NOT NULL;
count
-------
6210
(1 row)
and on whitelisted by stake_address_id=5 there are - and generally all stake_address_id fields are NULL:
sancho_test_config_shelley_whitelist=# select count(*) from tx_out;
count
--------
103980
(1 row)
sancho_test_config_shelley_whitelist=# select count(*) from tx_out where stake_address_id IS NOT NULL;
count
-------
0
(1 row)
So we should get rid off all those records with NULL stake_address_id and then it should be OK I think.
This is related to plutus whitelisting.
Assuming that config below is correct:
"insert_options": {
"tx_out": {
"value": "enable"
},
"ledger": "enable",
"shelley": {
"enable": true
},
"multi_asset": {
"enable": true
},
"metadata": {
"enable": true
},
"plututs": {
"enable": true,
"script_hashes":["34250edd1e9836f5378702fbf9416b709bc140e04f668cc355208518","67f33146617a5e61936081db3b2117cbf59bd2123748f58ac9678656", "bccf2a430c016bc960fbf31b02694011cd399d20da8882aac9d33611", "1b539b68e5389bdc3fe511c7694583e3755f58d4b38aef50384f4349"]
},
"governance": "enable",
"offchain_pool_data": "enable",
"json_type": "text"
},
I am seeing entries in script table that should not be there:
sancho_test_config_shelley_whitelist_plutus=# SELECT id, tx_id, hash, type, json
FROM script
WHERE hash NOT IN (
'\x34250edd1e9836f5378702fbf9416b709bc140e04f668cc355208518', -- hash for 'timelock' - 1st entry in script_hashes
'\x67f33146617a5e61936081db3b2117cbf59bd2123748f58ac9678656', -- hash for 'plutusV1' - 2nd entry in script_hashes
'\xbccf2a430c016bc960fbf31b02694011cd399d20da8882aac9d33611', -- hash for 'plutusV2' - 3rd entry in script_hashes
'\x1b539b68e5389bdc3fe511c7694583e3755f58d4b38aef50384f4349' -- hash for 'plutusV3' - 4th entry in script_hashes
)
ORDER BY type, id;
id | tx_id | hash | type | json
----+-------+------------------------------------------------------------+----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
6 | 17132 | \xeeb006239ad2562556b736c3990f20968fcf89bb6d4f7292aaa0c30a | timelock | {"type": "atLeast", "scripts": [{"type": "sig", "keyHash": "ff750767ac2496a659b8325bf7064f92d969672d931ce0bc0290f6ba"}, {"type": "sig", "keyHash": "b7b9594ef50295739c95144238654413221df009f388cbf9b8ca73b7"}, {"type": "sig", "keyHash": "0253cc2bc1ed8176c675f454dd730fae5bfaa147b73924bde70d786a"}], "required": 2}
24 | 28937 | \x6b0261483fa1fe471c6f941c2a917d75164cbd96a93a9e42f62efd1c | timelock | {"type": "sig", "keyHash": "6c6fe0b26f5bfa67ece9221683ea4e7ad19bdf14db8a9fae7b294b5e"}
9 | 19924 | \x56b0f0b597150e619c76bed60683f3b1e42d7bc0685ed951b882bfc5 | plutusV2 |
10 | 19924 | \x86bff95ba20e9d1d1b34899a56d86bbacc9fed999260b27dcc92d128 | plutusV2 |
11 | 19927 | \x0689163fa0280c34d6e3d78657e2b06980be7ac77854f0dd5201bf05 | plutusV2 |
12 | 21241 | \x8275a6621fdb65e60c117fddd5f162d43adf6364c2645cd865bf5482 | plutusV2 |
14 | 24872 | \x04b8c921d534b5debae418ae679a83ec5e628b8e680af1a241788790 | plutusV2 |
...
@ArturWieczorek thanks a lot for the very thorough reports 🙏
With latest version on this branch I am left stuck at Reached EpochNo 301 message.
I wonder if it has something with stopping and restarting as I got to 301 last week and that was the tip but now we're on 306. So this morning I restarted and like you it was just not moving forward. I can't fully tell it's it's node not providing more data or something is going on on our side. Will investigate more.
This is related to plutus whitelisting.
So it's doesn't just whitelist against script hash, it also checks payment creds. Which was the existing functionality. I'm unsure as to the reasoning but it's functionality was kept. New code here
What I would like to see for this is
- documentation about what db fields remain empty for the new options
- what is the testing coverage, testing method and what combinations of networks and options were used
- what ensures that if this options are not enabled, default db-sync schema is not affected by this pr
- possible remaining issues
@kderme
What I would like to see for this is
- documentation about what db fields remain empty for the new options
All tables effected were documented for Shelley stake address but just updated, Multi asset Policies + Plutus Script hashes
- what is the testing coverage, testing method and what combinations of networks and options were used
The test check that each whitelist does indeed omit data going into tables given whitelist (couldn't test sanchonet related tables as they are not currently filled) also there are tests for checking parsing of config for the whitelist.
- what ensures that if this options are not enabled, default db-sync schema is not affected by this pr
All existing implementation remain the same and functionality only changes if whitelists are present.
- possible remaining issues
Everything worked in preprod + mainet but had issue with sanchonet where db-sync would hang around epoch 301-302 as per @ArturWieczorek observations above. Artur should also be able to double check if adding whitelist doesn't break existing functionality with preprod + mainnet I've not tried testing multiple whitelist in conjunction with each other, but doubt that would cause issues, can always try.
There's still an HLint failure: https://ci.iog.io/build/3779994/nixlog/1
@Cmdv I ran into this: Error SNErrDefault: "resolveScriptHash tx hash \"205f623b31de74b6f6886202a81a781edfb06b7f028aca7cdb28a7fe10b587ca\""
[db-sync-node:Info:73] [2024-05-07 09:45:26.76 UTC] Starting epoch 29
[db-sync-node:Info:73] [2024-05-07 09:45:26.76 UTC] Insert Babbage Block: epoch 29, slot 10886468, block 188291, hash c254a37774be4de2ee204dc8e9d438045f359ce78d576f5aab3c45b35da6c0f1
[db-sync-node:Info:73] [2024-05-07 09:45:26.76 UTC]
Inserted epoch 28 from updateEpochWhenSyncing with Cache.
epoch: Epoch {epochOutSum = 17847453521738724661, epochFees = DbLovelace 877160908, epochTxCount = 4045, epochBlkCount = 14739, epochNo = 28, epochStartTime = 2022-10-19 00:00:04 UTC, epochEndTime = 2022-10-23 23:59:32 UTC}
[db-sync-node:Info:82] [2024-05-07 09:45:26.77 UTC] Asynchronously wrote a ledger snapshot to ledger-state/preprod_whitelist_stake_addresses/10886372-baa629e75d-28.lstate in 0.02101376s.
[db-sync-node:Info:73] [2024-05-07 09:45:28.86 UTC] Insert Babbage Block: epoch 29, slot 10921207, block 190000, hash d6b9ddf01626b84b8dcb8cba20464611ad0a40603294b6f2d45843d11b653339
[db-sync-node:Info:73] [2024-05-07 09:45:29.44 UTC] Inserted 1 EpochStake for EpochNo 30
[db-sync-node:Error:73] [2024-05-07 09:45:33.93 UTC] Error SNErrDefault: "resolveScriptHash tx hash \"205f623b31de74b6f6886202a81a781edfb06b7f028aca7cdb28a7fe10b587ca\""
[db-sync-node:Info:73] [2024-05-07 09:45:33.93 UTC] Shutting down DB thread
on preprod using :
"insert_options": {
"tx_out": {
"value": "enable"
},
"ledger": "enable",
"shelley": {
"enable": true,
"stake_addresses": ["e04bec2d4e59e40ffc8ac4ed7e3b1e5b32cdc821f5f4b8447cc06d021f", "e0849201473a18ed31ddf126eb36b9ece9d11c8ec5b98cac8dd9c2df6b", "e016a4092303be48b6477c2a0593a8cd483ca045284585eb4853e84726"]
},
"multi_asset": {
"enable": true
},
"metadata": {
"enable": true
},
"plutus": {
"enable": true
},
"governance": "enable",
"offchain_pool_data": "enable",
"json_type": "text"
},
db-sync-node/bin/cardano-db-sync --version
cardano-db-sync 13.2.0.1 - linux-x86_64 - ghc-8.10
git revision cc66f77e9951229f7b73275042b37c9c393c8d26
@ArturWieczorek could you double check this PR for me when you next get the chance? 👍
@Cmdv
On sanchonet:
"insert_options": {
"preset": "only_gov"
}
with:
./cardano-node version
cardano-node 9.0.0 - linux-x86_64 - ghc-8.10
git rev 2820a63dc934c6d5b5f450b6c2543b81c6476696
./db-sync-node/bin/cardano-db-sync version
git revision c4f7e83ba267b3784a019b31783e39131c6b1477
I ran into [db-sync-node:Error:77] [2024-07-25 11:23:45.57 UTC] Error SNErrDefault: "resolveGovActionProposal.queryGovActionProposalId missing GovAction (TxKey {unTxKey = SqlBackendKey {unSqlBackendKey = 1283}}, 0)" :
[db-sync-node:Info:77] [2024-07-25 11:23:36.81 UTC] Starting epoch 393
[db-sync-node:Info:77] [2024-07-25 11:23:36.81 UTC] Insert Conway Block: epoch 393, slot 33955211, block 1698926, hash 2bbcce48b0580dd58831de1fb1349133ae130b37b0462a3d72728e07892119b6
[db-sync-node:Info:85] [2024-07-25 11:23:36.85 UTC] Asynchronously wrote a ledger snapshot to ledger-state/sancho_only_gov/33955151-163957d025-392.lstate in 0.039110436s.
[db-sync-node:Info:77] [2024-07-25 11:23:37.60 UTC] Inserted 0 EpochStake for EpochNo 394
[db-sync-node:Info:77] [2024-07-25 11:23:38.84 UTC] Insert Conway Block: epoch 393, slot 33976998, block 1700000, hash 48490c35b1cbd36546ec51ccd8b89954b23be572455cf4e2857b4797f550bb60
[db-sync-node:Info:77] [2024-07-25 11:23:45.53 UTC] Removing old epoch boundary file ledger-state/sancho_only_gov/33436789-a577f01423-386.lstate
[db-sync-node:Info:77] [2024-07-25 11:23:45.54 UTC] Removing 0 orphaned rewards
[db-sync-node:Warning:77] [2024-07-25 11:23:45.54 UTC] validateEpochRewards: rewards spendable in epoch 394 expected total of 3 but got 0
[db-sync-node:Info:77] [2024-07-25 11:23:45.54 UTC] Persistant SQL Statement Cache size is 44
[db-sync-node:Info:77] [2024-07-25 11:23:45.54 UTC]
Cache Statistics:
Stake Addresses: cache sizes: 0 and 0, hit rate: 0%, hits: 0, misses: 97
Pools: cache size: 0, hit rate: 0%, hits: 0, misses: 1703194
Datums: cache capacity: 250000, cache size: 0, hits: 0, misses: 0
Multi Assets: cache capacity: 250000, cache size: 0, hits: 0, misses: 0
Previous Block: hit rate: 49%, hits: 1703193, misses: 1703194
TxId: cache size: 1390, cache capacity: 300000, hits: 0, misses: 0
[db-sync-node:Info:77] [2024-07-25 11:23:45.54 UTC] Starting epoch 394
[db-sync-node:Info:77] [2024-07-25 11:23:45.54 UTC] Insert Conway Block: epoch 394, slot 34041601, block 1703194, hash 2588a19fd69700dae6edaf56ee64116ccb19508b66faff27b8b840f096d042f1
[db-sync-node:Error:77] [2024-07-25 11:23:45.57 UTC] Error SNErrDefault: "resolveGovActionProposal.queryGovActionProposalId missing GovAction (TxKey {unTxKey = SqlBackendKey {unSqlBackendKey = 1283}}, 0)"
[db-sync-node:Info:77] [2024-07-25 11:23:45.57 UTC] Shutting down DB thread
@Cmdv Looks like full preset is missing data for tx_cbor table:
Full This is equivalent to enabling all other settings.
sancho_new_test_config_full=# select * from tx_cbor;
id | tx_id | bytes
----+-------+-------
(0 rows)
EDIT: @Cmdv Looks like it is correct behaviour. According to docs:
Preset is an aggregate setting that overrides all other properties. For example, setting preset to "full" will enable all insert options except "tx_cbor".
@Cmdv
I am on preview and have issues with multiasset filter.
EDIT: This is happening on 13.3.0 - not sure if code has changed in this branch for that feature.
This is my config - is it correct ?
"insert_options": {
"multi_asset": {
"enable": true,
"policies": [
"6c969320597b755454ff3653ad09725d590c570827a129aeb4385526",
"065270479316f1d92e00f7f9f095ebeaac9d009c878dc35ce36d3404"
]
}
}
and here are queries that show issue with filtering not happening.
preview_13_3_0_0_ma_filter=# SELECT count(*) FROM multi_asset WHERE policy IN ('\x6c969320597b755454ff3653ad09725d590c570827a129aeb4385526', '\x065270479316f1d92e00f7f9f095ebeaac9d009c878dc35ce36d3404');
count
-------
34
(1 row)
preview_13_3_0_0_ma_filter=# SELECT count(*) FROM multi_asset WHERE policy NOT IN ('\x6c969320597b755454ff3653ad09725d590c570827a129aeb4385526', '\x065270479316f1d92e00f7f9f095ebeaac9d009c878dc35ce36d3404');
count
-------
1719
(1 row)
@Cmdv So far it looks good:
"multi_asset": {
"enable": true,
"policies": [
"3da414859ddd0955cf6694a20cbbc000d60c4cb49dcbf46ad543568a",
"c70ef82c7ff32b5dda84e3519d8d2b8cbd29c1d2a93cc2e183797143"
]
},
and DB:
sancho_test_policies=# select * from multi_asset;
id | policy | name | fingerprint
----+------------------------------------------------------------+----------------------------------------------------------------+----------------------------------------------
1 | \x3da414859ddd0955cf6694a20cbbc000d60c4cb49dcbf46ad543568a | \x37f9a73217d919ef97332211ade0afd42732e38b11577c82bbddfd9f2301 | asset16vajwx08w7wgwazxu9fe6lvvpxakgtxftmfc4w
2 | \x3da414859ddd0955cf6694a20cbbc000d60c4cb49dcbf46ad543568a | \xe8a0e694ab11691ed96275dbb8e02c6ab7ae9af840e1e119830ae4e72300 | asset1uqx7htnvmw9az0qwvdhc5e06ep6qg535uwne2f
3 | \x3da414859ddd0955cf6694a20cbbc000d60c4cb49dcbf46ad543568a | \x1c64222c787d10d13247d24b92563a28da34edcbdd918a1c280d02aa2300 | asset1jm3gee8mm3mmrep49c0qx7xmz0wugulx8fcs7l
4 | \x3da414859ddd0955cf6694a20cbbc000d60c4cb49dcbf46ad543568a | \xeda822c2552732f57e749ed4d9c6056a0989884b12ac863301e83ecd2300 | asset129za0pu050u8ulhh0a08zdkwde8ued5cdgpcj2
5 | \xc70ef82c7ff32b5dda84e3519d8d2b8cbd29c1d2a93cc2e183797143 | \x5e249e4b897fe25aea28119031cbb7931ccd4ae135caec5c90fabe3d2301 | asset146w8yfx0c5gue58v44nasvz7rx6c52wmapdprn
6 | \xc70ef82c7ff32b5dda84e3519d8d2b8cbd29c1d2a93cc2e183797143 | \x5ea4fa27f12cda53293b37f721092eedad3f3221832c3a6c706e8d3a2300 | asset1enqrads9txxszt3kt4mx5sqwyq5ur8lne8yfak
7 | \xc70ef82c7ff32b5dda84e3519d8d2b8cbd29c1d2a93cc2e183797143 | \x72f5eeb05dcacbab058506ef94443c0d9a513c61b6a4e442513692702301 | asset1en2qcllrneuh7fc26h3kcwfst3syz5kr204era
8 | \xc70ef82c7ff32b5dda84e3519d8d2b8cbd29c1d2a93cc2e183797143 | \x0826e33c43dce633a67a41528d87352f7b940282d8e5a5a29f8c8abd2300 | asset13gddghj7j2y53tfdjlvcmfc762g56r6wfe26rr
(8 rows)
VS database synced with "standard" config:
sancho_normal=# select * from multi_asset where policy = '\x3da414859ddd0955cf6694a20cbbc000d60c4cb49dcbf46ad543568a' or policy = '\xc70ef82c7ff32b5dda84e3519d8d2b8cbd29c1d2a93cc2e183797143';
id | policy | name | fingerprint
----+------------------------------------------------------------+----------------------------------------------------------------+----------------------------------------------
1 | \x3da414859ddd0955cf6694a20cbbc000d60c4cb49dcbf46ad543568a | \x37f9a73217d919ef97332211ade0afd42732e38b11577c82bbddfd9f2301 | asset16vajwx08w7wgwazxu9fe6lvvpxakgtxftmfc4w
2 | \x3da414859ddd0955cf6694a20cbbc000d60c4cb49dcbf46ad543568a | \xe8a0e694ab11691ed96275dbb8e02c6ab7ae9af840e1e119830ae4e72300 | asset1uqx7htnvmw9az0qwvdhc5e06ep6qg535uwne2f
3 | \x3da414859ddd0955cf6694a20cbbc000d60c4cb49dcbf46ad543568a | \x1c64222c787d10d13247d24b92563a28da34edcbdd918a1c280d02aa2300 | asset1jm3gee8mm3mmrep49c0qx7xmz0wugulx8fcs7l
4 | \x3da414859ddd0955cf6694a20cbbc000d60c4cb49dcbf46ad543568a | \xeda822c2552732f57e749ed4d9c6056a0989884b12ac863301e83ecd2300 | asset129za0pu050u8ulhh0a08zdkwde8ued5cdgpcj2
5 | \xc70ef82c7ff32b5dda84e3519d8d2b8cbd29c1d2a93cc2e183797143 | \x5e249e4b897fe25aea28119031cbb7931ccd4ae135caec5c90fabe3d2301 | asset146w8yfx0c5gue58v44nasvz7rx6c52wmapdprn
6 | \xc70ef82c7ff32b5dda84e3519d8d2b8cbd29c1d2a93cc2e183797143 | \x5ea4fa27f12cda53293b37f721092eedad3f3221832c3a6c706e8d3a2300 | asset1enqrads9txxszt3kt4mx5sqwyq5ur8lne8yfak
38 | \xc70ef82c7ff32b5dda84e3519d8d2b8cbd29c1d2a93cc2e183797143 | \x72f5eeb05dcacbab058506ef94443c0d9a513c61b6a4e442513692702301 | asset1en2qcllrneuh7fc26h3kcwfst3syz5kr204era
55 | \xc70ef82c7ff32b5dda84e3519d8d2b8cbd29c1d2a93cc2e183797143 | \x0826e33c43dce633a67a41528d87352f7b940282d8e5a5a29f8c8abd2300 | asset13gddghj7j2y53tfdjlvcmfc762g56r6wfe26rr
(8 rows)
For
"shelley": {
"enable": true,
"stake_addresses": ["e092a56d8920af4451021a07a0d59f7d0d8a1f3359158c98e1d9cf2be4", "e003d205532089ad2f7816892e2ef42849b7b52788e41b3fd43a6e01cf"]
},
This does not seem right :
sancho_test_address_filtering=# select * from stake_address;
id | hash_raw | view | script_hash
----+--------------------------------------------------------------+------------------------------------------------------------------+-------------
1 | \xe092a56d8920af4451021a07a0d59f7d0d8a1f3359158c98e1d9cf2be4 | stake_test1uzf22mvfyzh5g5gzrgr6p4vl05xc58enty2cex8pm88jheqrpmt2u |
2 | \xe008f121b36abf2bcdd7daa2551c1e6653413a78c419e170d3319924d3 | stake_test1uqy0zgdnd2ljhnwhm23928q7vef5zwnccsv7zuxnxxvjf5c0swgr8 |
3 | \xe0e0d4c1ce0503af576155012f78f5601fe3593a93e4e7690b69813c3b | stake_test1ursdfswwq5p674mp25qj7784vq07xkf6j0jww6gtdxqncwcpy58kc |
4 | \xe072715e17c335384378239b830eca11de0f76f436c231337395a4acdf | stake_test1upe8zhshcv6nssmcywdcxrk2z80q7ah5xmprzvmnjkj2ehch9nrjd |
5 | \xe0c66ad2e4be6c8b6f4ff32b08be6d8b0c757782a6d116239a6903bf59 | stake_test1urrx45hyhekgkm607v4s30nd3vx82auz5mg3vgu6dypm7kgcnxs8z |
6 | \xe06d77fe1ae0aa1037e054657f1d91cd503efd0538f2566704d232c723 | stake_test1upkh0ls6uz4pqdlq23jh78v3e4gralg98re9vecy6gevwgcd32r5a |
7 | \xe0b17951d65a7ec99d008d3bc888b1986088b26c44792631bdbb29c8bd | stake_test1uzchj5wktflvn8gq35au3z93npsg3vnvg3ujvvdahv5u30gxuggdt |
8 | \xe09b632f3bc249952490afcd6557cd3be2f09f15637b097f24ae5bb093 | stake_test1uzdkxtemcfye2fys4lxk247d8030p8c4vdasjley4edmpycgmrxs2 |
9 | \xe0176b66617c48f76b3ee4a86e488b51e6969e0b5e81ed2147039159bf | stake_test1uqtkkenp03y0w6e7uj5xujyt28nfd8stt6q76g28qwg4n0c3tkc8s |
10 | \xe0c5ffc5c5445bb4cc9d8dc986768700959e7901e0916788d2512e08e9 | stake_test1urzll3w9g3dmfnya3hycva58qz2eu7gpuzgk0zxj2yhq36g466keq |
11 | \xe0c13582aec9a44fcc6d984be003c5058c660e1d2ff1370fd8b49ba73f | stake_test1urqntq4wexjylnrdnp97qq79qkxxvrsa9lcnwr7ckjd6w0cr04y4p |
12 | \xe0df1992dd82b1e2c3f1389ce800701cc49300de7ce71b7af57cb4feca | stake_test1ur03nykas2c79sl38zwwsqrsrnzfxqx70nn3k7h40j60ajsle7at7 |
13 | \xe01bf1e138f2f8beabc963c94cc28ee8ed4b41744601f2edaf50b21efd | stake_test1uqdlrcfc7tuta27fv0y5es5wark5kst5gcql9md02zepalg9yxxuz |
14 | \xe0ccf6ffabac700363af242c39fee6228e3a53b5fc7dae4c5675af8904 | stake_test1urx0dlat43cqxca0yskrnlhxy28r55a4l376unzkwkhcjpqeqdp0r |
15 | \xe07b0a8d15d72c6d497a43ae127b3d03b1dc34e1f18fcd86acd5457d77 | stake_test1upas4rg46ukx6jt6gwhpy7eaqwcacd8p7x8ump4v64zh6acrjhemn |
16 | \xe0992d25f1234f161763f5faaeea857baf06de242547bde90895019cde | stake_test1uzvj6f03yd83v9mr7ha2a6590whsdh3yy4rmm6ggj5qeehstm86lq |
17 | \xe031e66d448f7fe215f77ab30eb7409585f9196c7037487130b8c81eaa | stake_test1uqc7vm2y3al7y90h02esad6qjkzljxtvwqm5sufshrypa2sulwjzz |
18 | \xe0480b7d18a26a3d1c8097038cc2268066f88d09266ef771b17f375afd | stake_test1upyqklgc5f4r68yqjupces3xspn03rgfyeh0wud30um44lg583k50 |
19 | \xe023066388b51eefb6f405ae8bae44d11e1efc15062748b628f5c8be0b | stake_test1uq3svcugk50wldh5qkhghtjy6y0palq4qcn53d3g7hytuzcav7rp0 |
20 | \xe0e7cd215fa76c70b9987c4f983766ee63dc8e31cdd8cbd491a2807426 | stake_test1urnu6g2l5ak8pwvc038esdmxae3aer33ehvvh4y352q8gfszxrpxq |
21 | \xe08141d8651e0139b3bfc9c31f76d885431f83957329d75e0e53a65c2a | stake_test1uzq5rkr9rcqnnvale8p37akcs4p3lqu4wv5awhsw2wn9c2sey6zya |
22 | \xe08fb4b1cbf9dd257d83859ab5833541ade22c3d01ca02e22d5500ed10 | stake_test1uz8mfvwtl8wj2lvrskdttqe4gxk7ytpaq89q9c3d25qw6yqj5c6wh |
23 | \xe003d205532089ad2f7816892e2ef42849b7b52788e41b3fd43a6e01cf | stake_test1uqpayp2nyzy66tmcz6yjuth59pym0df83rjpk0758fhqrnc8dx6fl |
24 | \xe06f7b4bb2492b2ca9ebd5da5cc28ff453b35c71997a643957a7a1de9d | stake_test1uphhkjajfy4je20t6hd9es5073fmxhr3n9axgw2h57saa8gpvk6j8 |
25 | \xe03bdb8a34e61405507e0321511bf4afbcfa40a8ad87f97112a5777145 | stake_test1uqaahz35uc2q25r7qvs4zxl547705s9g4krljugj54mhz3gk5lc36 |
26 | \xe00062f6f5b4951b27d1c8948b7f5f07a434f6f7e5d629380e1648946c | stake_test1uqqx9ah4kj23kf73ez2gkl6lq7jrfahhuhtzjwqwzeyfgmq380acx |
27 | \xe0e356224fb5a313bbe98353e4ccd02f396baaafafbf85393bdbf680ad | stake_test1ur34vgj0kk338wlfsdf7fnxs9uukh24047lc2wfmm0mgptgq7849c |
28 | \xe0609ffae0a55723de7a69397c77f29c19bf0ef58f2c782877b8abbba1 | stake_test1upsfl7hq54tj8hn6dyuhcaljnsvm7rh43uk8s2rhhz4mhgg7slyup |
29 | \xe010ae59b652be66c7fd26a9dce502e56f3a64d77a531465383b1e30be | stake_test1uqg2ukdk22lxd3lay65aeegzu4hn5exh0ff3gefc8v0rp0sgkn67g |
30 | \xe01cf186c852a7e3644d84fdbe03ef221bd7647470c341bd11e9c90057 | stake_test1uqw0rpkg22n7xezdsn7muql0ygdawer5wrp5r0g3a8ysq4c72ax2e |
31 | \xe021af6d5199f45b1ea76320e598ccbf4e4d38c842e64f53e3280f2a58 | stake_test1uqs67m23n869k848vvswtxxvha8y6wxggtny75lr9q8j5kqgqsq3u |
32 | \xe09f01ce7841ab8eb33bb1baf48462cc9565aba9e85714b4c543827c18 | stake_test1uz0srnncgx4cavemkxa0fprzej2kt2afapt3fdx9gwp8cxqa5qrpn |
(32 rows)
sancho_test_address_filtering=# select * from stake_registration;
id | addr_id | cert_index | epoch_no | tx_id | deposit
----+---------+------------+----------+-------+---------
1 | 1 | 0 | 393 | 1352 | 2000000
2 | 23 | 0 | 437 | 23126 | 2000000
(2 rows)
Should not stake_address contain only those two addresses ?
@Cmdv Should the table instant_reward be present in the table list ? I do not see it.
EDIT: That table was renamed to reward_rest --> https://github.com/IntersectMBO/cardano-db-sync/pull/1869
Looks like there might be issue with reward_rest table - it contains addresses with different ids:
sancho_test_address_filtering=# select * from reward_rest LIMIT 3;
addr_id | type | amount | spendable_epoch | earned_epoch
---------+-----------------+--------------+-----------------+--------------
2 | proposal_refund | 100000000000 | 395 | 394
1 | proposal_refund | 100000000000 | 411 | 410
3 | proposal_refund | 100000000000 | 412 | 411
(3 rows)
sancho_test_address_filtering=# select * from stake_registration;
id | addr_id | cert_index | epoch_no | tx_id | deposit
----+---------+------------+----------+-------+---------
1 | 1 | 0 | 392 | 1238 | 2000000
(1 row)
but in filter there is only 1 address:
"insert_options": {
"shelley": {
"enable": true,
"stake_addresses": ["e0e0d4c1ce0503af576155012f78f5601fe3593a93e4e7690b69813c3b"]
},
},