go-spacemesh icon indicating copy to clipboard operation
go-spacemesh copied to clipboard

Enable scale codec for poet proofs, sync, weakcoin & transaction result

Open evt opened this issue 2 years ago • 6 comments

Motivation

Part of #3271

Changes

Applied scale codec to:

  • poet proofs (common/types/activation_scale.go)
  • sync request and response (timesync/peersync/sync_scale.go) - types renamed from private to public otherwise scalegen doesn't see them. Also please pay attention to https://github.com/spacemeshos/go-spacemesh/pull/3395#discussion_r942289571
  • weakcoin message (beacon/weakcoin/weak_coin.go) - also please pay attention to https://github.com/spacemeshos/go-spacemesh/pull/3395#discussion_r942360451
  • transaction status (https://github.com/spacemeshos/go-spacemesh/pull/3395/files#diff-4c0b5daa697c4d712de6a8fe0acff70856e0efb555fe61176adeab3f2df4969dR35)
  • p2p handshake ack/message (p2p/handshake/handshake_scale.go) - types renamed from private to public otherwise scalegen doesn't see them
  • p2p server response (p2p/server/server_scale.go) - type renamed from private to public otherwise scalegen doesn't see it
  • peersync request & response (timesync/peersync/sync_scale.go) - types renamed from private to public otherwise scalegen doesn't see them

activation/test_resources/poet.proof has been transcoded from xdr to scale. xdr2scale/scale2xdr tools first added to this PR, then removed as discussed with @dshulyak

Also had to convert certain uint fields to uint8/uint32/uint64 as scale codec doesn't support uint type.

Test Plan

  • [x] This PR does not require configuration changes (e.g., environment variables, GitHub secrets, VM resources)
  • [x] This PR does not affect public APIs
  • [x] This PR does not rely on a new version of external services (PoET, elasticsearch, etc.)
  • [x] This PR does not make changes to log messages (which monitoring infrastructure may rely on)

evt avatar Jul 31 '22 10:07 evt

please remove cmd/poetproof. i doubt that they will be useful on a regular basis. if you need them to be around for some reason, you can store them in your repo or in spacemeshos repo

dshulyak avatar Aug 05 '22 07:08 dshulyak

please remove cmd/poetproof. i doubt that they will be useful on a regular basis. if you need them to be around for some reason, you can store them in your repo or in spacemeshos repo

done

evt avatar Aug 05 '22 08:08 evt

xdr2scale binary needs to be removed from committed files

dshulyak avatar Aug 10 '22 09:08 dshulyak

@evt please update pr with information that helps to review it. i remember that we discussed something on slack, but i literally recall almost nothing from that

for instance why certain types are implemented manually, and what needs to be changed in order to support autogeneration. ideally also open issues with techdebt label so that we won't forget about them.

some things are certainly meant to be updated when post library is updated, comment will be useful what exactly

also a note that in order to support autogen we need to make types public would be helpful, i had to figure it out again, and for @countvonzero it would be even more confusing. we also need an issue to track it

please also doublecheck that uint64 is not casted to uint anywhere (you can cast other part). i found several places, but this is really should not be on me to be on a look out for such changes.

and one remaining bit is that it would be helpful to split such changes into a chain of prs. you have 4 relatively separate changes and there is no reason to merge them into one. it will speed up review

dshulyak avatar Aug 10 '22 09:08 dshulyak

xdr2scale binary needs to be removed from committed files

done, thanks

evt avatar Aug 10 '22 11:08 evt

@evt please update pr with information that helps to review it. i remember that we discussed something on slack, but i literally recall almost nothing from that

for instance why certain types are implemented manually, and what needs to be changed in order to support autogeneration. ideally also open issues with techdebt label so that we won't forget about them.

some things are certainly meant to be updated when post library is updated, comment will be useful what exactly

also a note that in order to support autogen we need to make types public would be helpful, i had to figure it out again, and for @countvonzero it would be even more confusing. we also need an issue to track it

please also doublecheck that uint64 is not casted to uint anywhere (you can cast other part). i found several places, but this is really should not be on me to be on a look out for such changes.

and one remaining bit is that it would be helpful to split such changes into a chain of prs. you have 4 relatively separate changes and there is no reason to merge them into one. it will speed up review

done and noted.

evt avatar Aug 10 '22 16:08 evt

i will merge my change https://github.com/spacemeshos/go-spacemesh/pull/3396 into this one. we can't merge them individually, since poet includes both changes already

dshulyak avatar Aug 12 '22 07:08 dshulyak

bors try

dshulyak avatar Aug 12 '22 07:08 dshulyak

try

Build failed:

bors[bot] avatar Aug 12 '22 08:08 bors[bot]

bors try

dshulyak avatar Aug 12 '22 08:08 dshulyak

there is a bug in this change somewhere:

{"L":"WARN","T":"2022-08-12T04:18:44.099-0400","N":"caef8.atxHandler   ","M":"failed to process atx gossip","node_id":"caef8a59959e07f8cc93e9ddc9df8114e286706ae744fa986375b521a90e9e4b","module":"atxHandler","requestId":"641a377e-4b0f-4f1c-a4a4-1c9798a073cc","errmsg":"received atx with missing references of prev or pos id cc0d3b53e4, 34a418d863, e42dc71574, {errmsg 26 0  fetch referenced atxs: too many elements to decode in collection with scale limit set: (69907023 > 1048576)}","name":"atxHandler"}

dshulyak avatar Aug 12 '22 08:08 dshulyak

try

Build failed:

bors[bot] avatar Aug 12 '22 09:08 bors[bot]

i traced this issue down to the changes in p2p/server. and without them i don't see any other regressions https://github.com/spacemeshos/go-spacemesh/pull/3437/commits/7a6f79aed28d90bb0d31b39f365fb243af31d046#diff-e47af31f7ed1c45fd080d64e4d187881e3d67fc964f8027a17a70aa464f62fab

i think we should remove those changes from this pr, and investigate them separately

dshulyak avatar Aug 12 '22 10:08 dshulyak

bors try

dshulyak avatar Aug 14 '22 15:08 dshulyak

try

Build succeeded:

bors[bot] avatar Aug 14 '22 16:08 bors[bot]

bors merge

dshulyak avatar Aug 14 '22 16:08 dshulyak

Pull request successfully merged into develop.

Build succeeded:

bors[bot] avatar Aug 14 '22 17:08 bors[bot]