cadence icon indicating copy to clipboard operation
cadence copied to clipboard

Measure performance impact of compact codec

Open robert-e-davidson3 opened this issue 3 years ago • 3 comments

Criteria for closing:

  • [x] Preliminary performance impact of compact codec vs JSON.
  • [ ] Performance impact with real data.

robert-e-davidson3 avatar Aug 12 '22 18:08 robert-e-davidson3

Preliminary performance impact vs the JSON codec is significant: Bigger speedup and reduction is better for the new codec.
The two tests are with fake values (a string and a resource).
Only the codecs themselves are tested here. The overall performance impact is significantly smaller because encode and decode are only a small part of our overall runtime and memory footprint.

Cadence Value:  "This sentence is composed of exactly fifty-nine characters."
Encoding Time (ns):
	JSON Average:  4205
	JSON Minimum:  791
	JSON Maximum:  322958
	Value Average:  740
	Value Minimum:  208
	Value Maximum:  45208
	Speedup: 5.6824324324324325
Decoding Time (ns):
	JSON Average:  4010
	JSON Minimum:  2458
	JSON Maximum:  79042
	Value Average:  569
	Value Minimum:  250
	Value Maximum:  23042
	Speedup: 7.047451669595782
Size (bytes):
	JSON Average:  88
	JSON Minimum:  88
	JSON Maximum:  88
	Value Average:  64
	Value Minimum:  64
	Value Maximum:  64
	Reduction: 1.375

Cadence Value:  A.0102030405060708.A.12345678.NFT(Awesome?: true)
Encoding Time (ns):
	JSON Average:  2216
	JSON Minimum:  750
	JSON Maximum:  91708
	Value Average:  622
	Value Minimum:  375
	Value Maximum:  2417
	Speedup: 3.562700964630225
Decoding Time (ns):
	JSON Average:  4781
	JSON Minimum:  2333
	JSON Maximum:  60042
	Value Average:  682
	Value Minimum:  416
	Value Maximum:  2916
	Speedup: 7.010263929618769
Size (bytes):
	JSON Average:  139
	JSON Minimum:  139
	JSON Maximum:  139
	Value Average:  65
	Value Minimum:  65
	Value Maximum:  65
	Reduction: 2.1384615384615384

robert-e-davidson3 avatar Sep 05 '22 15:09 robert-e-davidson3

Cadence Value:  A.0102030405060708.A.12345678.NFT(Awesome?: true)
Encoding Time (ns):
	JSON Average:  2216 ~ 3x
	JSON Minimum:  750 ~ 1x
	JSON Maximum:  91708 ~ 120x 

How this can be so crazy ranges like this?

bluesign avatar Sep 05 '22 16:09 bluesign

Cadence Value:  A.0102030405060708.A.12345678.NFT(Awesome?: true)
Encoding Time (ns):
	JSON Average:  2216 ~ 3x
	JSON Minimum:  750 ~ 1x
	JSON Maximum:  91708 ~ 120x 

How this can be so crazy ranges like this?

I ran it on my macbook. It's likely Goland did something intense during that run.

This is definitely a preliminary test to check if the codec works better at all. Getting it to the point where I could test it took so long that I wanted feedback ASAP.

For the proper test, I intend to use a dedicated machine and real-world transactions. Probably my personal laptop because it's a fairly lean Arch box but maybe we have something in CI that's similar to a production environment?

I believe @fxamacker is gathering real-world transactions to test codec performance much more accurately.

robert-e-davidson3 avatar Sep 05 '22 16:09 robert-e-davidson3

Closing in favor of redesigned solution captured in: https://github.com/onflow/cadence/issues/2157

j1010001 avatar Nov 22 '22 18:11 j1010001