Optimize predeclared values, refactor runtime into reusable part

[turbolent]

Closes #1822

Description

• Optimize checker/interpreter predeclared values by refactoring redeclaration of variables in activation for each checker/interpreter instantiation to a single activation creation which can be injected and thus reused
• Moved the majority of the standard library implementation from runtime to the stdlib package. Instead of requiring the full runtime.Interface, each value declaration depends on a separate interface. This allows standard library declarations to be declared once/reused, by making the target (runtime.Interface) switchable. It also ensures the declarations are implemented consistently and allows them to be tested individually.
• Refactored the majority of the reusable part of the runtime into an "environment". The environment can be reused across multiple transactions/scripts. Only the interface and storage is re-targeted.
• This change unlocks #1823: In a follow-up PR, the checker and interpreter options will get refactored into a reused configuration.

This is a large refactor. It is best reviewed commit by commit. We should have a review session to go over it together.

---

• [x] Targeted PR against master branch
• [x] Linked to Github issue with discussion and accepted design OR link to spec that describes this work
• [x] Code follows the standards mentioned here
• [x] Updated relevant documentation
• [x] Re-reviewed Files changed in the Github PR explorer
• [x] Added appropriate labels

[codecov[bot]]

Codecov Report

Merging #1825 (d264d5f) into master (674324f) will increase coverage by 0.15%. The diff coverage is 86.20%.

@@            Coverage Diff             @@
##           master    #1825      +/-   ##
==========================================
+ Coverage   77.72%   77.87%   +0.15%     
==========================================
  Files         294      300       +6     
  Lines       62193    62217      +24     
==========================================
+ Hits        48340    48452     +112     
+ Misses      12129    12046      -83     
+ Partials     1724     1719       -5     

Flag	Coverage Δ
unittests	77.87% <86.20%> (+0.15%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
runtime/interpreter/interpreter_import.go	100.00% <ø> (ø)
runtime/repl.go	0.00% <0.00%> (ø)
runtime/runtime.go	87.88% <ø> (+1.70%)	:arrow_up:
runtime/sema/check_import_declaration.go	91.86% <ø> (+2.10%)	:arrow_up:
runtime/sema/elaboration.go	86.48% <ø> (-0.36%)	:arrow_down:
runtime/sema/import.go	100.00% <ø> (+19.04%)	:arrow_up:
runtime/sema/type.go	89.16% <ø> (-0.09%)	:arrow_down:
runtime/stdlib/assert.go	54.54% <0.00%> (-20.46%)	:arrow_down:
runtime/stdlib/bls.go	92.98% <ø> (-0.36%)	:arrow_down:
runtime/stdlib/hashalgorithm.go	89.09% <ø> (-0.57%)	:arrow_down:
... and 53 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[github-actions[bot]]

Cadence Benchstat comparison

This branch with compared with the base branch onflow:master commit 674324f72f84fa87829928a3ac7a79725dd80885 The command for i in {1..N}; do go test ./... -run=XXX -bench=. -benchmem -shuffle=on; done was used. Bench tests were run a total of 7 times on each branch.

Collapsed results for better readability

	old.txt	new.txt
	time/op		delta
CheckContractInterfaceFungibleTokenConformance-2	135µs ± 0%	136µs ± 6%	~	(p=0.755 n=5+7)
ContractInterfaceFungibleToken-2	40.0µs ± 1%	39.7µs ± 1%	~	(p=0.065 n=6+6)
InterpretRecursionFib-2	3.73ms ± 1%	3.50ms ± 2%	−6.16%	(p=0.001 n=7+6)
NewInterpreter/new_interpreter-2	1.18µs ± 1%	1.13µs ± 1%	−4.48%	(p=0.001 n=7+6)
NewInterpreter/new_sub-interpreter-2	2.52µs ± 6%	2.41µs ± 2%	−4.32%	(p=0.004 n=7+7)
ParseArray-2	8.58ms ± 1%	8.62ms ± 1%	~	(p=0.534 n=6+7)
ParseDeploy/byte_array-2	13.0ms ± 1%	13.0ms ± 1%	~	(p=0.836 n=7+6)
ParseDeploy/decode_hex-2	1.14ms ± 2%	1.15ms ± 5%	~	(p=0.383 n=7+7)
ParseFungibleToken/With_memory_metering-2	209µs ± 2%	208µs ± 3%	~	(p=0.535 n=7+7)
ParseFungibleToken/Without_memory_metering-2	159µs ± 0%	160µs ± 2%	~	(p=0.535 n=7+7)
ParseInfix-2	7.85µs ± 3%	7.92µs ± 2%	~	(p=0.209 n=7+7)
QualifiedIdentifierCreation/One_level-2	2.34ns ± 0%	2.34ns ± 0%	~	(p=0.604 n=6+7)
QualifiedIdentifierCreation/Three_levels-2	142ns ± 1%	138ns ± 1%	−3.08%	(p=0.001 n=7+6)
RuntimeFungibleTokenTransfer-2	920µs ±32%	718µs ±26%	~	(p=0.259 n=7+7)
RuntimeResourceDictionaryValues-2	6.33ms ± 3%	5.01ms ± 1%	−20.85%	(p=0.001 n=7+6)
ValueIsSubtypeOfSemaType-2	90.8ns ± 1%	100.7ns ± 1%	+10.91%	(p=0.001 n=6+7)
	alloc/op		delta
CheckContractInterfaceFungibleTokenConformance-2	62.4kB ± 0%	61.7kB ± 0%	−1.18%	(p=0.002 n=6+6)
ContractInterfaceFungibleToken-2	24.4kB ± 0%	24.3kB ± 0%	−0.53%	(p=0.008 n=6+7)
InterpretRecursionFib-2	1.51MB ± 0%	1.51MB ± 0%	~	(p=0.449 n=7+7)
NewInterpreter/new_interpreter-2	928B ± 0%	848B ± 0%	−8.62%	(p=0.001 n=7+7)
NewInterpreter/new_sub-interpreter-2	1.46kB ± 0%	1.36kB ± 0%	−6.59%	(p=0.001 n=7+7)
ParseArray-2	2.81MB ± 2%	2.79MB ± 2%	~	(p=0.383 n=7+7)
ParseDeploy/byte_array-2	4.44MB ± 3%	4.42MB ± 3%	~	(p=0.259 n=7+7)
ParseDeploy/decode_hex-2	214kB ± 0%	214kB ± 0%	+0.07%	(p=0.002 n=6+6)
ParseFungibleToken/With_memory_metering-2	36.3kB ± 0%	36.3kB ± 0%	~	(p=0.851 n=7+7)
ParseFungibleToken/Without_memory_metering-2	36.3kB ± 0%	36.3kB ± 0%	~	(p=0.432 n=6+7)
ParseInfix-2	2.17kB ± 0%	2.17kB ± 0%	~	(p=1.000 n=5+6)
QualifiedIdentifierCreation/One_level-2	0.00B	0.00B	~	(all equal)
QualifiedIdentifierCreation/Three_levels-2	64.0B ± 0%	64.0B ± 0%	~	(all equal)
RuntimeFungibleTokenTransfer-2	185kB ± 1%	119kB ± 1%	−35.40%	(p=0.001 n=7+7)
RuntimeResourceDictionaryValues-2	2.23MB ± 0%	2.27MB ± 0%	+2.13%	(p=0.001 n=7+7)
ValueIsSubtypeOfSemaType-2	48.0B ± 0%	48.0B ± 0%	~	(all equal)
	allocs/op		delta
CheckContractInterfaceFungibleTokenConformance-2	1.03k ± 0%	1.02k ± 0%	−0.78%	(p=0.001 n=7+7)
ContractInterfaceFungibleToken-2	428 ± 0%	426 ± 0%	−0.47%	(p=0.001 n=7+7)
InterpretRecursionFib-2	33.5k ± 0%	33.5k ± 0%	~	(all equal)
NewInterpreter/new_interpreter-2	13.0 ± 0%	12.0 ± 0%	−7.69%	(p=0.001 n=7+7)
NewInterpreter/new_sub-interpreter-2	42.0 ± 0%	41.0 ± 0%	−2.38%	(p=0.001 n=7+7)
ParseArray-2	70.0k ± 0%	70.0k ± 0%	~	(p=0.245 n=7+6)
ParseDeploy/byte_array-2	105k ± 0%	105k ± 0%	~	(p=0.140 n=7+6)
ParseDeploy/decode_hex-2	77.0 ± 0%	77.0 ± 0%	~	(all equal)
ParseFungibleToken/With_memory_metering-2	1.06k ± 0%	1.06k ± 0%	~	(all equal)
ParseFungibleToken/Without_memory_metering-2	1.06k ± 0%	1.06k ± 0%	~	(all equal)
ParseInfix-2	66.0 ± 0%	66.0 ± 0%	~	(all equal)
QualifiedIdentifierCreation/One_level-2	0.00	0.00	~	(all equal)
QualifiedIdentifierCreation/Three_levels-2	2.00 ± 0%	2.00 ± 0%	~	(all equal)
RuntimeFungibleTokenTransfer-2	3.48k ± 0%	2.38k ± 0%	−31.81%	(p=0.001 n=7+7)
RuntimeResourceDictionaryValues-2	37.1k ± 0%	37.0k ± 0%	−0.24%	(p=0.001 n=7+7)
ValueIsSubtypeOfSemaType-2	1.00 ± 0%	1.00 ± 0%	~	(all equal)

[turbolent]

CI is failing for a potentially flaky LS test. Pinged @sideninja

[devbugging]

@turbolent I can't seem to reproduce that flakiness locally. I've created a PR I think might help fix the test issue, but I also haven't detected that during the CI, so making these changes based on the stack from this CI: https://github.com/onflow/cadence/pull/1890

[turbolent]

@sideninja Thank you for investigating. 🙏 I'll re-run the job and see if it shows up again

[devbugging]

@sideninja Thank you for investigating. 🙏 I'll re-run the job and see if it shows up again

Now it's something else?

[turbolent]

@sideninja it's happening again and I can reproduce it locally, even on master. It only happens 1 out of ~5000 calls ... looked into it and can't see how this could ever produce anything else, though testify's mock library uses quite some low-level hacks, hmm.

Given that the LS uses interfaces anyway, maybe just don't use mocking and just implement the interfaces normally?

╭─bastian@bm2 ~/Documents/work/cadence/languageserver ‹master●›
╰─$ go test --count 100 --parallel 8 ./...
?   	github.com/onflow/cadence/languageserver	[no test files]
?   	github.com/onflow/cadence/languageserver/cmd/languageserver	[no test files]
?   	github.com/onflow/cadence/languageserver/conversion	[no test files]
--- FAIL: Test_DeployContract (0.00s)
    --- FAIL: Test_DeployContract/successful_deploy_contract (0.00s)
panic: assert: arguments: Cannot call Get(0) because there are 0 argument(s). [recovered]
	panic: assert: arguments: Cannot call Get(0) because there are 0 argument(s).

goroutine 442 [running]:
testing.tRunner.func1.2({0x103f2ace0, 0x140003f8de0})
	/opt/homebrew/Cellar/go/1.19/libexec/src/testing/testing.go:1396 +0x1c8
testing.tRunner.func1()
	/opt/homebrew/Cellar/go/1.19/libexec/src/testing/testing.go:1399 +0x378
panic({0x103f2ace0, 0x140003f8de0})
	/opt/homebrew/Cellar/go/1.19/libexec/src/runtime/panic.go:884 +0x204
github.com/stretchr/testify/mock.Arguments.Get(...)
	/Users/bastian/go/pkg/mod/github.com/stretchr/[email protected]/mock/mock.go:800
github.com/onflow/cadence/languageserver/integration.(*mockFlowClient).DeployContract(0x140001e9ae0, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1}, {0x140003f2ac8, ...}, ...)
	/Users/bastian/Documents/work/cadence/languageserver/integration/mock_flow_test.go:47 +0x2f8
github.com/onflow/cadence/languageserver/integration.(*commands).deployContract(0x1400042e4e0, {0x14000085f10, 0x3, 0x140003fe540?})
	/Users/bastian/Documents/work/cadence/languageserver/integration/commands.go:241 +0x238
github.com/onflow/cadence/languageserver/integration.Test_DeployContract.func1(0x0?)
	/Users/bastian/Documents/work/cadence/languageserver/integration/commands_test.go:198 +0x3c8
testing.tRunner(0x140005061a0, 0x14000500120)
	/opt/homebrew/Cellar/go/1.19/libexec/src/testing/testing.go:1446 +0x10c
created by testing.(*T).Run
	/opt/homebrew/Cellar/go/1.19/libexec/src/testing/testing.go:1493 +0x300
FAIL	github.com/onflow/cadence/languageserver/integration	0.355s
?   	github.com/onflow/cadence/languageserver/jsonrpc2	[no test files]
ok  	github.com/onflow/cadence/languageserver/protocol	0.087s [no tests to run]
ok  	github.com/onflow/cadence/languageserver/server	3.753s
?   	github.com/onflow/cadence/languageserver/test	[no test files]
FAIL

[turbolent]

@dsainati1 I had another look to re-familiarize myself with that part: The check is not necessary anymore, because GlobalValues and GlobalTypes is not set anymore.

https://github.com/onflow/cadence/pull/1825/files#diff-74d0954852bf407e9259a7c9bb55e8af093f080f00f5e35689c250886c29860aL134-L135

[devbugging]

Can we try merging the PR I made in? I think it could be related to the fact I'm returning nils. https://github.com/onflow/cadence/pull/1890