candid experiment: modify candid tooling to preserve source occurrences of blob types when parsing and pretty-printing candid sources

Dfx selectively processes handwritten candid to inline imports and/or service arguments. It does so by first parsing, manipulating then pretty-printing the candid ast. During parsing, blob abbreviations are expanded and thus lost when pretty-printing. This is problematic for Motoko which has the dubious feature of using the candid source syntax to determine whether to import Candid blob/vec nat8 are Motoko Blob/[Nat8]. The former are equivalent Candid types, but inequivalent Motoko types.

This PR preserves the blob abbreviations when parsing allowing backends to selectively elide or retain them and Motoko translations to be consistent before and after simple candid AST manipulation.

This implementation assumes that blob is a keyword and can't be used as a Candid identifier (which is probably not entirely true since it can be accessed using a quoted identifier).

IIRC, I extended the intial TypeEnv with an abbrevatation type blob = vec nat8 to avoid changing other Rust type definitions, which Yan regarded as an undesirable breaking change in the Candid API.

Mar 18 '24 18:03 crusso

Name Max Mem (Kb) Encode Decode

blob 4_224 20_466_180 ($\textcolor{red}{0.01\%}$) 12_084_180 ($\textcolor{green}{-0.00\%}$)

btreemap 75_456 4_814_049_311 ($\textcolor{red}{0.00\%}$) 15_381_026_406 ($\textcolor{red}{0.00\%}$)

nns 192 2_331_960 ($\textcolor{red}{2.52\%}$) 14_355_820 ($\textcolor{red}{0.20\%}$)

option_list 576 7_313_201 ($\textcolor{red}{0.03\%}$) 33_889_575 ($\textcolor{red}{0.00\%}$)

text 6_336 28_849_631 ($\textcolor{red}{0.01\%}$) 17_839_969 ($\textcolor{red}{0.00\%}$)

variant_list 128 7_520_738 ($\textcolor{red}{0.02\%}$) 25_530_526 ($\textcolor{red}{0.00\%}$)

vec_int16 16_704 168_592_937 ($\textcolor{red}{0.00\%}$) 1_031_830_254 ($\textcolor{red}{0.00\%}$)

Name	Max Mem (Kb)	Encode	Decode
blob	4_224	20_466_180 ($\textcolor{red}{0.01\%}$)	12_084_180 ($\textcolor{green}{-0.00\%}$)
btreemap	75_456	4_814_049_311 ($\textcolor{red}{0.00\%}$)	15_381_026_406 ($\textcolor{red}{0.00\%}$)
nns	192	2_331_960 ($\textcolor{red}{2.52\%}$)	14_355_820 ($\textcolor{red}{0.20\%}$)
option_list	576	7_313_201 ($\textcolor{red}{0.03\%}$)	33_889_575 ($\textcolor{red}{0.00\%}$)
text	6_336	28_849_631 ($\textcolor{red}{0.01\%}$)	17_839_969 ($\textcolor{red}{0.00\%}$)
variant_list	128	7_520_738 ($\textcolor{red}{0.02\%}$)	25_530_526 ($\textcolor{red}{0.00\%}$)
vec_int16	16_704	168_592_937 ($\textcolor{red}{0.00\%}$)	1_031_830_254 ($\textcolor{red}{0.00\%}$)

Parser cost: 18_479_040 ($\textcolor{red}{0.70\%}$)
Extra args: 3_353_189 ($\textcolor{green}{-0.39\%}$)

Click to see raw report


---------------------------------------------------

Benchmark: blob
  total:
    instructions: 32.55 M (0.01%) (change within noise threshold)
    heap_increase: 66 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    instructions: 20.47 M (0.01%) (change within noise threshold)
    heap_increase: 66 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    instructions: 12.08 M (-0.00%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: text
  total:
    instructions: 46.69 M (0.00%) (change within noise threshold)
    heap_increase: 99 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    instructions: 28.85 M (0.01%) (change within noise threshold)
    heap_increase: 99 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    instructions: 17.84 M (0.00%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_int16
  total:
    instructions: 1.20 B (0.00%) (change within noise threshold)
    heap_increase: 261 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    instructions: 168.59 M (0.00%) (change within noise threshold)
    heap_increase: 261 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    instructions: 1.03 B (0.00%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap
  total:
    instructions: 20.20 B (0.00%) (change within noise threshold)
    heap_increase: 1179 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    instructions: 4.81 B (0.00%) (change within noise threshold)
    heap_increase: 257 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    instructions: 15.38 B (0.00%) (change within noise threshold)
    heap_increase: 922 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: option_list
  total:
    instructions: 41.20 M (0.01%) (change within noise threshold)
    heap_increase: 9 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    instructions: 7.31 M (0.03%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    instructions: 33.89 M (0.00%) (change within noise threshold)
    heap_increase: 9 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: variant_list
  total:
    instructions: 33.05 M (0.01%) (change within noise threshold)
    heap_increase: 2 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    instructions: 7.52 M (0.02%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    instructions: 25.53 M (0.00%) (change within noise threshold)
    heap_increase: 2 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: nns
  total:
    instructions: 35.92 M (0.60%) (change within noise threshold)
    heap_increase: 3 pages (no change)
    stable_memory_increase: 0 pages (no change)

  0. Parsing (scope):
    instructions: 18.48 M (0.70%) (change within noise threshold)
    heap_increase: 3 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    instructions: 2.33 M (regressed by 2.52%)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    instructions: 14.36 M (0.20%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: extra_args
  total:
    instructions: 3.35 M (-0.39%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------
Successfully persisted results to canbench_results.yml

Mar 18 '24 18:03 github-actions[bot]

I’ve encountered a similar issue where there’s a mismatch between Blob and [Nat8] types when dealing with subaccounts in the ledgers. This mismatch seems to happen occasionally and can be quite tricky to debug. a consistent workaround or fix for this would be totally cool.

Aug 14 '24 07:08 successaje