WARDuino icon indicating copy to clipboard operation
WARDuino copied to clipboard

Published 20 hours ago • verified publisher icon TOPLLab

Tinyjson too large for ESP32s

Open tolauwae opened this issue 2 years ago • 3 comments

The Rhysd's tinyjson is far from tiny enough.

tolauwae avatar Sep 23 '22 12:09 tolauwae

Update: still too little memory left to allocate a Wasm memory page.

Error(./src/Memory/mem.cpp:53): 
Could not allocate 131072 bytes for Module->memory.bytes
abort() was called at PC 0x400d3ec7 on core 0

We are now using jsmn and optimize for size when compiling from Rust. (see Cargo.toml)

Compiling with ESP-IDF gives the following sizes:

Total sizes:
Used static DRAM:   46936 bytes ( 133800 remain, 26.0% used)
      .data size:   43184 bytes
      .bss  size:    3752 bytes
Used static IRAM:   46610 bytes (  84462 remain, 35.6% used)
      .text size:   45583 bytes
   .vectors size:    1027 bytes
Used Flash size :  271035 bytes
      .text     :  193795 bytes
      .rodata   :   76984 bytes
Total image size:  360829 bytes (.bin may be padded larger)
Built target size

Compared to the typical small programs we have been running (such as the default upload.h) we see mostly an increase in the static DRAM usage, from around 8% to 26%. This is due to the larger data size, 43184 bytes compared to around 10k bytes usually.

This root cause is a much larger .wasm file, still 32.2 KB after severe optimisation and using a much smaller library (jsmn). Rust still includes large portions of data in Wasm memory. Many seem to be there for Rust error messages, which will never be printed anyway. This could surely be optimized down tremendously.

Still left to check:

  • [ ] Is there infrastructure for the rust compiler to optimize away much of the static data in Wasm memory?
  • [ ] Is PSRAM being used when using the IDF platform?
  • [x] Code seems to try and allocate 2 memory pages. But the Wasm only lists 1 memory? Wasm program calls memory.grow

reference commit: 5944f12e9fde216f069b31801518e451b3bb6c10

tolauwae avatar Oct 05 '22 08:10 tolauwae

The datastructures generated by jsmn are stored in Wasm linear memory. This probably leads to Rust allocating more memory pages. But this doesn't seem necessary at first glance.

  (export "memory" (memory 0))
  (export "main" (func 12))
  (export "__data_end" (global 1))
  (export "__heap_base" (global 2))
  (elem (;0;) (i32.const 1) func 6 11 31 48 49 4 30 50 64 65 3 83 25 26 27 29 76 34 35 59 69 70 71 72 73 82 79 81 78 80 77 91 55 89 90 92 93)
  (data (;0;) (i32.const 4096) "\0b\00\00\00\01\00\00\00\01\00\00\00\0c\00\00\00weather: iterations=50 runtime: us\0a\00\10\10\00\00 \00\00\000\10\00\00\03\00\00\00\0a    {\0a       \22sensor\22: {\0a            \22value\22: 50.75,\0a            \22last-measurement\22: \222019-04-12T10:22:51\22\0a        }\0a    }\0a    src/bin/sensor.rs\00\00\00\c4\10\00\00\11\00\00\00\1b\00\00\00%\00\00\00Parsed: \e8\10\00\00\08\00\00\00\00\00\00\00 \00\00\00\04\00\00\00\02\00\00\00\00\00\00\00\02\00\00\00\00\00\00\00\03\00\00\00\0d\00\00\00\04\00\00\00\04\00\00\00\0e\00\00\00\0f\00\00\00\10\00\00\00\0d\00\00\00\00\00\00\00\01\00\00\00\11\00\00\00library/alloc/src/raw_vec.rscapacity overflow\00\00\00\5c\11\00\00\11\00\00\00@\11\00\00\1c\00\00\00\05\02\00\00\05\00\00\00a formatting trait implementation returned an errorlibrary/alloc/src/fmt.rs\00\bb\11\00\00\18\00\00\00U\02\00\00\1c\00\00\00..\00\00\e4\11\00\00\02\00\00\00BorrowMutError\00\00\12\00\00\00\00\00\00\00\01\00\00\00\13\00\00\00\ec \00\00\00\00\00\00index out of bounds: the len is  but the index is \00\00\18\12\00\00 \00\00\008\12\00\00\12\00\00\00==assertion failed: `(left  right)`\0a  left: ``,\0a right: ``: ^\12\00\00\19\00\00\00w\12\00\00\12\00\00\00\89\12\00\00\0c\00\00\00\95\12\00\00\03\00\00\00`\00\00\00^\12\00\00\19\00\00\00w\12\00\00\12\00\00\00\89\12\00\00\0c\00\00\00\b8\12\00\00\01\00\00\00: \00\00\ec \00\00\00\00\00\00\dc\12\00\00\02\00\00\00\12\00\00\00\0c\00\00\00\04\00\00\00\14\00\00\00\15\00\00\00\16\00\00\00     {\0a,\0a,  { } { .. } }(\0a()\0a[]0x00010203040506070809101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899\00\00\00\12\00\00\00\04\00\00\00\04\00\00\00\17\00\00\00\18\00\00\00\19\00\00\00truefalserange start index  out of range for slice of length \00\00\00\15\14\00\00\12\00\00\00'\14\00\00\22\00\00\00library/core/src/slice/index.rs\00\5c\14\00\00\1f\00\00\004\00\00\00\05\00\00\00range end index \8c\14\00\00\10\00\00\00'\14\00\00\22\00\00\00\5c\14\00\00\1f\00\00\00I\00\00\00\05\00\00\00slice index starts at  but ends at \00\bc\14\00\00\16\00\00\00\d2\14\00\00\0d\00\00\00\5c\14\00\00\1f\00\00\00\5c\00\00\00\05\00\00\00library/core/src/str/mod.rs[...]byte index  is out of bounds of `\00\00\00 \15\00\00\0b\00\00\00+\15\00\00\16\00\00\00\b8\12\00\00\01\00\00\00\00\15\00\00\1b\00\00\00k\00\00\00\09\00\00\00begin <= end ( <= ) when slicing `\00\00l\15\00\00\0e\00\00\00z\15\00\00\04\00\00\00~\15\00\00\10\00\00\00\b8\12\00\00\01\00\00\00\00\15\00\00\1b\00\00\00o\00\00\00\05\00\00\00\00\15\00\00\1b\00\00\00}\00\00\00-\00\00\00 is not a char boundary; it is inside  (bytes ) of ` \15\00\00\0b\00\00\00\d0\15\00\00&\00\00\00\f6\15\00\00\08\00\00\00\fe\15\00\00\06\00\00\00\b8\12\00\00\01\00\00\00\00\15\00\00\1b\00\00\00\7f\00\00\00\05\00\00\00library/core/src/unicode/printable.rs\00\00\00<\16\00\00%\00\00\00\1a\00\00\006\00\00\00\00\01\03\05\05\06\06\02\07\06\08\07\09\11\0a\1c\0b\19\0c\1a\0d\10\0e\0d\0f\04\10\03\12\12\13\09\16\01\17\04\18\01\19\03\1a\07\1b\01\1c\02\1f\16 \03+\03-\0b.\010\031\022\01\a7\02\a9\02\aa\04\ab\08\fa\02\fb\05\fd\02\fe\03\ff\09\adxy\8b\8d\a20WX\8b\8c\90\1c\dd\0e\0fKL\fb\fc./?\5c]_\e2\84\8d\8e\91\92\a9\b1\ba\bb\c5\c6\c9\ca\de\e4\e5\ff\00\04\11\12)147:;=IJ]\84\8e\92\a9\b1\b4\ba\bb\c6\ca\ce\cf\e4\e5\00\04\0d\0e\11\12)14:;EFIJ^de\84\91\9b\9d\c9\ce\cf\0d\11):;EIW[\5c^_de\8d\91\a9\b4\ba\bb\c5\c9\df\e4\e5\f0\0d\11EIde\80\84\b2\bc\be\bf\d5\d7\f0\f1\83\85\8b\a4\a6\be\bf\c5\c7\ce\cf\da\dbH\98\bd\cd\c6\ce\cfINOWY^_\89\8e\8f\b1\b6\b7\bf\c1\c6\c7\d7\11\16\17[\5c\f6\f7\fe\ff\80mq\de\df\0e\1fno\1c\1d_}~\ae\af\7f\bb\bc\16\17\1e\1fFGNOXZ\5c^~\7f\b5\c5\d4\d5\dc\f0\f1\f5rs\8ftu\96&./\a7\af\b7\bf\c7\cf\d7\df\9a@\97\980\8f\1f\d2\d4\ce\ffNOZ[\07\08\0f\10'/\ee\efno7=?BE\90\91Sgu\c8\c9\d0\d1\d8\d9\e7\fe\ff\00 _\22\82\df\04\82D\08\1b\04\06\11\81\ac\0e\80\ab\05\1f\09\81\1b\03\19\08\01\04/\044\04\07\03\01\07\06\07\11\0aP\0f\12\07U\07\03\04\1c\0a\09\03\08\03\07\03\02\03\03\03\0c\04\05\03\0b\06\01\0e\15\05N\07\1b\07W\07\02\06\16\0dP\04C\03-\03\01\04\11\06\0f\0c:\04\1d%_ m\04j%\80\c8\05\82\b0\03\1a\06\82\fd\03Y\07\16\09\18\09\14\0c\14\0cj\06\0a\06\1a\06Y\07+\05F\0a,\04\0c\04\01\031\0b,\04\1a\06\0b\03\80\ac\06\0a\06/1M\03\80\a4\08<\03\0f\03<\078\08+\05\82\ff\11\18\08/\11-\03!\0f!\0f\80\8c\04\82\97\19\0b\15\88\94\05/\05;\07\02\0e\18\09\80\be\22t\0c\80\d6\1a\0c\05\80\ff\05\80\df\0c\f2\9d\037\09\81\5c\14\80\b8\08\80\cb\05\0a\18;\03\0a\068\08F\08\0c\06t\0b\1e\03Z\04Y\09\80\83\18\1c\0a\16\09L\04\80\8a\06\ab\a4\0c\17\041\a1\04\81\da&\07\0c\05\05\80\a6\10\81\f5\07\01 *\06L\04\80\8d\04\80\be\03\1b\03\0f\0d\00\06\01\01\03\01\04\02\05\07\07\02\08\08\09\02\0a\05\0b\02\0e\04\10\01\11\02\12\05\13\11\14\01\15\02\17\02\19\0d\1c\05\1d\08$\01j\04k\02\af\03\bc\02\cf\02\d1\02\d4\0c\d5\09\d6\02\d7\02\da\01\e0\05\e1\02\e7\04\e8\02\ee \f0\04\f8\02\fa\02\fb\01\0c';>NO\8f\9e\9e\9f{\8b\93\96\a2\b2\ba\86\b1\06\07\096=>V\f3\d0\d1\04\14\1867VW\7f\aa\ae\af\bd5\e0\12\87\89\8e\9e\04\0d\0e\11\12)14:EFIJNOde\5c\b6\b7\1b\1c\07\08\0a\0b\14\1769:\a8\a9\d8\d9\097\90\91\a8\07\0a;>fi\8f\92o_\bf\ee\efZb\f4\fc\ff\9a\9b./'(U\9d\a0\a1\a3\a4\a7\a8\ad\ba\bc\c4\06\0b\0c\15\1d:?EQ\a6\a7\cc\cd\a0\07\19\1a\22%>?\e7\ec\ef\ff\c5\c6\04 #%&(38:HJLPSUVXZ\5c^`cefksx}\7f\8a\a4\aa\af\b0\c0\d0\ae\afno\93^\22{\05\03\04-\03f\03\01/.\80\82\1d\031\0f\1c\04$\09\1e\05+\05D\04\0e*\80\aa\06$\04$\04(\084\0bNC\817\09\16\0a\08\18;E9\03c\08\090\16\05!\03\1b\05\01@8\04K\05/\04\0a\07\09\07@ '\04\0c\096\03:\05\1a\07\04\0c\07PI73\0d3\07.\08\0a\81&RN(\08*\16\1a&\1c\14\17\09N\04$\09D\0d\19\07\0a\06H\08'\09u\0b?A*\06;\05\0a\06Q\06\01\05\10\03\05\80\8bb\1eH\08\0a\80\a6^\22E\0b\0a\06\0d\13:\06\0a6,\04\17\80\b9<dS\0cH\09\0aFE\1bH\08S\0dI\81\07F\0a\1d\03GI7\03\0e\08\0a\069\07\0a\816\19\80\b7\01\0f2\0d\83\9bfu\0b\80\c4\8aLc\0d\84/\8f\d1\82G\a1\b9\829\07*\04\5c\06&\0aF\0a(\05\13\82\b0[eK\049\07\11@\05\0b\02\0e\97\f8\08\84\d6*\09\a2\e7\813-\03\11\04\08\81\8c\89\04k\05\0d\03\09\07\10\92`G\09t<\80\f6\0as\08p\15F\80\9a\14\0cW\09\19\80\87\81G\03\85B\0f\15\84P\1f\80\e1+\80\d5-\03\1a\04\02\81@\1f\11:\05\01\84\e0\80\f7)L\04\0a\04\02\83\11DL=\80\c2<\06\01\04U\05\1b4\02\81\0e,\04d\0cV\0a\80\ae8\1d\0d,\04\09\07\02\0e\06\80\9a\83\d8\05\10\03\0d\03t\0cY\07\0c\04\01\0f\0c\048\08\0a\06(\08\22N\81T\0c\15\03\05\03\07\09\1d\03\0b\05\06\0a\0a\06\08\08\07\09\80\cb%\0a\84\06library/core/src/unicode/unicode_data.rs\00\00\00\dd\1b\00\00(\00\00\00K\00\00\00(\00\00\00\dd\1b\00\00(\00\00\00W\00\00\00\16\00\00\00\dd\1b\00\00(\00\00\00R\00\00\00>\00\00\00Error\00\00\00\00\03\00\00\83\04 \00\91\05`\00]\13\a0\00\12\17 \1f\0c `\1f\ef,\a0+*0 ,o\a6\e0,\02\a8`-\1e\fb`.\00\fe 6\9e\ff`6\fd\01\e16\01\0a!7$\0d\e17\ab\0ea9/\18\a190\1c\e1G\f3\1e!L\f0j\e1OOo!P\9d\bc\a1P\00\cfaQe\d1\a1Q\00\da!R\00\e0\e1S0\e1aU\ae\e2\a1V\d0\e8\e1V \00nW\f0\01\ffW\00p\00\07\00-\01\01\01\02\01\02\01\01H\0b0\15\10\01e\07\02\06\02\02\01\04#\01\1e\1b[\0b:\09\09\01\18\04\01\09\01\03\01\05+\03<\08*\18\01 7\01\01\01\04\08\04\01\03\07\0a\02\1d\01:\01\01\01\02\04\08\01\09\01\0a\02\1a\01\02\029\01\04\02\04\02\02\03\03\01\1e\02\03\01\0b\029\01\04\05\01\02\04\01\14\02\16\06\01\01:\01\01\02\01\04\08\01\07\03\0a\02\1e\01;\01\01\01\0c\01\09\01(\01\03\017\01\01\03\05\03\01\04\07\02\0b\02\1d\01:\01\02\01\02\01\03\01\05\02\07\02\0b\02\1c\029\02\01\01\02\04\08\01\09\01\0a\02\1d\01H\01\04\01\02\03\01\01\08\01Q\01\02\07\0c\08b\01\02\09\0b\06J\02\1b\01\01\01\01\017\0e\01\05\01\02\05\0b\01$\09\01f\04\01\06\01\02\02\02\19\02\04\03\10\04\0d\01\02\02\06\01\0f\01\00\03\00\03\1d\02\1e\02\1e\02@\02\01\07\08\01\02\0b\09\01-\03\01\01u\02\22\01v\03\04\02\09\01\06\03\db\02\02\01:\01\01\07\01\01\01\01\02\08\06\0a\02\010\1f1\040\07\01\01\05\01(\09\0c\02 \04\02\02\01\038\01\01\02\03\01\01\03:\08\02\02\98\03\01\0d\01\07\04\01\06\01\03\02\c6@\00\01\c3!\00\03\8d\01` \00\06i\02\00\04\01\0a \02P\02\00\01\03\01\04\01\19\02\05\01\97\02\1a\12\0d\01&\08\19\0b.\030\01\02\04\02\02'\01C\06\02\02\02\02\0c\01\08\01/\013\01\01\03\02\02\05\02\01\01*\02\08\01\ee\01\02\01\04\01\00\01\00\10\10\10\00\02\00\01\e2\01\95\05\00\03\01\02\05\04(\03\04\01\a5\02\00\04\00\02\99\0b1\04{\016\0f)\01\02\02\0a\031\04\02\02\07\01=\03$\05\01\08>\01\0c\024\09\0a\04\02\01_\03\02\01\01\02\06\01\a0\01\03\08\15\029\02\01\01\01\01\16\01\0e\07\03\05\c3\08\02\03\01\01\17\01Q\01\02\06\01\01\02\01\01\02\01\02\eb\01\02\04\06\02\01\02\1b\02U\08\02\01\01\02j\01\01\01\02\06\01\01e\03\02\04\01\05\00\09\01\02\f5\01\0a\02\01\01\04\01\90\04\02\02\04\01 \0a(\06\02\04\08\01\09\06\02\03.\0d\01\02\00\07\01\06\01\01R\16\02\07\01\02\01\02z\06\03\01\01\02\01\07\01\01H\02\03\01\01\01\00\02\00\05;\07\00\01?\04Q\01\00\02\00.\02\17\00\01\01\03\04\05\08\08\02\07\1e\04\94\03\007\042\08\01\0e\01\16\05\01\0f\00\07\01\11\02\07\01\02\01\05\00\07\00\01=\04\00\07m\07\00`\80\f0\00SomeNone/home/tom/.cargo/registry/src/github.com-1ecc6299db9ec823/jsmn-0.1.5/src/lib.rs\00\00\8b\1f\00\00O\00\00\00\dc\00\00\00&\00\00\00\8b\1f\00\00O\00\00\00\e6\00\00\00\19\00\00\00\8b\1f\00\00O\00\00\00\c8\00\00\00\1f\00\00\00\8b\1f\00\00O\00\00\00\cc\00\00\00.\00\00\00\8b\1f\00\00O\00\00\00\c1\00\00\00\19\00\00\00\8b\1f\00\00O\00\00\00\a3\00\00\00*\00\00\00\8b\1f\00\00O\00\00\00\b4\00\00\00*\00\00\00\8b\1f\00\00O\00\00\00\8b\00\00\00&\00\00\00\8b\1f\00\00O\00\00\00\f3\00\00\00\10\00\00\00Tokenkind\00\00\00\1a\00\00\00\04\00\00\00\04\00\00\00\1b\00\00\00start\00\00\00\1c\00\00\00\04\00\00\00\04\00\00\00\1d\00\00\00endsize\00\1e\00\00\00\04\00\00\00\04\00\00\00\1f\00\00\00PrimitiveStrArrayObjectUndefinedNoMemoryInvalidPart\00already borrowedcalled `Option::unwrap()` on a `None` value\00 \00\00\00\00\00\00\00\01\00\00\00!\00\00\00called `Result::unwrap()` on an `Err` value\00\22\00\00\00\08\00\00\00\04\00\00\00#\00\00\00 \00\00\00\04\00\00\00\04\00\00\00$\00\00\00 \00\00\00\04\00\00\00\04\00\00\00%\00\00\00library/std/src/rt.rs\00\00\00\94!\00\00\15\00\00\00_\00\00\00\0d\00\00\00use of std::thread::current() is not possible after the thread's local data has been destroyedlibrary/std/src/thread/mod.rs\00\1a\22\00\00\1d\00\00\00\a2\02\00\00#\00\00\00failed to generate unique thread ID: bitspace exhausted\00H\22\00\007\00\00\00\1a\22\00\00\1d\00\00\00\10\04\00\00\11\00\00\00\1a\22\00\00\1d\00\00\00\16\04\00\00*\00\00\00\00library/std/src/io/stdio.rs\a9\22\00\00\1b\00\00\00p\02\00\00\13\00\00\00library/std/src/sync/once.rs\d4\22\00\00\1c\00\00\00\14\01\00\002\00\00\00assertion failed: state_and_queue.addr() & STATE_MASK == RUNNINGOnce instance has previously been poisoned\00\00@#\00\00*\00\00\00\02\00\00\00\d4\22\00\00\1c\00\00\00\ff\01\00\00\09\00\00\00\d4\22\00\00\1c\00\00\00\0c\02\00\005\00\00\00PoisonErrorlibrary/std/src/sys_common/thread_info.rs\a3#\00\00)\00\00\00\16\00\00\003\00\00\00\a3#\00\00)\00\00\00+\00\00\00+\00\00\00library/std/src/panicking.rs\ec#\00\00\1c\00\00\00G\02\00\00\1e\00\00\00condvar wait not supported\00\00\18$\00\00\1a\00\00\00library/std/src/sys/wasm/../unsupported/locks/condvar.rs<$\00\008\00\00\00\17\00\00\00\09\00\00\00cannot recursively acquire mutex\84$\00\00 \00\00\00library/std/src/sys/wasm/../unsupported/locks/mutex.rs\00\00\ac$\00\006\00\00\00\17\00\00\00\09\00\00\00library/std/src/sys_common/thread_parker/generic.rs\00\f4$\00\003\00\00\00!\00\00\00&\00\00\00inconsistent park state\008%\00\00\17\00\00\00\f4$\00\003\00\00\00/\00\00\00\17\00\00\00park state changed unexpectedly\00h%\00\00\1f\00\00\00\f4$\00\003\00\00\00,\00\00\00\11\00\00\00inconsistent state in unpark\a0%\00\00\1c\00\00\00\f4$\00\003\00\00\00f\00\00\00\12\00\00\00\f4$\00\003\00\00\00t\00\00\00\1f\00\00\00")
  (data (;1;) (i32.const 9704) "\01\00\00\00\00\00\00\00"))

tolauwae avatar Oct 05 '22 08:10 tolauwae

Benchmarks actually do work with Arduino. I believe this means PSRAM is not being used with IDF compiled code.

Caveat: Arduino crashes after 5 iterations:

Total heap:
275672

Free heap:
248680

Total PSRAM:
4192139

Free PSRAM: 
4192139
LOADED 

START


Free heap:
176608
sensor: iterations=50 runtime: 10734989us
END

LOADED 

START


Free heap:
115428
sensor: iterations=50 runtime: 10885942us
END

LOADED 

START


Free heap:
59708
sensor: iterations=50 runtime: 10887551us
END

LOADED 

START


Free heap:
55340
sensor: iterations=50 runtime: 10882033us
END

LOADED 

START

Free heap:
56104
sensor: iterations=50 runtime: 10885068us
END

assert failed: insert_free_block heap_tlsf.c:239 (current && "free list cannot have a null entry")

Backtrace:0x40083e89:0x3ffb25b00x4008ee05:0x3ffb25d0 0x400941cd:0x3ffb25f0 0x40093552:0x3ffb2720 0x40093bed:0x3ffb2740 0x40093e1a:0x3ffb2760 0x400842ed:0x3ffb2780 0x400941fd:0x3ffb27a0 0x400e6ff2:0x3ffb27c0 0x400d26a9:0x3ffb27e0 0x400f4959:0x3ffb2820 

ELF file SHA256: 0000000000000000
Rebooting...
ets Jul 29 2019 12:21:46

tolauwae avatar Oct 05 '22 11:10 tolauwae