Minic Minic for ESP32 and other microcontrollers

There is any possibility to use Minic engine for non Windows/Linux/Mac operation systems like FreeRTOS? Cheap hardware that I'm trying to use ESP32-WROOM-32D DevKitc: 2x240MHz CPU, 320KB RAM, 4MB Flash 320KB RAM enough for that? Software possibilities that I have here: gcc with std=c++2a and almost full STL with dynamic allocation and threads (Newlib) As I correctly understand engine works like separate process. Can it be used as library instead?

Thanks!

May 11 '22 15:05 vovagorodok

Hi, thanks for your interest in Minic, Based on (almost) full stl support and a recent g++ this might be feasible.

UCI engine are more meant to communicate with pipes (stdin, stdout) than being libraries but it would be very easy to handle calls like a library would indeed.

Wanna play with a NOKIA 5110 ?

May 11 '22 19:05 tryingsomestuff

Wanna play with a NOKIA 5110 ?

Almost :p I'm trying to find brain to some cheap electronic chess board, prototype is in progress. I can record movie at the and.

As I understand all is configured like that

// *** options
#define WITH_UCI    // include or not UCI protocol support
// #define WITH_XBOARD // include or not XBOARB protocol support
// #define WITH_MAGIC  // use magic bitboard or HB ones
// #define WITH_NNUE   // include or not NNUE support
// #define WITH_STATS  // produce or not search statitics
//#define WITH_MPI    // support "distributed" version or not
#ifndef _MSC_VER
#define WITH_SYZYGY // include or not syzgy ETG support
#endif

#define WITHOUT_FILESYSTEM              // some compiler don't support whole std:filesystem

First compilation error is:

lib/Minic/src/definition.hpp:643:21: error: 'aligned_alloc' is not a member of 'std'
    void* ret = std::aligned_alloc(alignment, size);

adding:

#elif defined(INC_FREERTOS_H)
  return malloc(size);

fixes that. Next issue:

lib/Minic/src/attack.hpp:59:161: error: 'empty' was not declared in this scope
 template < Piece > [[nodiscard]] inline BitBoard coverage      (const Square x, const BitBoard occupancy = 0, const Color c = Co_White) { assert(false); return empty; }

A correctly I understand we should prepare another set of flags inside definition.hpp and add #ifdef INC_FREERTOS_H at OS dependent places?

May 11 '22 23:05 vovagorodok

Yes your on the right track indeed.

the "empty" missing is a mistake on my side, as I not often compile the non- magic bitboard attack code. It should be "emptyBitBoard" instead.

-template < Piece > [[nodiscard]] inline BitBoard coverage      (const Square x, const BitBoard occupancy = 0, const Color c = Co_White) { assert(false); return empty; }
+template < Piece > [[nodiscard]] inline BitBoard coverage      (const Square x, const BitBoard occupancy = 0, const Color c = Co_White) { assert(false); return emptyBitBoard; }

May 12 '22 06:05 tryingsomestuff

Thanks, fixed! Looks that it can be last compilation issue:

Dependency Graph
|-- <NimBLE-Arduino> 1.3.8
|-- <StepperDriver> 1.3.1
|-- <Minic>
Building in release mode
Compiling .pio/build/esp32dev/lib71f/Minic/attack.cpp.o
Compiling .pio/build/esp32dev/lib71f/Minic/definition.cpp.o
Compiling .pio/build/esp32dev/lib71f/Minic/distributed.cpp.o
Compiling .pio/build/esp32dev/lib71f/Minic/dynamicConfig.cpp.o
In file included from lib/Minic/src/bitboard.hpp:3,
                 from lib/Minic/src/bitboardTools.hpp:3,
                 from lib/Minic/src/attack.hpp:3,
                 from lib/Minic/src/attack.cpp:1:
lib/Minic/src/attack.cpp: In function 'BitBoard BBTools::rankAttack(BitBoard, Square)':
lib/Minic/src/attack.cpp:139:25: error: 'x' was not declared in this scope
    const int f = SQFILE(x);
                         ^
lib/Minic/src/definition.hpp:187:51: note: in definition of macro 'SQFILE'
 #define SQFILE(s)              static_cast<File>((s)&7)
                                                   ^
*** [.pio/build/esp32dev/lib71f/Minic/attack.cpp.o] Error 1

As I understand x is globał and we should include that

May 12 '22 20:05 vovagorodok

Very sorry again, I don't compile HQBB code very often and some refactoring were thus not well wrote.

Here it's simply SQFILE(s) (in BitBoard rankAttack(const BitBoard occupancy, const Square s) )

BTW, why are you not using magic bitboard ?

May 13 '22 05:05 tryingsomestuff

Unfortunately :):

Dependency Graph
|-- <Minic>
Building in release mode
Compiling .pio/build/esp32dev/libc2b/Minic/egt.cpp.o
Compiling .pio/build/esp32dev/libc2b/Minic/extendedPosition.cpp.o
Compiling .pio/build/esp32dev/libc2b/Minic/material.cpp.o
Compiling .pio/build/esp32dev/libc2b/Minic/moveGen.cpp.o
lib/Minic/Source/egt.cpp:11:10: fatal error: tbprobe.h: No such file or directory

*****************************************************************
* Looking for tbprobe.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:tbprobe.h"
* Web  > https://registry.platformio.org/search?q=header:tbprobe.h
*
*****************************************************************

 #include "tbprobe.h"
          ^~~~~~~~~~~
compilation terminated.
*** [.pio/build/esp32dev/libc2b/Minic/egt.cpp.o] Error 1
lib/Minic/Source/material.cpp: In function 'ScoreType MaterialHash::helperKPK(const Position&, Color, ScoreType, DepthType)':
lib/Minic/Source/material.cpp:191:11: error: 'DynamicConfig' has not been declared
       if (DynamicConfig::armageddon) {
           ^~~~~~~~~~~~~
*** [.pio/build/esp32dev/libc2b/Minic/material.cpp.o] Error 1
lib/Minic/Source/extendedPosition.cpp: In lambda function:
lib/Minic/Source/extendedPosition.cpp:257:13: error: 'NNUEEvaluator' was not declared in this scope
             NNUEEvaluator    evaluator;
             ^~~~~~~~~~~~~
lib/Minic/Source/extendedPosition.cpp:258:18: error: 'struct ExtendedPosition' has no member named 'associateEvaluator'
             extP.associateEvaluator(evaluator);
                  ^~~~~~~~~~~~~~~~~~
lib/Minic/Source/extendedPosition.cpp:258:37: error: 'evaluator' was not declared in this scope
             extP.associateEvaluator(evaluator);
                                     ^~~~~~~~~
lib/Minic/Source/extendedPosition.cpp:258:37: note: suggested alternative: 'evalDanger'
             extP.associateEvaluator(evaluator);
                                     ^~~~~~~~~
                                     evalDanger
lib/Minic/Source/extendedPosition.cpp:259:18: error: 'struct ExtendedPosition' has no member named 'resetNNUEEvaluator'
             extP.resetNNUEEvaluator(extP.evaluator());
                  ^~~~~~~~~~~~~~~~~~
lib/Minic/Source/extendedPosition.cpp:259:42: error: 'struct ExtendedPosition' has no member named 'evaluator'
             extP.resetNNUEEvaluator(extP.evaluator());
                                          ^~~~~~~~~
*** [.pio/build/esp32dev/libc2b/Minic/extendedPosition.cpp.o] Error 1

How configure definition.hpp in order to have minimal version without any dependencies?

Here are draft changes that are done until now:
https://github.com/vovagorodok/Minic/commit/fff591dfab208af94d21d5f59801a4aebc6a3e81
And here is esp32 project where Minic is submodule in lib folder:
https://github.com/vovagorodok/minic_esp32

May 13 '22 17:05 vovagorodok

First, about tbprobe, you'll have to get submodules (git submodule update --init) in order to have fathom, or you can disable (comment) #define WITH_SYZYGY

Then, for some reason I don't get yet, #include "dynamicConfig.hpp" may be missing in egt.cpp, I really wonder how it compiles everyday, but anyway.

And finally, a use of NNUEEvaluator around line 257 in extendedPosition.cpp is not protected by a #ifdef. Should reads like this

#ifdef WITH_NNUE            
            NNUEEvaluator    evaluator;
            extP.associateEvaluator(evaluator);
            extP.resetNNUEEvaluator(extP.evaluator());
#endif

I'll push those little fixes on master tomorrow probably.

May 13 '22 19:05 tryingsomestuff

Nice! Additional fixes was added. List of all chenges: https://github.com/tryingsomestuff/Minic/compare/master...vovagorodok:master

Compilation success! Now I have linking issues:

|-- <Minic>
Building in release mode
Linking .pio/build/esp32dev/firmware.elf
/home/vova/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/esp32dev/libc2b/libMinic.a(minic.cpp.o): in function `init(int, char**)':
/home/vova/PlatformIO/Projects/minic_esp32/lib/Minic/Source/minic.cpp:30: multiple definition of `init(int, char**)'; .pio/build/esp32dev/src/main.cpp.o:/home/vova/PlatformIO/Projects/minic_esp32/src/main.cpp:32: first defined here
/home/vova/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/esp32dev/libc2b/libMinic.a(minic.cpp.o): in function `finalize()':
/home/vova/PlatformIO/Projects/minic_esp32/lib/Minic/Source/minic.cpp:55: multiple definition of `finalize()'; .pio/build/esp32dev/src/main.cpp.o:/home/vova/PlatformIO/Projects/minic_esp32/src/main.cpp:57: first defined here
/home/vova/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/esp32dev/firmware.elf section `.dram0.bss' will not fit in region `dram0_0_seg'
/home/vova/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: DRAM segment data does not fit.
/home/vova/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: DRAM segment data does not fit.
/home/vova/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: section .iram0.vectors VMA [0000000040080000,0000000040080402] overlaps section .dram0.bss VMA [000000003ffc1780,0000000041b4c537]
/home/vova/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: region `dram0_0_seg' overflowed by 28771128 bytes
collect2: error: ld returned 1 exit status
*** [.pio/build/esp32dev/firmware.elf] Error 1

Looks that compiled code take a lot of static memory (overflowed by 28771128 bytes). I have ~1.2MB flash and ~320KB RAM. ~29MB instead 320KB, we almost there :) What can I additionally disable/optiomize/(cat arrays size)/(recursions depth) in order to achieve using less than 320K RAM?

May 13 '22 19:05 vovagorodok

Ok so the link issues are likely related to the fact you are trying to build Minic as a lib (libMinic.a), maybe we'll work on that later.

Using only 320K of RAM is a real challenge. By default Minic is using

Size of TT 96Mb (dynamic allocation)
Size of Pawn TT 10752Kb (dynamic allocation)
Material hash size : 27Mb (static allocation)
KPK table size : 24Kb (static allocation)

Minic can totally disable transposition table and pawn table (I just need to expose this to the user). Let's say KPK table is small enough. But disabling the 27Mb Material hash table is not that easy ... I need a few hours to think of that.

Is 320K, the total RAM or the size available for static allocation ?

May 14 '22 05:05 tryingsomestuff

Wow! great analysis. And have the same numbers as in linking error. If it will succeed, we will have first strong chess engine/library ported to such tinny arduino devices.

I'm using this developing board. It has 512KB RAM and by default 320K allocated for static. System allocates around 30KB with Bluetooth/Wifi stack.

May 14 '22 10:05 vovagorodok

ok so with 512K RAM, I really need to make material table optionnal. I'll work on that this week end.

May 14 '22 10:05 tryingsomestuff

ok got it to work without the material table. The risk is that it'd be unable to play simple KRK or KBNK end-game so i'll also find a way to patch for those two ...

May 14 '22 14:05 tryingsomestuff

Perfecto. I'll align to master then and will see whan happens ..

May 14 '22 14:05 vovagorodok

ok I pushed to master, I think most of your discoveries were fixed in those various commits. You only have to tune definition.hpp to fit your need, and we'll have to discuss how to use Minic in as library in the next days I guess. Feel free to reach on discord (SF, TCEC or openbench server for instance).

May 14 '22 20:05 tryingsomestuff

Aligned!

Linking .pio/build/esp32dev/firmware.elf
/home/vova/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/esp32dev/firmware.elf section `.dram0.bss' will not fit in region `dram0_0_seg'
/home/vova/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: DRAM segment data does not fit.
/home/vova/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: DRAM segment data does not fit.
/home/vova/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: region `dram0_0_seg' overflowed by 115104 bytes
collect2: error: ld returned 1 exit status
*** [.pio/build/esp32dev/firmware.elf] Error 1

Wow, overhead just 115KB it means that Minic with OS now use only ~435KB of static memory Current diff: https://github.com/tryingsomestuff/Minic/compare/master...vovagorodok:master For this moment I just rename minic.cpp to minic.hpp in order to not compile it main.cpp now is 99% copy-pasta of minic.cpp

I think now if we use it as library instead whole project we will achieve success. Currently I'm not discord user, I'll register there

May 14 '22 21:05 vovagorodok

mmm so still 115k to be removed ... let me think of it a bit.

May 15 '22 08:05 tryingsomestuff

I would suggest splitting definition.hpp to definition.hpp and config.hpp (or another name) where will be only configuration definitions. And include config.hpp in definition.hpp

May 15 '22 09:05 vovagorodok

config.hpp pushed.

I was wondering how much the classic killer, history, counter tables, ... cost

info string Info  2022-05-15 15:32:37-346: Size of killer table 1016b
info string Info  2022-05-15 15:32:37-346: Size of history table 16Kb
info string Info  2022-05-15 15:32:37-346: Size of history piece table 1664b
info string Info  2022-05-15 15:32:37-346: Size of history captude table 9Kb
info string Info  2022-05-15 15:32:37-346: Size of history counter table 1352Kb

and it looks like the counter history table is 1.3Mb. So it not that clear to me anymore to what the mentioned 115KB overflow (above 320K) in static memory is refering too

May 15 '22 13:05 tryingsomestuff

Oh sorry, my fault and misunderstanding of dram0_0_seg segment. According to esp32 linker script: this segment is for heap. For static bss/data/stack segment we have 512 - 320 = ~176KB or even memory from 4MB flash is segmented for that. I should better discover this topic. All that time it was dynamic memory segment allocation linker error

May 15 '22 14:05 vovagorodok

Ok, this documentation is more descriptive and means that 320KB is for whole dram segment. bss\dadat\stack is inside this segment and left space is for heap.

Even errors approve this:

.../firmware.elf section `.dram0.bss' will not fit in region `dram0_0_seg'

Then why it doesn't show 1.3MB overhead? I'll take look to the measurements.. Or maybe I disable code of history counter by macro?

May 15 '22 16:05 vovagorodok

i installed esp32 toolchain but I understand I cannot "cross compile" without a device plugged in. So it's hard to investigate.

I added a way to set TT size in kb instead of Mb to reduce future dynamic allocations size but I also see Minic executable itself is already more than 500kb so I looks difficult to fit all this is memory.

May 15 '22 17:05 tryingsomestuff

Hi, I would like to help in the topic:)

Then why it doesn't show 1.3MB overhead?

As I found counter_history, which is part of Searcher is created here https://github.com/tryingsomestuff/Minic/blob/6cf7102dd6a334d11d71018159ce4481043d8ea5/Source/threading.cpp#L43

It would go to heap not to bss. `dram0_0_seg' overflowed by 115240 bytes hast to be caused by some different probably static variable.

May 15 '22 18:05 dkwach

i installed esp32 toolchain but I understand I cannot "cross compile" without a device plugged in. So it's hard to investigate.

it should be possible to compile without esp devide. What kind of error do you have?

May 15 '22 18:05 dkwach

i installed esp32 toolchain but I understand I cannot "cross compile" without a device plugged in. So it's hard to investigate.

it should be possible to compile without esp devide. What kind of error do you have?

none, not tried yet ;)

May 15 '22 18:05 tryingsomestuff

You can compile without device. I'm using Visual Studio Code with PlatformIO extension. And just open folder with example where Minic is in lib folder. And than push build button:

It will automatically install toolchains and dependencies for esp32dev device (as in platformio.ini)

May 15 '22 18:05 vovagorodok

how to you tell platform-io that it has to compile lib/Minic and to find headers in there ?