Proposal: Rework of code and adding features

[munzili]

Hallo!

I'm currently working on refactoring the code in some places and adding also new features. As those are some "breaking changes" (topic changes, code setup process changes) i want to discuses them before creating a "final" pull request. The code is not fully tested yet but compiles and runs - so this is a early sate.

I just want to be sure that those changes are OK or maybe as those are breaking changes this should be released under a new version/tag/branch ... At the moment i did not update the documentation directory - will be done when changes should be ok.

After all i'm trying also to add CAN support to read the Rotex/Altherma CAN Bus based on the pyhspu project. Not sure if it gona work but i will give it a try. I also had ideas to refactore the code setup process in generell. At the end there should not be any code changes needed. Just download a precompiled firmware and config the ESP32 over Wifi/Webserver. Would make it simpler for non-programmers to use it.

So far i made those changes:

• added cooling modus control. If correctly connected, both heating and cooling can be controlled
• added ESP32 WROOM dev board
• fixed state remembering for SG1 and SG2
• changed setup process. Instead of modifying a file(setup.h and a altherma*.h definition file), witch is registered in git and creates modification messages/conflicts on pull, user should copy a local version witch gets ignored from git and modify it. This way a pull will not create a conflict. This should be done for setup.h and the desired altherma*.h register file
• extracted EEPROM methods into own header file
• added CAN lib to be used in future version (Read Rotex/Altherma CAN Bus also)
• renamed MySerial to a more clear description SerialX10A
• refactored topic names for callback to have same semantic
• added header makros to avoid recursive include

[munzili]

So, after some time i almosted finished by ideas and features mentioned. My latest commit is a huge one but allmost replaced the whole codebase of the existing ESPAltherma implementation. Following changes have been done

Code Base

• Refactored code into more cpp and hpp files and classes. This allows for faster compilation when changes are done.
• Changed default filesystem to LittleFS with min_spiffs.csv as the default partition.
• Added ESP-IDF as a framework to compile a custom version. ESP32-Arduino is precompiled with an ESP-IDF that doesn't support Rev2 and upwards ESP32 chips, which are needed for 20kbps CAN support for the SJA1000 controller.
• Added ESP32 board support.
• Dropped ArduinoOTA support and replaced it with Web OTA.

Definition Files X10A

• Formatted them to JSON format.
• Language files from the same model will now be generated from a base file. This reduces maintenance of the definition files.
• Added presets for models to be selected.

Definition Files CAN-Bus

• Created a CAN-Bus definition file in JSON format (for now, only HPSU commands have been created).
• Language files from the same model will now be generated from a base file. This reduces maintenance of the definition files.

Config/Setup Process

• All configs are now web-based.
• By default, a standalone WiFi will be generated so the configuration can be changed easily.
• Configuration can be easily changed at any time without the need to recompile and upload the firmware.
• Added more config options.
• Added cooling pin configuration. It is now possible to set a pin/relay for Heating and Cooling mode of the Heatpump.

WebUI

• Supports Ethernet/WiFi and MQTT configuration options. Allows to scan reachable WiFi networks and select them.
• Supports X10A parameter selection and tests.
• Supports CAN-Bus commands selection and tests.
• Supports firmware update over the web.
• Supports WebSerial page for debugging and testing.
• Added automated gzip compression for HTML/CSS/JS files.
• Split config options into different categories.
• Added API call to restart the ESP32.
• Added export and import of configuration.

MQTT

• Allows customized topics.
• Dropped MQTT Serial support (replaced with WebSerial)

CAN-Bus

• Supports CAN-Controller MCP2515, ELM327 and SJA1000.
• Added CAN-Sniffing mode so commands of the Heatpump can be easily analyzed and new definition files created.
• Added readonly mode to ensure Heatpump CAN-Bus not getting disturbed.
• Added different CAN Poll modes (disabled, passive, and auto).
• Custom commands can be added via the web.

X10A

• Allows selection of presets/filtered list of parameters.
• All available X10A commands in a parameters file or from the select list can be prefetched via WebUI and verified. These have correct values.

Todo

• ~~Merge master-branch changes since May 2022.~~
• Update readme to new functions and features.
• Implement CAN-Bus data preview.
• Version the firmware and display it on the WebUI.
• Refactor the code base more to be more SOLID.
• CAN-Controller ELM327 is untested and needs to be tested.
• Homeassistant support not implemented yet. Maybe more smarthome options should be supported (For me Loxone is in the todo list).
• Create a CI script to generate an automated release build with precompiled binaries and definition files. This allows for simple updates by downloading and uploading it on ESPAltherma.

---

The above Todo list is still open, and the code is currently in beta status. I have been running my new code on an ESP32 (specifically the ESP32-WROOM-32E) for some time now, and it has been functioning well with a Rotex HPSU compact that has X10A and CAN (SJA1000) support. I would appreciate any feedback you may have. Additionally, I am wondering how such a significant change could be merged into the master branch. Perhaps it would be better to maintain it as a separate branch or version.

[DieterTHeck]

First feedback: Wow. What is the best way to beta-test it? Compile your :rework files?

[DieterTHeck]

regarding X10a definitons. Clever move to decouple translation and actual addresses. I wonder if it would not be even easier to maintain if we had "vectors" with English, German, French, ... names for the same thing in one row but differnt columns. Better overview, less files. For English take entry[,0], for German entry[,1] etc.

Further feedback once I had time to have an actual look.

[munzili]

To test/tun it:

• Clone repo with platformio
• clone submodules with: git submodule update --init --recursive
• compile and upload it

To generate CAN and X10A definition files (will be created in build directory):

• run script "scripts/build_can_commands.py"
• run script "scripts/build_x10a_commands.py"

[raomin]

Hey @munzili, quite some impressive work in here indeed. Thanks!

I'll need some time to review it and would suggest to split it in smaller PR easier to discuss and merge.

regarding X10a definitons. Clever move to decouple translation and actual addresses. I wonder if it would not be even easier to maintain if we had "vectors" with English, German, French, ... names for the same thing in one row but differnt columns. Better overview, less files. For English take entry[,0], for German entry[,1] etc.

I also would need to have a look but that would mean to load significantly the flash with all the translations. That's significant.

[munzili]

Splitting into smaller pieces for PR could be quite difficult.

The main change is to use web UI to configure and control the system instead of compiling the config into it.

This means setup.h and the definitions can't be .h files anymore. Therefore, a bigger code rewrite is needed to support it. Those changes are just coupled together and can't be created in a separate PR.

What can be split into an own PR is CAN Support and all changes to support it, like including own ESP-IDF and therefore dropping ESP8266 support. But this feature requires a WebUI, so the PR would depend on the first (WebUI) one.

Of course, there are new features added with the web UI which can be created into an own PR, but those would not make sense in my opinion as those changes belong together. These are changes like:

• Export/import config over the web
• Firmware update over the web (back to Arduino OTA other way)
• Webserial support (back to log text over MQTT again)
• Live X10A preview of selected commands

[munzili]

fixed merge (created too much history comments)

[maromme]

wow, that looks like a bunch of work... I've to admit, I can't get it running. The description above doesn't help me much. Is it possible to download a full "version" I just have to compile and upload?

[bzq-pl]

I must admit I couldn't compile :-( I have uploaded recent version of code and during linking I have such error:

---

Linking .pio/build/esp32/firmware.elf */.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/esp32/src/main.o: in function setup()': ###/src/main.cpp:65: undefined reference to fs::LittleFSFS::begin(bool, char const, unsigned char, char const)' ***/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/esp32/src/Config/config.o:(.literal._Z10readConfigv+0x4): undefined reference to LittleFS' ***/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/esp32/src/WebUI/webui.o:(.literal._Z19onWebSerialCallbackPKhj+0x30): undefined reference to fs::LittleFSFS::usedBytes()' ***/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/esp32/src/WebUI/webui.o:(.literal._Z19onWebSerialCallbackPKhj+0x34): undefined reference to fs::LittleFSFS::totalBytes()' ***/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/esp32/src/WebUI/webui.o:(.literal._Z15formatDefaultFSv+0x10): undefined reference to fs::LittleFSFS::end()' */.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/esp32/src/WebUI/webui.o:(.literal._Z15formatDefaultFSv+0x14): undefined reference to fs::LittleFSFS::format()' ***/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/esp32/src/WebUI/webui.o:(.literal._Z15formatDefaultFSv+0x18): undefined reference to fs::LittleFSFS::begin(bool, char const, unsigned char, char const)' ***/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/esp32/src/WebUI/webui.o: in function onWebSerialCallback(unsigned char const*, unsigned int)': ###/src/WebUI/webui.cpp:1005: undefined reference to fs::LittleFSFS::usedBytes()' ***/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: ###/src/WebUI/webui.cpp:1005: undefined reference to fs::LittleFSFS::totalBytes()' ***/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/esp32/src/WebUI/webui.o: in function formatDefaultFS()': ###/src/WebUI/webui.cpp:12: undefined reference to fs::LittleFSFS::end()' ***/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: ###/src/WebUI/webui.cpp:13: undefined reference to fs::LittleFSFS::format()' */.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: ###/src/WebUI/webui.cpp:18: undefined reference to `fs::LittleFSFS::begin(bool, char const, unsigned char, char const)' collect2: error: ld returned 1 exit status *** [.pio/build/esp32/firmware.elf] Error 1

---

Looks like some problem with LittleFSFS library - I have checked and I have this library installed ...

[munzili]

@bzq-pl Please make sure:

• Use "Clean All" inside platformio to get a clean build directory
• run command "git submodule update --init --recursive" in terminal before compile. This will download the LittleFS submodule/ESP-IDF component.
• run "Build" command inside platformio

[bzq-pl]

@munzili thanks!!! - that works! :-)

[munzili]

@maromme where exactly do you have problems? Do you have problems compile it? What board and environment do you use?

[maromme]

@munzili well platformio and using github for more than downloading projects is new to me, there I would need more detailed instructions. I did all my previous projects (a lot on esp32) in arduino IDE. guess I'm lost track of the fancy new possibilities in programming. I started quite a while ago with html without wysiwyg editors and php without big libraries...

[munzili]

@maromme i made a quick and dort readme file update in my rework branch. I updated the sedition Getting started to descript the build process and config options. This is just a fast change, are more clear update inside readme will be done later.

Please take a look if it helps building it.

[maromme]

well, that's getting a bit off topic now: I can't exec "git submodule update --init --recursive". Guess I need some kind extension to interact with github? Besides that, thank's for the explanations. Think I'll be fine with that then

[munzili]

@maromme please check if you have git installed (https://git-scm.com/downloads). Git is needed so the command can be runned. also updated it in the readme files

[ap20132022]

@munzili: thank you very much for this update. It's a major great update, compliment. After some beginning's problem, I installed this pull request to my device. It's looks like it works. Now my questions:

1. firstly I couldn't find the SSID to connect to the ESP. I think and hope that at first ESP allows me to connect (e.g. over Smartphone) to get access to ESP to make some changes (like Tasmota). Is it true? After a while, I change the setup.h file to let ESP connect to my router. It works.
2. I have now a valid IP for my ESP. How can I get access to the webUI? If I type the IP on my Chrome, the website can't be shown. Do I need anything more? Sketch? https://lastminuteengineers.com/esp32-ota-updates-arduino-ide/?utm_content=cmp-true
3. which CAN-Device is recommended? I have already buy an USB-CAN. So it wouldn't work with this ESP? Thank you very much for your help.

[maromme]

@munzili : thank's for your support! I can compile, but get an error. See attached file. I run the git submodule... etc before compiling log (2).txt

edit: maybe it's my "thing": the nice thing about the main branch of @raomin is the simplicity of installing and setup (if you find the right definition file). that opens up the project for users with minimal programming skills. Right now, I've got the feeling that the installing process (compiling and upload) is a bit "trappy". There are many more things that could go wrong. To keep the simplicity, I guess a project like that should be more "closed": no external dependencies etc. If I think of my #273 I don't like the external library as well. Another approach could be pre compiled files that are uploaded with a flash tool for certain boards?

[munzili]

@maromme maybe you miss understood the status of this PR. This PR is in draft status - not able to be merged and marked as work in progress. It's not stable or finished yet and under heave development where changes are made daily. Please do expect problems and errors when using a in development branch that is not stable. You can test it and give me feedback and report problems so i can fix them before marking this PR as ready for review or write some code/things in a different way - like your comment about using external libs. But external libs will be required on such features - be aware that even esp32-arduino and espressiff32 IS an external lib. I don't see problems here - libs are well tested and therefore safer to use.

As already mentioned in a comment above there are still Todos open:

Todo

• ~Merge master-branch changes since May 2022.~
• Update readme to new functions and features.
• Implement CAN-Bus data preview.
• Version the firmware and display it on the WebUI.
• Refactor the code base more to be more SOLID.
• CAN-Controller ELM327 is untested and needs to be tested.
• Homeassistant support not implemented yet. Maybe more smarthome options should be supported (For me Loxone is in the todo list).
• Create a CI script to generate an automated release build with precompiled binaries and definition files. This allows for simple updates by downloading and uploading it on ESPAltherma.

There will be a precompiled firmware ready to be used and automatically generated. But those things are not done yet.

[munzili]

@ap20132022 are you sure you used the correct branch and code? There doesn't exists a setup.h anymore in my "rework" branch. ESPAltherma will generate a WiFi where you can connect to it with your smartphone and do the configs.

[maromme]

I do understand the status. It was just ment as a feedback... As you stated the percompiled firmware is anyway on your todo list the question on the libs is anyway obsolet

[DieterTHeck]

I have some trouble compiling as well.

It breaks here: Linking .pio\build\m5stampc3u\firmware.elf ../.platformio/packages/[email protected]+2021r2-patch5/bin/../lib/gcc/riscv32-esp-elf/8.4.0/../../../../riscv32-esp-elf/bin/ld.exe: .pio\build\m5stampc3u\src\main.o: in function setup()': ..\ESPAltherma/src/main.cpp:65: undefined reference to LittleFS'

Seems to be the same LittleFS issue as above...

I did the following:

• I use VSCode with plattformio + git plugins
• cloned munzili/ESPAltherma
• selected rework branch
• ESP32 env
• I run git submodule update --init --recursive >> multiple times
• i used clean all
• 2 python chripts run as well

The git submodule update does not give any prompt like below. Is that expected? Is there a way to check it ran successfully?

..\ESPAltherma_Test\ESPAltherma>git submodule update --init --recursive [no return whatsoever] ..\ESPAltherma_Test\ESPAltherma>

[ap20132022]

@ap20132022 are you sure you used the correct branch and code? There doesn't exists a setup.h anymore in my "rework" branch. ESPAltherma will generate a WiFi where you can connect to it with your smartphone and do the configs.

thank you very much for your answer. I can connect now to the ESP over Wifi. It looks great. One small problem I got. I use VScode on ubuntu. By compiling, I had some error with X10a and X10A. I had to correkt the pfad. Now I have overall X10A. I will try to find some time to test it more. Have a nice long weekend.

[DieterTHeck]

Hi @munzili

I was able to compile and upload the rework successfully onto an ESP32c3. I had to: a) compile the "feature-webui" branch first - only there "git submodule update --init --recursive" worked. Afterwards the "rework" branch worked as well. b) disable Bluetooth (ESP32c3 just has Bluetooth LE).

I noticed that your "rework" branch does not have a folder "components", while "feature-webui" has. Maybe that was the reason for my initial failures to compile...

Feedback so far:

• why is Bluetooth activated by default, is it necessary for something?

• the standalone WIFI works for me, I could connect to 192.168.1.1

• however, I was not able to set up my WIFI in the Web interface initially:

  • "Select WifI in range" does not seem to finish. After browser reload, I get this error.
  • Manually populating the fields gives me the following error.

  I found that it was due to missing MQTT entries on the next page. That is confusing. Suggest to check for completeness on each page - or to amend the error message to something like: "please check mandatory (=marked red) fields on all Tabs of the config website"

[DieterTHeck]

Further on....

When not configuring the PINS in the first setup, just WiFi and MQTT in my case, the system seems to get stuck after reboot. WebInterface apparently not accessible anymore, so no easy way to correct it.

Here extract from serial monitor:

Connecting to [SSID] .....Connected. IP Address: [IP ADDR] [ 2781][E][esp32-hal-gpio.c:102] __pinMode(): Invalid pin selected E (2818) gpio: gpio_set_intr_type(147): GPIO number error E (2818) gpio: gpio_isr_handler_add(466): GPIO number error Guru Meditation Error: Core 0 panic'ed (Load access fault). Exception was unhandled.

It is likely safer for the standard end user if this is handled somehow, so that I get into the Webinterface and can edit the setting.

[DieterTHeck]

When configuring the GPIO PINs in the WebUI, some are not available. I assume this is due to the standard ESP32 setting. Other ESP32 derivatives may have differnt GPIO Pin. In my case (ESP32C3) I'd prefer to use 20/21 for RX/TX. But 20 is not selectable.

Should this become a bit more generic to allow for broader use?

[munzili]

ap20132022 @DieterTHeck Thanks for your testing and responses!

I fixed some of the build problems in the branch "rework" and "feature-webui":

• X10A filename in webuiBackgroundTasks.hpp. Tested compilation via platformio-cli on Ubuntu
• "submodule not found by git".

• why is Bluetooth activated by default, is it necessary for something?

feature-webui is my development, testing, and experimenting branch (I try to merge only stable code back to rework). I prepared code to test ELM327 via BT connection as I have it not wired with my dev ESP32. There exists an esp32_debug_noBT build target to skip BT build/enabling.

However, I was not able to set up my WIFI in the Web interface initially:

• "Select WifI in range" does not seem to finish. After browser reload, I get this error.
• Manually populating the fields gives me the following error.

I found that it was due to missing MQTT entries on the next page. That is confusing. Suggest checking for completeness on each page - or to amend the error message to something like: "please check mandatory (=marked red) fields on all Tabs of the config website"

Indeed, here still needs to be done some work - at the moment I simply make a javascript "focus()" call, which is a reliquie of my first webui-layout (no tabs). The view should open the correct tab and jump to the mandatory field. I will fix this before finishing PR.

ESP32 WiFi scanning is "not so easy". Sadly the ESP32 disabled his WiFi signal during scan and reenabled it when finished. Most smartphones get disconnected during this time and need to be connected manually again. My webui code looks each 3 seconds if the WiFi is restored. If so, the scanning result will be fetched. The whole process is not so stable but should work - at least on my tests. As you describe, you get an error message at "starting" the Wifi. May you tell me what response you get from http://192.168.1.1/wifi/loadNetworks? What browser do you use and which device?

It is likely safer for the standard end user if this is handled somehow so that I get into the Webinterface and can edit the setting.

When configuring the GPIO PINs in the WebUI, some are not available. I assume this is due to the standard ESP32 setting. Other ESP32 derivatives may have different GPIO Pin. In my case (ESP32C3) I'd prefer to use 20/21 for RX/TX. But 20 is not selectable.

The problem here is that I tried to allow the user to simply select a GPIO from a list instead of enter a number. The list of GPIO is generated from defined ESP32 variants in the code (as you mentioned already). I'm now not sure if this is really a good idea because of the different ESP32 types and GPIOs available. Maybe it's better to let the user enter a number as like now. In your serial log it says "invalid pin" so my list shows a pin number for you that does not exist - therefore it crashes. I will look into this and check if it is possible to "validate" a pin number before use in ESP32.

My current task is to implement the CAN data preview on the web to test the commands and verify if they are working correctly. The commands have been ported from the fhemHPSU and pyHPSU GitHub projects, so they should already be in use and working. However, they still need some verification. Afterward, I will fix the above-mentioned problems.

[DieterTHeck]

hi @munzili

I prepared code to test ELM327 via BT connection There exists an esp32_debug_noBT build target

That's what I saw and copied the flags from into my specific esp32c3 ENV. Is there an alternative library for Bluetooth LE that would do the same (I have no clue about the BT specifics)?

May you tell me what response you get from http://192.168.1.1/wifi/loadNetworks? What browser do you use and which device?

I connected from my PC via Wifi (Win11). I used latest Firefox (with Adblocks etc enabled, but that did not seem to be the problem) and also with a naked Chrome. Both showed the same error. In Firefox the blue "circle" moved for ages without anything happening after clicking on the blue button. After refresh/reload of the website I got the error. In Chrome I got the error message instantly (<0.5 second) after clicking on the blue button. Just tested Apple mobile Phone / Safari. Same thing.

Your link gives this message No change to the text within minutes. No change on refresh.

But maybe this is becasue if have ESP32c3 and no original one. Don't mind if it is just not working for me.

because of the different ESP32 types and GPIOs available. Maybe it's better to let the user enter a number as like now.

Is it possible to use ESP-specific GPIO lists/files, similar to the X10a configuration stuff? It could be selected by having a link between the ENV used for compiling and the GPIO list for that particular ENV/device. I personally think a config page like the one used in tasmota would be most user friendly. Different devices might have different GPIOs and a different default value for RX/TX/Reset. All others set initially to NONE, but one can select CAN+ etc via dropdown. Probably you are familiar with that tasmota config page. You could probalby even embed an exisitng external URL showing the actual physical location of the pins or whatever.

[DieterTHeck]

Just saw that my last suggestion is basically already there... :-)

So my bug above was that I ran into this I guess and did not change it in the UI.

#else
  const String response = "{\"Pins\": {}, \"Default\": {}}";

One more thing. Does this actually work? both ENV cannot be defined at the same time, right?

#if defined(ARDUINO_M5Stick_C)
  const String response =
    "{"
      "\"Pins\": {"
        "\"0\": \"GPIO0 - ADC2_1,TOUCH1\","
        #if defined(ARDUINO_M5Stick_C_Plus)
            "\"25\": \"GPIO25 - ADC2_8,DAC_1\","
        #endif
        "\"26\": \"GPIO26 - ADC2_9,DAC_2\","

[munzili]

That's what I saw and copied the flags from into my specific esp32c3 ENV. Is there an alternative library for Bluetooth LE that would do the same (I have no clue about the BT specifics)?

Well, as mentioned, the BT feature is just being used to test the ELM327. I'm not sure if it will make it to the rework branch because the BT library takes up a lot of flash space. There is only the default "BluetoothSerial" library available in the Arduino framework, and while I have searched for an alternative library that is smaller, I haven't been able to find one

I connected from my PC via Wifi (Win11). I used latest Firefox (with Adblocks etc enabled, but that did not seem to be the problem) and also with a naked Chrome. Both showed the same error. In Firefox the blue "circle" moved for ages without anything happening after clicking on the blue button. After refresh/reload of the website I got the error. In Chrome I got the error message instantly (<0.5 second) after clicking on the blue button. Just tested Apple mobile Phone / Safari. Same thing.

well i use almost the same setting to test it. The response of http://192.168.1.1/wifi/loadNetworks is correct, the state here will not change as this request just "starts" the scan. The "check if has a finished" will be retrieved here http://192.168.1.1/wifi/loadFinished. So, may you please do following steps to test the scan:

1. Reset your ESP32 to get a clean enviroment and connect to your ESP32 WiFi
2. open the URL http://192.168.1.1/wifi/loadNetworks or use the IP you use for your ESP32 (do not click scan on config page before)
3. open the URL http://192.168.1.1/wifi/loadFinished and refresh this page until you get a result. Scan should take max 10 seconds but keep in mind that your PC will get disconnected after the first URL is opened. Check if your PC reconnects to the WiFi signal. If the scan works, you will get a JSON list with available WiFi in the near.

So my bug above was that I ran into this I guess and did not change it in the UI.

#else
  const String response = "{\"Pins\": {}, \"Default\": {}}";

No, you will get some select options with your ESP32 on the web UI, so this #else statement will not be called. It uses the default ESP32 configurations, but your ESP32 variation doesn't have all those pins. Selecting and uploading a file again to work with pins will complicate the setup process, and users may be confused about which file to choose, so this shouldn't be necessary. Instead, I will replace the select option and give the user a textbox with a default value. I cannot create a pin definition for each possible variation, as it would be hard to maintain.

One more thing. Does this actually work? both ENV cannot be defined at the same time, right?

#if defined(ARDUINO_M5Stick_C)
  const String response =
    "{"
      "\"Pins\": {"
        "\"0\": \"GPIO0 - ADC2_1,TOUCH1\","
        #if defined(ARDUINO_M5Stick_C_Plus)
            "\"25\": \"GPIO25 - ADC2_8,DAC_1\","
        #endif
        "\"26\": \"GPIO26 - ADC2_9,DAC_2\","

Yes, this works (as it does in ESPAltherma already). ARDUINO_M5Stick_C is defined in the M5Stick-C board, and ARDUINO_M5Stick_C_Plus will be defined separately if the build target is M5StickCPlus.