[FEATURE REQUEST]Add grblHAL support : https://github.com/grblHAL

[luc-github]

As started to discuss here https://github.com/grblHAL/ESP32/issues/39

[terjeio]

An initial update for WebUI 3.0 of the grblHAL WebUI plugin is now available. I am testing with a Nucleo STM32F756 board and the plugin should also work with the iMXRT1062 (Teensy 4.x) driver. Note that the ESP32 grblHAL driver does not use the plugin, I will backport the code changes to that later on.

@dresco is working on a STM32H7xx driver that has ethernet and SD card support, it will be possible to use the plugin for this driver when it is updated to use the latest grblHAL core.

[luc-github]

Ok understood - unfortunatly I have no STM boards, neither Teensy 4.x, but I can emulate webserver and board responses in webUI dev mode

I saw the the https://github.com/grblHAL/Plugin_WebUI but I was thinking was same code in https://github.com/grblHAL/ESP32 so I did not realized it was V3 based - I will have a look

[luc-github]

@terjeio you wrote

my plan to contact you about some issues I have encountered

Can you tell which ones ? I can start by these first

I have added the code base to test and build grblHAL webUI https://github.com/luc-github/ESP3D-WEBUI/commit/bac40f99ee64e1ed310ba924acc120e5f1bcf166

I need some confirmation :

• Is grblHal typo correct ? I saw also different writing like GRBLHal and GRBLHAL

• do you have a logo for grblHal ? I have only found is there and svg version of it ?

• about github link Do you want to go to https://github.com/grblHAL/ ? or target the specific FW like https://github.com/grblHAL/ESP32 or https://github.com/grblHAL/STM32F7xx, in that case it would need extra info in [ESP800] response

• about webUI update I see on esp32 version you cannot update the webUI as do not use the maintenance page and directly flashed the webUI, is that the same with pluggin ?

I will check what play SD job commands you are using, reading https://github.com/grblHAL/core/wiki/SD-Card-implementation,-a-technical-walktrough it seems $F=<filename but I will double check

• I see extra real time commands: https://github.com/grblHAL/core/wiki/For-sender-developers#single-character-real-time-commands so I need to add them in https://github.com/luc-github/ESP3D-WEBUI/blob/3.0/src/targets/CNC/grblHAL/realCommandsTable.js Is that list complete / up to date ?

[terjeio]

Issues encountered:

ESP701 for querying the status of a running job suddenly (randomly) is no longer sent after a running a few. Reloading resumes the querying.

If wrong password is given on login correcting it and resubmitting leads to the USER field in the form data missing.

The default file system is FLASHFS, could this be the first listed in the target supportedFileSystems (in files.js) or perhaps read from the ESP800 response?

https://github.com/luc-github/ESP3D-WEBUI/blob/bac40f99ee64e1ed310ba924acc120e5f1bcf166/src/components/Panels/Files.js#L509-L512

I've changed the code above and it seems to work but not sure if this has any side-effects (other than failure if none are specified):

        if (useUiContextFn.getValue("autoload") && currentFS == "") {
            currentFS = files.supported[0].value
            onSelectFS()
        }

Is grblHal typo correct ? I saw also different writing like GRBLHal and GRBLHAL

"grblHAL" is the correct version. The welcome message from the controller is not correct, but I have been reluctant to change it as it may break senders.

do you have a logo for grblHal

There are a few proposals but none has been decided upon. I have to make my mind up...

Do you want to go to https://github.com/grblHAL/ ?

Yes, I have changed this already.

I see on esp32 version you cannot update the webUI as do not use the maintenance page and directly flashed the webUI, is that the same with pluggin ?

Yes. For the plugin changing this will require a flash filing system. I may add this later. If the WebUI is compiled into the binary then adding index.html.gz to the SD card www folder will override the one from the binary so there is an update path available. More work on this is needed on my end - perhaps adding download of a new version to the www folder as a first step?

I will check what play SD job commands you are using

I am using $F=, I can also use ESP220 (IIRC this is used by WebUI 2.1) but when I check your ESP3D source it seems to me that this (and some other commands) has been repurposed? The commands that may have been reporposed have the help text "??" in this list.

I see extra real time commands: ... Is that list complete / up to date ?

I am not sure it is up to date, I'll have to check. Most can be ignored? Perhaps the most important change is the new Tool state, it would be great if WebUI could support grblHAL style tool change handling. And yes, this works even when running a job from the SD card.

Here are the preferences and target folder I created:

preferences.zip

grblHAL.zip

---

Should grblHAL get its own firmware target name and id?

Some ideas:

Can WebUI fetch the settings labels from the controller? grblHAL has dynamic settings handling - plugins may add new ones, even user defined plugins. In my sender the settings UI is built from information requested from the controller. I can easily add an ESP command for fetching a json formatted list of setting ids and labels (and even descriptions) in a json formatted response.

On a longer term todo list will it be possible to use the WebUI style settings handling? It will require the support for floats and a flag type where bit 0 controls the enable status for the rest.

[luc-github]

ESP701 for querying the status of a running job

Currently this command is for the ESP3D Gcode host not used for running job, why not use the ? like GRBL_ESP32 ? I need to see your implementation, originally ESP command are send by http and answer come back by http too, unlike others commands which use websocket, if polling is stopped it means there is a crash that should be visible in dev console or an error in notifications panel

If wrong password is given on login correcting it and resubmitting leads to the USER field in the form data missing. I need to check but I may did that on purpose to display or not the error message

The default file system is FLASHFS,

This is already the case, no ? Flash is the default because it is the one I know it is present for sure, so not sure what issue you meet

I am using $F=, I can also use ESP220

Yes these commands are used by GRBL_ESP32 which is only supported by WebUI 2.1, and since the GRBL_ESP32 is dropped, I did not maintained them in WebUI 3.0

I am not sure it is up to date, I'll have to check. Most can be ignored

it is just to allow these command to be used by macro using a text descriptor as they are not printable,

I also see some commands : ?,~,!` need to be update by the not printable code to be handled properly in grblHal, so will do

would be great if WebUI could support grblHAL style tool change handling.

You will have to tell me more what you expect for UI and behavior, but I see no issue to support it

target name and id?

The current webUI use grblhal as name, for the ID I did not defined it yet, but because it is not selectable for ESP3D, it is not major issue, here the current list : https://github.com/luc-github/ESP3D/blob/3.0/esp3d/src/include/defines.h#L28-L36 If you are ok to use ID=80 I will set it, it will be used to be sure WebUI match the target FW

Can WebUI fetch the settings labels from the controller? currently it is the case, everything is dynamic, these labels are then translated by translation tool to give better look and correct language

So actually you can add any sections / items you want in settings just follow current syntax, if need more type of settings, just need to define type, type constraint, rendering form and should be ok like others types

[luc-github]

Reading your version now

• preferences content has some changes already, polling commands can have difference frequency
• your issue with Flash FS as first is due to the order put in files.js but in latest version of WebUI order is correct
• changed the play command by $F= as you did
• Added all the modes you added

[luc-github]

I have updated and build the index.html.gz according your changes https://github.com/luc-github/ESP3D-WEBUI/blob/3.0/dist/CNC/GRBLHal/index.html.gz

I still need to update the real commands parts

I have also noticed you do not support the one session feature so many webUI can be open at once, is that a choice ? I won't recommend it

[terjeio]

ESP701 for querying the status of a running job

Currently this command is for the ESP3D Gcode host not used for running job, why not use the ? like GRBL_ESP32 ? I need to see your implementation, originally ESP command are send by http and answer come back by http too, unlike others commands which use websocket, if polling is stopped it means there is a crash that should be visible in dev console or an error in notifications panel

Polling for the real time status continues, only the ESP701 request stops. If I remove the ESP701 request the stream status is not displayed even if it is available in the real-time report. I assume this could be changed to use the real time report instead?

The default file system is FLASHFS,

This is already the case, no ? Flash is the default because it is the one I know it is present for sure, so not sure what issue you meet

It is not present in the controllers using the plugin.

I am using $F=, I can also use ESP220

Yes these commands are used by GRBL_ESP32 which is only supported by WebUI 2.1, and since the GRBL_ESP32 is dropped, I did not maintained them in WebUI 3.0

Ok, this explains it. They should not have been reassigned to a different use then? Or was this a customization made by bdring and his team?

target name and id?

If you are ok to use ID=80 I will set it, it will be used to be sure WebUI match the target FW

If not needed I am ok with using ID=10 as I do not know what it is used for in your code.

Can WebUI fetch the settings labels from the controller?

currently it is the case, everything is dynamic, these labels are then translated by translation tool to give better look and correct language

Ok, what is the ESP command to use for returning those. And to be clear I am talking about the grblHAL settings where those not known by the WebUI does not have useful labels:

preferences content has some changes already, polling commands can have difference frequency

Ok, will update. Sun came out today after a period of rain - so it may take some time. And I will be away for a period starting Tuesday 2nd.

I have also noticed you do not support the one session feature so many webUI can be open at once, is that a choice ?

Definitely not - or at least until an abitration protocol can be added. Only one client should be in control at the time. As a first step I might add support for additional read-only clients - but this is longer term.

[luc-github]

It is not present in the controllers using the plugin.

I need to check if I have added a dependency on ESP800 settings

[ESP800]
{
"cmd": "800",
"status": "ok",
"data": {
 "FWVersion": "3.0.0.a111",
 "FWTarget": "grblhal",
 "FWTargetID": "80",
 "Setup": "Enabled",
 "SDConnection": "direct",
 "SerialProtocol": "raw",
 "Authentication": "Disabled",
 "WebCommunication": "Synchronous",
 "WebSocketIP": "localhost",
 "WebSocketPort": "81",
 "Hostname": "esp3d",
 "WiFiMode": "STA",
 "WebUpdate": "Enabled",
 "FileSystem": "LittleFS",
 "Time": "none",
}
}

grblHAL settings

these are in language pack - they are just translation from commands id : https://github.com/luc-github/ESP3D-WEBUI/blob/3.0/src/targets/CNC/grblHAL/translations/en.json

the one session feature

this feature is actually mostly handled by webUI, the FW just broadcast the last connected ID by websocket , so the old ID webUI automaticaly close - not hard to implement - I may try to do it if you want

I need to setup an esp32 system for grblHal as I am afraid I will do some blind code and may misunderstood some commands

if I replace the plugin files to existing ESP32 code would it work ? any other changes should I do - I can do changes manually and this will let me know also why current version is always in reset error on my board

Or was this a customization made by bdring and his team

In FluidNC they renamed all commands https://github.com/bdring/FluidNC/wiki/FluidNC-Commands-and-Settings

[dresco]

@dresco is working on a STM32H7xx driver that has ethernet and SD card support, it will be possible to use the plugin for this driver when it is updated to use the latest grblHAL core.

Just fyi, have started looking at this. First issue is that I used the STMCubeIDE provided middleware for SD card access (using DMA & SDMMC peripheral) - which uses an older FatFs version. Am looking into what it's going to take to upgrade..

@terjeio should I aim for R0.13c as per F4/F7 etc, or latest R0.14b?

[terjeio]

the one session feature

this feature is actually mostly handled by webUI, the FW just broadcast the last connected ID by websocket , so the old ID webUI automaticaly close - not hard to implement - I may try to do it if you want

Wait... One step at a time please. I have some support issues to clear before I can dig into the details.

if I replace the plugin files to existing ESP32 code would it work ?

No, it is a major refactoring. Ideally I should update the ESP32 driver and the plugin so it could be used directly - I do not like to maintain duplicate code.

why current version is always in reset error on my board

Oops, I forgot to asnwer this, by default grblHAL uses normally closed switches for all inputs. Connect the reset pin to ground and you should be fine.

@dresco - the SD card plugin can work with earlier versions of FatFS, I should have used the version id provided by FatFS but are using the target MCU family for now for detection... If you update aim for the latest version.

[dresco]

@dresco - the SD card plugin can work with earlier versions of FatFS, I should have used the version id provided by FatFS but are using the target MCU family for now for detection... If you update aim for the latest version.

Thanks, was actually the webui plugin that complained, may just be something minor...

[luc-github]

WebUI is updated to a7

• Additional real time commands have been added for grblHal and replaced in WebUI panels accordingly : https://github.com/luc-github/ESP3D-WEBUI/blob/3.0/Memo/realtimecmd.txt
• FlashFS won't be listed/used if in ESP800 the FileSystem is set to none

[ESP800]
{
"cmd": "800",
"status": "ok",
"data": {
 "FWVersion": "3.0.0.a111",
 "FWTarget": "grblhal",
 "FWTargetID": "80",
 "Setup": "Enabled",
 "SDConnection": "direct",
 "SerialProtocol": "raw",
 "Authentication": "Disabled",
 "WebCommunication": "Synchronous",
 "WebSocketIP": "localhost",
 "WebSocketPort": "81",
 "Hostname": "esp3d",
 "WiFiMode": "STA",
 "WebUpdate": "Enabled",
 "FileSystem": "none",
 "Time": "none",
}
}

Now will check : https://github.com/grblHAL/core/wiki/Report-extensions because additional grblHal states are not supported by webUI currently

[luc-github]

about : Can WebUI fetch the settings labels from the controller? it is possible just need to define the format you want to use and I can customize the grblHal panel for settings because it is specific, currently I use the GRBL $$ display but we can imagine another command that do same and add label and constraint for each like for ESP3D settings even section if necessary and it will dynamically build the UI accordingly

[terjeio]

I have just committed an update to the ESP32 driver that uses (parts of) the WebUI plugin. Add preferences.json to the root of the SD card.

I left the SerialProtocol value to Socket in the ESP800 response - not sure ift that is correct but it works for me. There is flash filesystem support for the ESP32, I will change the Filesystem value later to reflect that.

Note that forms based autentication is not yet supported for the ESP32 driver. I'll put improved settings handling down the list until the basic stuff is completed - forms based login and unified SD card handling are the largest items.

And BTW the ESP32 webserver is really slow compared to e.g. the iMXRT1062 controller where WebUI loads in the blink of an eye...

[luc-github]

I have just committed an update to the ESP32 driver that uses (parts of) the WebUI plugin.

Ok thank you I give a try tomorrow - I was out all day today and need to finish some code first

SerialProtocol value to Socket in the [ESP800 response] - it is fine - there is no use currently in WebUI but better to be consistent, if the need popup

Ho yes ESP32 sync webserver is very slow, the test I did on current grblHal is already far faster (it use the idf async http server right ?) I am not surprised on another chip the performances are better - on reprap they claim to have wifi file transfer of 1.2MB/s but they do not use esp webserver ... I have also noticed performances issues related to SD https://github.com/luc-github/ESP3D/issues/576, so I use SDfat instead of ESP32 SD library in ESP3D

thank you for the porting It will be easier for me to experiment - I will feedback you tomorrow

[luc-github]

Hi I have flashed my board with latest git and webui 3.0, very cool job thank you,

so I have some questions :

• Is that a choice to not display the IP at start ? it is painful when board is in STA/Client mode but start in reset and so it is locked , cannot send any command to know the IP, I see mdns is present but no SSDP, so on windows, the only choice is to find info on DHCP server ...

• ESP800

[ESP800]
FW version:1.1f
FW target:grblHAL
FW ID:80
SD connection:direct
Serial protocol:Socket
Authentication:Disabled
Web Communication:Synchronous
Web Socket IP:192.168.2.108
Web Socket Port:81
Hostname:Grbl
WiFi mode:AP
File system:directsd
Hostname:Grbl

Some comments :

• In STA mode - it still display WiFi mode:AP
• File system:directsd directsd is not supported should be none or spiffs or littlefs
• Above status lead me to add a new flag : hostFS, because you mentioned you do not use Flash on some system, so the index.html.gz is hosted on SD (or USB may be in future), so WebUI update, preferences.json should be read/uploaded from/to SD/www, using the new flag hostFS would tell webUI where to load/save preferences.json, what do you think ?

• Settings the settings use old format of 2.1 instead of 3.0, so they cannot be loaded, and webUI raise this error : WebUI and Firmware version are not compatible, WebUI 3.X has new output format for all ESP commands, the same as you used in about page, I enclose both format

3.0.txt 2.X.txt

• About page / [ESP420] the CPU freq is 19 - I guess this is wrong 😄 or it is very slow setting

• $G give me [GC:G0 G54 G17 G21 G90 G94 G49 G98 G50 M5 M9 T0 F0 S0.] what is the meaning of . in S0. ? Form another side, is that normal to have F0 instead of F0.0 as mentioned : https://github.com/gnea/grbl/wiki/Grbl-v1.1-Interface#grbl-response-messages Because I need to know to adjust the regex catching patern to get informations and processing them, and there is not information on https://github.com/grblHAL/core/wiki/Report-extensions page

• New flags / states That is for me to read https://github.com/grblHAL/core/wiki/Report-extensions and adjust each triger / rendering one by one

Sorry to ask so many questions, hope you can answer before you will going to vacations - so I won't bother you 😸

Edit I see: $ES so I may use it instead of $$ to list grblHal settings, is that ok ? change command still seems same $<ID>=<value>, right ? Edit2: if I read well need to send $EG then $ES then $$ to be able to display settings properly ? any reason not to add the value in $ES ? it make a lot of queries and parsing that could at least be simplified by just adding value at last position of $ES line format, sender knowing $ES command should not need the $$, because $$ would be only for old senders for compatibility IMHO, I will handle the multiple queries anyway

[terjeio]

Is that a choice to not display the IP at start ?

Yes, as it cannot be displayed for all the platforms grblHAL supports, see this wiki page. I may add SSDP and/or add the IP address to the full status real-time report to fix this.

This is the latest v3 json version:

ESP800

{"cmd":"800","status":"ok","data":{"FWVersion":"1.1f","FWTarget":"grblHAL","FWTargetID":"80","Setup":"Enabled","SDConnection":"direct","SerialProtocol":"Socket","Authentication":"Enabled","WebCommunication":"Synchronous","WebSocketIP":"10.0.0.22","WebSocketPort":"81","Hostname":"Grbl","WiFiMode":"STA","WebUpdate":"Disabled","FileSystem":"none","Time":"none"}}

Note that the backend supports v2.1 as well as v3.0, I use the version parameter added to the ESP800 command to detect 3.0 and set the mode for all subsequent commands (until a new ESP800 is requested): http://10.0.0.22/command?cmd=[ESP800]json=yes%20version=3.

the CPU freq is 19 - I guess this is wrong

Yep, it is the stepper code clock frequency, I have to add a new HAL value for the MCU clock...

$G ... what is the meaning of . in S0. ?

I guess it is for implying that the value is a float - inherited from legacy Grbl.

Form another side, is that normal to have F0 instead of F0.0

Good question, I have to check my code.

I see $ES so I may use it instead of $$ to list grblHal settings, is that ok ?

No, $ES is for getting information about the settings, not the values. There is an internal API for querying setting details which I use to generate the ESP400 response, for most grblHAL settings this is relatively straight forward. The settings API is designed to allow user defined settings (in plugins or board support code), these are also possible to query and will show up in the $ES response etc.

change command still seems same $<ID>=<value>, right ?

Correct. I try to be backwards compatible with the legacy Grbl specification, at least in spirit.

if I read well need to send $EG then $ES then $$ to be able to display settings properly ?

I would rather like to add a ESP command for querying setting details, same format as ESP400. I can even add most to ESP400 by just looping trough them instead of outputting those that your backend outputs.

any reason not to add the value in $ES ? it make a lot of queries and parsing that could at least be simplified by just adding value at last position of $ES line format, sender knowing $ES command should not need the $$, because $$ would be only for old senders for compatibility IMHO, I will handle the multiple queries anyway

The reason is that $ES is used once at startup (at least in my sender) to build the settings UI. $$ is then used to fetch the values as needed.

[luc-github]

Note that the backend supports v2.1 as well as v3.0, I use the version parameter added to the ESP800 command to detect 3.0 and set the mode for all subsequent commands (until a new ESP800 is requested):

Ok so I see the issue - I have manually sent [ESP800] in terminal to verify settings when checking the File system:directsd issue and that generate the problem

I would rather like to add a ESP command for querying setting details, same format as ESP400

The ESP400 management is not part of specific FW but common to all target ESP3D ESP3DLib GRBL_ESP32 grblHal etc...

if you want to merge all settings in ESP400 like format - I can remove the tab Features for grblHal, and handle these settings in grblHal tab, which give you all flexibility for the additional types, I would prefer not using [ESP400] command to avoid confusion if possible but something like $EX for extended settings display @terjeio What do you think ?

Yes, as it cannot be displayed for all the platforms grblHAL supports, see this wiki page. I may add SSDP and/or add the IP address to the full status real-time report to fix this.

My question was because you already display [MSG:WIFI STA ACTIVE] so adding [MSG: IP XXX.XXX.XXX.XXX] when is possible is still better than nothing by hooking the get Ip event I think IMHO

When locked the ? seems not working so I am not sure putting IP in report is necessary, additionnaly it will add extra info that should not change, and this info is already in $I

Final question : what sender currently support all/most grblHal extra features ? I have tried https://github.com/terjeio/ioSender/releases/download/2.0.40/ioSender.2.0.40.zip but it does not work on my Win11 and My norton AV does not like it, more over it remove each part of the sofware one by one saying there is a problem

This is mostly to have a reference to verify I interpret properly the new states in webUI

[terjeio]

I would prefer not using [ESP400] command to avoid confusion if possible but something like $EX for extended settings display @terjeio What do you think ?

IMO it will be better to add a new ESP command that outputs the information in json format, possibly the same as ESP400 uses as you already have parser/UI builder code for that.

My question was because you already display [MSG:WIFI STA ACTIVE] so adding [MSG: IP XXX.XXX.XXX.XXX] when is possible is still better than nothing by hooking the get Ip event I think IMHO

I can add that but there will still be a problem with some controllers that uses native USB for communication as most of these loses any info sent before a client is connected. Not easy this...

When locked the ? seems not working

It should work - are you sure about this.

Final question : what sender currently support all/most grblHal extra features ?

Only ioSender as far as I know.

... but it does not work on my Win11 and My norton AV does not like it

Is it Norton blocking it? I have not signed the code as the certificate for doing so is expensive.

This is mostly to have a reference to verify I interpret properly the new states in webUI

There is only one new state Tool in the real-time report, Alarm and Run can have substates added if enabled in settings. These substates are added in the same way as for Hold. The $G report has some missing parser states added as well as some new ones.

[luc-github]

IMO it will be better to add a new ESP command

if you really want to use ESP command then keep ESP400, the parser need to be modified for the new types only - the command itself is not an issue if you use json format - the parsing is piece of cake, in all case I will remove the feature tab as it is duplicate with grblHal settings

It should work - are you sure about this.

I have reflashed to test the fixes in webUI I did and I cannot duplicate anymore ... I am stumped...

Is it Norton blocking it?

yes and deleting all files one by one, I have paused Norton and now everything seems working - only the SD tab seems unresponsive but it is working in webuI so - it is not a problem for me

The $G report has some missing parser states added as well as some new ones

Yes I need to add the news states [GC:G0 G54 G17 G21 G90 G94 G49 G98 G50 M5 M9 T0 F0 S0. ] for GrblHal [GC:G0 G54 G17 G21 G90 G94 _______________M5 M9 T0 F0 S0 _] forGRBL

Forget my comment about F0.0, it seems my GRBL fw do also F0instead ofF0.0`

I will check everything - thank you for all feed back and enjoy your vacations, I have everything to work now I think - this will keep me busy for a couple of days ^_^

[terjeio]

IMO it will be better to add a new ESP command

if you really want to use ESP command then keep ESP400, the parser need to be modified for the new types only - the command itself is not an issue if you use json format - the parsing is piece of cake, in all case I will remove the feature tab as it is duplicate with grblHal settings

Yes - I guess this makes it simpler for both me and you. The datatypes not directly supported by WebUI are 2 (exclusive bitfield), 4 (axismask) and 6 (decimal or float if you will, later I might add double precision support in the core as an option...).

Examples in ioSender:

$22 - homing enable (exclusive bitfield) - can be emulated imperfecly by your B type:

Disabled

Enabled

$5 - invert limit pins (axismask)

IMO this can be emulated by your B type so not really unsupported.

Yes I need to add the news states [GC:G0 G54 G17 G21 G90 G94 G49 G98 G50 M5 M9 T0 F0 S0. ] for GrblHal [GC:G0 G54 G17 G21 G90 G94 _______________M5 M9 T0 F0 S0 _] forGRBL

There are many more new ones, here is the source code in report.c and motion modes supported in gcode.h. grblHAL also supports the G59.1, G59.2 and G59.3 offsets.

If you aim for perfection mapping of error and alarm codes to text should be pulled from the controller? Note that these can also be added to by third party code.

[luc-github]

. The datatypes not directly supported by WebUI are 2 (exclusive bitfield), 4 (axismask)

ho yes mask controls are a pain - I had some before and I converted all them to simple booleans, but I will see what I can do

If you aim for perfection mapping of error and alarm codes to text should be pulled from the controller?

No I am not - I will keep this part for then end because anyway all text need to be in translations packs so I currently just refer to code for that

There are many more new ones, here is the source code

Yes saw it - thank you

[terjeio]

I have committed some updates, noatbly forms based login (as used by WebUI 3.0) should now work and I have added IP address to the WIFI STA ACTIVE message.

[luc-github]

Ok will check that - I am still working on sanity check if not Flash is enabled - it was not a case I had planned so I am following every rat holes to be sure I did not forget anything

that bring me back to my question :

[ESP800] File system:"directsd" directsd is not supported should be none or spiffs or littlefs

Above status lead me to add a new flag : hostFS, because you mentioned you do not use Flash on some system, so the index.html.gz is hosted on SD (or USB may be in future), so WebUI update, preferences.json should be read/uploaded from/to SD/www, So add a flag hostFS: "SD/www" in ESP800 answer would tell webUI where to load/save preferences.json, what do you think ?

Note: I have prepared a code to handle Bit Masks, but I will wait your decision of the format of [ESP400] with all settings to implement corresponding controls

[terjeio]

[ESP800] File system:"directsd" directsd is not supported should be none or spiffs or littlefs

none is a bit counterintuitive, could virtual be a more logical option?

Above status lead me to add a new flag : hostFS, because you mentioned you do not use Flash on some system, so the index.html.gz is hosted on SD (or USB may be in future), so WebUI update, preferences.json should be read/uploaded from/to SD/www,

In fact it is possible to fetch the WebUI from flash, but then from a read only file system. I do check if the file is available in the www folder and if so it will used instead.

So add a flag hostFS: "SD/www" in ESP800 answer would tell webUI where to load/save preferences.json, what do you think ?

That could work, even when file system virtualization is implemented. Perhaps hostFiles or hostPath is a better name? Which file system becomes irrelevant when virtualization is implemented?

Just for the fun of I added a few lines to auto push the real time status report (along the $G-parser state report if enabled and changes to the report) and it works!

Just add this code before webui_init() if you want to test it:

static on_execute_realtime_ptr on_execute_realtime;

static void auto_report (sys_state_t state)
{
    static uint32_t ms = 0;

    if(hal.get_elapsed_ticks() >= ms) {
        ms = hal.get_elapsed_ticks() + 3000; //ms
        if(hal.stream.state.webui_connected)
            protocol_enqueue_realtime_command(CMD_STATUS_REPORT);
    }

    on_execute_realtime(state);
}

and in webui_init() add:

    on_execute_realtime = grbl.on_execute_realtime;
    grbl.on_execute_realtime = auto_report;

Change $10 to enable auto-reporting of the parser state:

If this is of interest there should be a way to set the timeout?

[luc-github]

File system:"directsd"

Well the wording is confusing now I should admit, initially it was meant for Flash only, as I did not expected Falsh was not available - I may rename it to FlashFS instead for clarity

In fact it is possible to fetch the WebUI from flash, but then from a read only file system.

For this part it does not really matter where the original webUI is located as server already streamed it and it is loaded on client side, unless you want a read only preferences which make no sense I think - if FS is not accessible it is better to say there is no Flash FS available IMHO, it avoid user confusion and limit user interaction (and unexpected situation)

That could work, even when file system virtualization is implemented. Perhaps hostFiles or hostPath is a better name?

I think hostPath is fine if on SD it would be SD/www as done today, indeed only location is important, not FS actually

If this is of interest there should be a way to set the timeout? well if you put a time out configurable then you can use the timeout to disable the autoreporting using value of timeout to 0 and no need to put a flag in report option, just my 2 cents, another solution is no timeout but send report each time a command is received executed, or status change, but that may be more complex to implement

I will give a try tomorrow to this new version thank you 😸

Edit: I see your / to serve files is on / of SD so hostPath: www should be ok, I must also change the path for SD files to allow download by removing SD/

[luc-github]

Unfortunatly I cannot compile latest git I have raise issue https://github.com/grblHAL/ESP32/issues/40 I found some sanity check are missing but still have issues

I have also did some test on current version to test the upload of preferences.json and found that any SD sub folder cannot be parsed or even have some weird behavior listing the root content sometimes,

Upload to these sub directories seems working as I got [MSG:Upload ok] and I can download the file if I write proper path in browser, but listing is not working and this is not limited to www subdirectory but all subdirectory is that normal ?

EDIT: I found the root cause of parsing issue of subfolders command sent to grblHal : http://192.168.2.108/sdfiles?path=%2Fconfig&action=list&t=1659321435917

give : {"files":[],"path":"/confi","total":"3.71 GB","used":"44.59 MB","occupation":"1","mode":"direct","status":""} which is wrong should be config not confi the last char seems trailed confirmed by double it : http://192.168.2.108/sdfiles?path=%2Fconfigg&action=list&t=1659321435917 which give better output : {"files":[{"name":"pen_position.json","shortname":"pen_position.json","datetime":"","size":"24 B"},{"name":"readme.txt","shortname":"readme.txt","datetime":"","size":"361 B"},{"name":"voice.json","shortname":"voice.json","datetime":"","size":"132 B"},{"name":"wifi.json","shortname":"wifi.json","datetime":"","size":"43 B"},{"name":"index.html.gz","shortname":"index.html.gz","datetime":"","size":"76.05 KB"}],"path":"/config","total":"3.71 GB","used":"44.59 MB","occupation":"1","mode":"direct","status":""}

I notice the entry ,"mode":"direct", what is for ? also "status":"" is empty - is that normal ? should be ok if no issue, no ?

EDIT2: looking at code I found you already fixed the issue in latest git I can see in scard.c , but as I cannot compile I cannot experiment if any other issue

if(pathlen > 1 && path[pathlen - 1] != '/') {
            path[pathlen] = '/';
            path[pathlen + 1] = '\0';
        }

//        char *fullname = ((file_server_data_t *)req->user_ctx)->scratch;

        strcat(strcpy(fullname, path), filename);

[terjeio]

I notice the entry ,"mode":"direct", what is for ?

I do not know, I guess it is from the WebUI 2 reverse engineering I did way back then. To be removed?

also "status":"" is empty - is that normal ? should be ok if no issue, no ?

That is a bug, fixed in the latest commit.

---

I have added the auto real-time report pushing but not yet with a run-time setting. Enable here. Interval is ms.

I have also added check for "webui" subprotocol in the websocket code, IMO this is a more robust way to telling grblHAL that the WebUI is charge than later checking for ESP commands arriving via HTTP. Is this something that you may consider to add when connecting to the websocket? It should work with other implementations as well as the supported subprotocols has to be reported back on connection accepted. The subprotocol could also be used for later versioning to ensure backwards compatibility.

[luc-github]

To be removed?

is it up to you , webUI does not use it, I though it was used for another webUI

I have also [added check for "webui" subprotocol...you may consider to add when connecting to the websocket?

I need to see the impact - any change in general protocol / API will impact

• ESP3D
• embedded page of ESP3D
• WebUI
• ESP3DLib
• embedded page of ESP3DLib

So I need to check carefuly

[terjeio]

is it up to you , webUI does not use it, I though it was used for another webUI

ok, I'll leave it for now.

I need to see the impact - any change in general protocol / API will impact

Should not be any in the short term (if the backend websocket implementation is compliant with RFC6455). It is a matter of just adding the string when connecting. Later it could be used for version checking, if the backend does not support the requested subprotocol then the WebUI could gracefully stop with an error message.

[luc-github]

Ok I have tested latest git and now compilling !!!! Thank you I compiled with :

OPTION(WiFi "Wifi + protocols" ON)
OPTION(SDcard "SD Card Streaming" ON)
OPTION(WebUI "WebUI services" ON)
OPTION(WebAuth "WebUI authentication" OFF)
OPTION(SoftAP "Enable soft AP mode" ON) # WiFi only

I can see IP [MSG:WIFI STA ACTIVE, IP=192.168.2.108]

But I also see a bug related to path, I saw it before but I though the fix mentioned above will fix it also so I did not want to overflow the issue list now can list sub directory :+1: but subdir creation failed due to missing / if I am in www directory and create a subdirectory subdir I got wwwsubdir created at root instead of www/subdir

Also I can see the [ESP400] output is incorrect for password - it is set to byte/boolean and it is sent in clear, I wrote a note that should be up to date ESP3D [ESP400] format.txt if you change the network/sta it won't display the scan network button, this is minor issue - as it will be custom ESP400 I can change the trigger - but once defined it is better not to change until major revision is released

Sorry for path issue - it looks like rat hole issue 😓

I will work on adding the Host path:www support to [ESP800] to check to see if it cover all needs, it should be ready when you will be back

[terjeio]

But I also see a bug related to path, I saw it before but I though the fix mentioned above will fix it also so I did not want to overflow the issue list

Odd as this is working for me - I can create subdirectories no problem. Try clean the project and recompile?

Also I can see the [ESP400] output is incorrect for password - it is set to byte/boolean and it is sent in clear, I wrote a note that should be up to date

Will fix. I have to improve password handling elsewhere too - I want to report passwords in clear when a local client is connected and not if a remote is.

if you change the network/sta it won't display the scan network button, this is minor issue - as it will be custom ESP400 I can change the trigger - but once defined it is better not to change until major revision is released

I'll look into this later - do not change the trigger for now?

[luc-github]

Odd as this is working for me - I can create subdirectories no problem. Try clean the project and recompile?

I use fresh download git according https://github.com/grblHAL/ESP32#how-to-build-using-esp-idf-v43 So using webUI provided no change but config And got issue...

Now I did:

• idf.py clean
• idf.py build
• idf.py -p COM3 flash

and seems indeed make it work ...orz ... weird - I will keep watching, thank you for the feedback

I'll look into this later - do not change the trigger for now?

No need - just to confirm, I wait for your [ESP400] with all settings to work on grbHal panel, there is not need to spend time on $EG $ES $$ if you change to [ESP400] because parser is different, and need to define new type syntax

I will test and adjust webUI for grblHal - enjoy you vaccations ^_^

EDIT: I had a look to your protocol suggestion and indeed it is good idea - I will work on this too

[terjeio]

I have just committted a fix for wrong mapping of password fields and added masking of passwords. I have also fixed the CPU frequency reported by ESP420.

As for adding support for grblHAL datatypes float (decimal) is the most important - best if you can handle formatting/input masking, I know it is not easy if not supported by the framework. I'll try emulation for the others when I am back - I will be offline for a while...

[luc-github]

best if you can handle formatting/input masking

I can support any type so it is not an issue I will update the docs for float / masks and exclusive masks which should be same as previous with on additionnal flag for first bit as exclusive

enjoy your offline time ^_^

[luc-github]

I have updated / wrote the : For new types : ESP400 format.md
It should cover all of your existing extra types

For websocket websocket.md

For connections values ESP800 Format.md

Let me know what you think 😸

[dresco]

@dresco - the SD card plugin can work with earlier versions of FatFS, I should have used the version id provided by FatFS but are using the target MCU family for now for detection... If you update aim for the latest version.

Thanks, was actually the webui plugin that complained, may just be something minor...

Quick update, was a bumpy journey, but I seem to have WebUI 3 running well on my H7 port now..

The compilation issues turned out to be minor - some different variable names etc in the older FatFS version. But I've upgraded to the latest version anyway. That turned out to be the easy bit..

Was then having reliability problems with sdcard access on the Nucleo. This uncovered a new DMA/cache issue that I'd missed previously, and a separate signal integrity issue (using a breakout with the Nucleo) - which was resolved with a slower SDMMC peripheral clock.

But my biggest issue was the httpd service (mostly) failing to send data - which took me some time to debug. Long story short - I needed more Ethernet TX DMA descriptors. The way the http server builds a response with the dynamic headers, it uses up a descriptor for each header + a descriptor for the data.

Then just a final hiccup with needing the -u _printf_float to get the sdcard files listed in WebUI, and is now happily running through imaginary jobs on the bench.. Nice!

[luc-github]

@terjeio I see you are back - I have implemented your suggestion for websocket subprotocol in webUI V3, and wrote all docs for websocket/ ESP800/ ESP400 let me know if you need me to do anything