hoeken
Porting from ESPAsyncWebServer to PsychicHttp. No AsyncCallbackJsonWebHandler (for now... can add if needed)
I am replacing (trying to ) ESPAsyncWebServer with PsychicHttp on the project on https://github.com/EvEggelen/ESP32-sveltekit. The ESPAsyncWebServer has to many issues, so it would really increase the stability of this framework. However the framework does use AsyncCallbackJsonWebHandler from ESPAsyncWebServer. Would it be possible to add support for this ? Currently it is blocking the porting.
The use can be found on : https://github.com/EvEggelen/ESP32-sveltekit/blob/main/lib/framework/HttpEndpoint.h
Regarding AsyncJsonResponse from ESPAsyncWebServer. I assume you are not planning to support something like AsyncJsonResponse. So best to rewrite this code. Right ?
server.on("/json", HTTP_ANY, [](AsyncWebServerRequest * request) {
AsyncJsonResponse * response = new AsyncJsonResponse();
JsonObject& root = response->getRoot();
root["key1"] = "key number one";
JsonObject& nested = root.createNestedObject("nested");
nested["key1"] = "key number one";
response->setLength();
request->send(response);
});
It should be fairly straightforward to add this functionality.
I basically just ported the minimum for my project and then have been doing the rest of the features as other people request them.
Give me a day or two and I'll have something for ya.
On Sat, Dec 16, 2023, 15:02 EvEggelen @.***> wrote:
I am replacing (trying to ) ESPAsyncWebServer with PsychicHttp on the project on https://github.com/EvEggelen/ESP32-sveltekit. The ESPAsyncWebServer has to many issues, so it would really increase the stability of this framework. However the framework does use AsyncCallbackJsonWebHandler from ESPAsyncWebServer. Would it be possible to add support for this ? Currently it is blocking the porting.
The use can be found on : https://github.com/EvEggelen/ESP32-sveltekit/blob/main/lib/framework/HttpEndpoint.h
Regarding AsyncJsonResponse from ESPAsyncWebServer. I assume you are not planning to support something like AsyncJsonResponse. So best to rewrite this code. Right ?
server.on("/json", HTTP_ANY, [](AsyncWebServerRequest * request) {
AsyncJsonResponse * response = new AsyncJsonResponse(); JsonObject& root = response->getRoot(); root["key1"] = "key number one"; JsonObject& nested = root.createNestedObject("nested"); nested["key1"] = "key number one";
response->setLength(); request->send(response);
});
— Reply to this email directly, view it on GitHub https://github.com/hoeken/PsychicHttp/issues/28, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEHSTOBTWEFSHOUWSDXVDYJX45PAVCNFSM6AAAAABAXZSHNOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA2DIOJTHE4DGMQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>
I started my port today from AsyncWebServer and stumbled across the AsyncJsonResponse which I use a lot. It would be good to have this supported. I'm keeping notes of what I'm migrating so it can be added to the resdme/wiki
It should be fairly straightforward to add this functionality. I basically just ported the minimum for my project and then have been doing the rest of the features as other people request them. Give me a day or two and I'll have something for ya.
that'll be great if you could add.
Okay, just pushed the first pass of the port. A few minor differences from ESPAsync:
- need to return your error value from the callback
- classes renamed to Psychic*
- No PrettyJsonResponse yet. Anyone really need that?
- Still no HTTP_ANY support, so you need to specify your method
- Sends chunked responses if your json is bigger than JSON_BUFFER_SIZE (4k). It will be much slower on already large files. Best to set that #define to your max json file size.
- Handler doesn't take a url (use server.on)
- you need to pass the request pointer into the constructor for the Response object
//JsonResponse example
server.on("/json", HTTP_GET, [](PsychicRequest *request)
{
PsychicJsonResponse response = PsychicJsonResponse(request);
char key[16];
char value[32];
JsonObject root = response.getRoot();
for (int i=0; i<100; i++)
{
sprintf(key, "key%d", i);
sprintf(value, "value is %d", i);
root[key] = value;
}
return response.send();
});
//add in a json handler
PsychicJsonHandler* jsonHandler = new PsychicJsonHandler();
jsonHandler->onRequest([](PsychicRequest *request, JsonVariant &json)
{
JsonObject jsonObj = json.as<JsonObject>();
//what did we get?
String output;
serializeJson(jsonObj, output);
Serial.println(output);
PsychicJsonResponse response = PsychicJsonResponse(request);
JsonObject root = response.getRoot();
root["foo"] = "bar";
return response.send();
});
server.on("/rest/endpoint", HTTP_POST, jsonHandler);
}
Thanks for your commit. Currently I am trying the latest code and continue with my port. I run into an API that seems is not supported. So I would like to see if it is worth adding or you recommend to work around it.
The following code is from : https://github.com/EvEggelen/ESP32-sveltekit/blob/main/lib/framework/HttpEndpoint.h starting at line 121. The code uses the API: request->onDisconnect(..). And that does not seem supported.
void updateSettings(AsyncWebServerRequest *request, JsonVariant &json)
{
if (!json.is<JsonObject>())
{
request->send(400);
return;
}
JsonObject jsonObject = json.as<JsonObject>();
StateUpdateResult outcome = _statefulService->updateWithoutPropagation(jsonObject, _stateUpdater);
if (outcome == StateUpdateResult::ERROR)
{
request->send(400);
return;
}
if (outcome == StateUpdateResult::CHANGED)
{
request->onDisconnect([this]()
{ _statefulService->callUpdateHandlers(HTTP_ENDPOINT_ORIGIN_ID); });
}
AsyncJsonResponse *response = new AsyncJsonResponse(false, _bufferSize);
jsonObject = response->getRoot().to<JsonObject>();
_statefulService->read(jsonObject, _stateReader);
response->setLength();
request->send(response);
}
I am not super familiar with the code of ESPAsyncWebServer. But I do not fully understand when the onDisconnect callback should be called. My assumption is that you can only get at this place of the code updateSettings when the front end requests the URL successfully. The only thing that I can think of is that when the connection is broken between receiving and send request this callback is called. But when look at the code ( https://github.com/theelims/ESP32-sveltekit/blob/main/lib/ESPAsyncWebServer/src/WebRequest.cpp) line 711, I am wondering what happens. Because this onDisconnect is called when response is a nullptr, I do not see that happening without a NULL pointer de-reference ( jsonObject = response->getRoot().to<JsonObject>(); / response->setLength(); ). So maybe I am reading the code wrong. I hope you do know what is intended ( both code is not written by me ) here and how I can do this with PsychicHttp
void AsyncWebServerRequest::send(AsyncWebServerResponse *response){
_response = response;
if(_response == NULL){
_client->close(true);
_onDisconnect();
return;
}
if(!_response->_sourceValid()){
delete response;
_response = NULL;
send(500);
}
else {
_client->setRxTimeout(0);
_response->_respond(this);
}
}
Any help is appreciated.
@EvEggelen I see you're using Rick's esp8266-react library. I'm using a similar library based off this which I've updated in the last years, and also porting over to remove AsyncTCP and AsyncWebServer. Most of the code has been done so you may want to wait a week or two and see how I did it.
@proddy. Thanks for the message. The code I am working on is in indeed derived from esp8266-react library. But not the same. Then I will stop my work ( had already done quite some work :-( ) and see if I can reuse your work.
Did you run into the same issue ? How did you fix it ?
I will be adding the onConnect and onDisconnect callbacks to the web handler classes tomorrow, so hold tight :)
See: https://github.com/hoeken/PsychicHttp/issues/12#issuecomment-1867243131
Thanks. With some code changes it fixed the problem. It seems the intention is to notify the class(s) when the response is send out. Still wondering if that really happens with ESPAsyncWebServer. Maybe I spend the time looking into the code to satisfy my curiosity....
Anyway, hope I can look into the work of proddy soon and see if I can use it to speedup finishing porting the last files....
Thanks. With some code changes it fixed the problem. It seems the intention is to notify the class(s) when the response is send out. Still wondering if that really happens with ESPAsyncWebServer. Maybe I spend the time looking into the code to satisfy my curiosity....
Anyway, hope I can look into the work of proddy soon and see if I can use it to speedup finishing porting the last files....
I'l find some time over the next days to finish my port.
@EvEggelen I finished the port, although still some rough patches to smooth out. I haven't enabled HTTPS/SSL yet or ported the Event Streaming, but that should go quick and smooth. And when that's done I'll do a performance test.
The good news
- I no longer use AsyncWebServer, AsyncJson or AsyncTCP anymore
- the code is lighter and cleaner, and it'll benefit from IDF updates in the future
The not so good news
- It uses 9KB more Heap memory and my max alloc buffer also decreased by 8MB. So need to debug why PsychicHttp is using more heap
My project was based of esp8266-react but has gone through many changes and optimizations in the last 3 years. You may not recognize the core anymore. When you do your port I would start with how I handled registering the URIs in HttpEndpoint.h.
the code is in https://github.com/proddy/EMS-ESP32/tree/https_36
good luck!
@proddy Thanks !. I will look into the code. Give me some time to digest it.
I know that you can configure IDF HTTP server a during runtime and compile time. Maybe it worth checking how it is configure in the code and arduino SDK. Potentially it influences the memory allocation on the heap.
This is what I have in my IDF project for the HTTP server.
#
# HTTP Server
#
CONFIG_HTTPD_MAX_REQ_HDR_LEN=512
CONFIG_HTTPD_MAX_URI_LEN=512
CONFIG_HTTPD_ERR_RESP_NO_DELAY=y
CONFIG_HTTPD_PURGE_BUF_LEN=32
# CONFIG_HTTPD_LOG_PURGE_DATA is not set
CONFIG_HTTPD_WS_SUPPORT=y
# CONFIG_HTTPD_QUEUE_WORK_BLOCKING is not set
# end of HTTP Server
I'd rather use the native IDF library that is supported by platformio/platform-espressif32 and not build myself if possible.
The only settings you can change are what are defined in HTTPD_DEFAULT_CONFIG() which is
#define HTTPD_DEFAULT_CONFIG() { \
.task_priority = tskIDLE_PRIORITY+5, \
.stack_size = 4096, \
.core_id = tskNO_AFFINITY, \
.server_port = 80, \
.ctrl_port = 32768, \
.max_open_sockets = 7, \
.max_uri_handlers = 8, \
.max_resp_headers = 8, \
.backlog_conn = 5, \
.lru_purge_enable = false, \
.recv_wait_timeout = 5, \
.send_wait_timeout = 5, \
.global_user_ctx = NULL, \
.global_user_ctx_free_fn = NULL, \
.global_transport_ctx = NULL, \
.global_transport_ctx_free_fn = NULL, \
.enable_so_linger = false, \
.linger_timeout = 0, \
.open_fn = NULL, \
.close_fn = NULL, \
.uri_match_fn = NULL \
}
I played with increasing the stack size and also changed the task priority to the same thread using server.config.task_priority = uxTaskPriorityGet(nullptr) but it makes a marginal difference.
@proddy Are you sure Psychic uses the defaults ? Did you try changing ".max_open_sockets ". I assume it allocates buffers and context for each connection. As it is run time, I assume it goes to the heap. Just my 2 cents....
I decided not to implement multiple server side events streams and moved to a single web socket connection to save on socket connections. This single socket connection now emulates multiple SSE streams + web sockets. This will help when moving over to HTTPS is needed.
the default settings used are at https://github.com/hoeken/PsychicHttp/blob/26b49f73dca48fa333361fb4a4cfeebca802eeda/src/PsychicHttpServer.cpp#L21
and these modified
config.global_user_ctx = this;
config.global_user_ctx_free_fn = destroy;
config.max_uri_handlers = 20;
I'll experiment with the #sockets and also enabling ENABLE_ASYNC
Enabling ENABLE_ASYNC didn't work for me, it crashed on the handler in async_worker.cpp.