arduino-esp32
arduino-esp32 copied to clipboard
espressif
Modify to "String HTTPClient::getString(size_t size)"
Related area
Significantly reduce delay
Hardware specification
All ESP32 boards
Is your feature request related to a problem?
In cases where Server sends no Content-Length header, the above getString() function gets "_size=-1", and will wait as long as 2 minutes before returning.
Describe the solution you'd like
Further to: https://github.com/espressif/arduino-esp32/issues/2667
re. https://github.com/espressif/arduino-esp32/blob/master/libraries/HTTPClient/src/HTTPClient.h and https://github.com/espressif/arduino-esp32/blob/master/libraries/HTTPClient/src/HTTPClient.cpp
In cases where Server sends no Content-Length header, the above getString() function gets "_size=-1", and will wait as long as 2 minutes before returning.
eg. URL=http://api.openweathermap.org/data/2.5/weather?lon=-1.264&lat=52.371&units=metric&lang=en&appid=???
(The http.setConnectTimeout(3000) and http.setTimeout(3000) commands do not change this.)
A fix for this is to set a limit on the minimum data size you need (ie. the maximum to receive) as follow:
Modify "HTTPClient.h" to say: String getString(size_t size=0);
Modify / add 1 line to "HTTPClient.cpp" to:
String HTTPClient::getString(size_t size) {
if ((_size==-1) && (size!=0)) _size=size;
...
This allows existing "String s=http.getString()" commands to still work fine.
But will also allow the new "String s=http.getString(500)" command which for the above URL example returns the data I need and reduces the wait time to under 1 second.
Although this modification is not as universal as implementing a selectable-timeout on this function, it is far simpler.
I can write a PR for this if someone is able to consider approving/commiting it?
Describe alternatives you've considered
No response
Additional context
No response
I have checked existing list of Feature requests and the Contribution Guide
- [X] I confirm I have checked existing list of Feature requests and Contribution Guide.
HTTPClient::writeToStream does call functions from _client, which has its own timeout.
So it is a bit strange it will not obey those timeouts.
@Jason2866: Sadly the "http.setTimeout(1000)" still does not work for "http.getString()" (which instead times out after 120 seconds).
{I am using "https://github.com/espressif/arduino-esp32/tree/3.2.0-RC2" as at 16Mar25.) (I am using an ESP32-Dev-Module.)
The below sketch demonstrates this:
#include <WiFi.h>
#include <HTTPClient.h>
const char *WIFI_AP_SSID="????????";
const char *WIFI_AP_PW="????????";
String weatherAPIKey = String("0123456789abcdef0123456789abcdef"); // use your own API key
String weatherURL = String("http://api.openweathermap.org/data/2.5/weather?lon=-1.000&lat=50.000"); // open weather api
String weatherUnit = String("metric");
String weatherLang = String("en");
void setup() {
Serial.begin(115200); delay(3000);
Serial.println("Booting...");
WiFi.begin(WIFI_AP_SSID,WIFI_AP_PW);
if (WL_CONNECTED == WiFi.waitForConnectResult()) { // attempt to connect for 10s
Serial.print("Connected to: "); Serial.print(WiFi.SSID());
Serial.print(", Local IP="); Serial.println(WiFi.localIP());
getWeatherData(weatherUnit, weatherLang, weatherURL, weatherAPIKey);
} else Serial.println("No Wifi!");
}
void loop() {}
// https://github.com/espressif/arduino-esp32/issues/10375
// Modify "HTTPClient.h" to say: String getString(size_t size=0);
// Modify/add-1-line to "HTTPClient.cpp" to:
// String HTTPClient::getString(size_t size) {
// if ((_size==-1) && (size!=0)) _size=size;
// if...
void getWeatherData(String units, String lang, String url, String apiKey) {
HTTPClient http; // Use Weather API for live data if WiFi is connected
http.setConnectTimeout(3000); // 3 second max timeout
String weatherQueryURL = url + String("&units=") + units + String("&lang=") + lang + String("&appid=") + apiKey;
Serial.print("weatherQueryURL="); Serial.println(weatherQueryURL);
http.setTimeout(1000); // doesn't seem to work!
http.begin(weatherQueryURL.c_str());
int httpResponseCode = http.GET();
Serial.printf("httpResponseCode=%i\n",httpResponseCode);
if (httpResponseCode == 200) {
Serial.println("Asking for getString() [takes 2mins, but should take ~1ms]...");
// http.setTimeout(1000); // doesn't seem to work!
uint32_t mstart=millis();
String payload=http.getString(); // "Time=121367ms" which is about 2 minutes
//String payload=http.getString(500); // "Time=1ms" // can use if we modify "HTTPClient.h"/"HTTPClient.cpp" as detailed above
Serial.printf("Time=%ums\n",millis()-mstart);
Serial.println("Got\n---"); Serial.println(payload);
http.end();
} else Serial.println("Bad Response (we were expecting 200)");
}
Closing since issue cant not be reproduced with Arduino Core 3.2.0. Using the provided example sketch with a valid API key results in.
rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:4660
load:0x40078000,len:15568
ho 0 tail 12 room 4
load:0x40080400,len:4
load:0x40080404,len:3152
entry 0x400805a0
Booting...
Connected to: Jason2866, Local IP=192.168.2.156
weatherQueryURL=http://api.openweathermap.org/data/2.5/weather?lon=-1.000&lat=50.000&units=metric&lang=en&appid=removed
httpResponseCode=200
Asking for getString() [takes 2mins, but should take ~1ms]...
Time=0ms
Got
---
{"coord":{"lon":-1,"lat":50},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}],"base":"stations","main":{"temp":8.72,"feels_like":4.42,"temp_min":8.72,"temp_max":8.72,"pressure":1018,"humidity":74,"sea_level":1018,"grnd_level":1018},"visibility":10000,"wind":{"speed":10.44,"deg":316,"gust":11.69},"clouds":{"all":2},"dt":1743195086,"sys":{"country":"FR","sunrise":1743140948,"sunset":1743186523},"timezone":0,"id":3016536,"name":"Gatteville-le-Phare","cod":200}
@Jason2866: I still get a 2minute responce time (as at 10Apr25):
I am using "https://github.com/espressif/arduino-esp32/tree/3.2.0-RC2" as at 10Apr25. (I've also tried with "https://github.com/espressif/arduino-esp32/tree/3.2.0" but get pretty identical results.)
I am using an ESP32-Dev-Module. I am using: #define OPENWEATHERMAP_APIKEY "f058fe1cad2afe8e2ddc5d063a64cecb" // use your own API key
My SSID (test1) is a mobile-hotspot connected to a 4G-LTE network (O2 in the UK).
-------------------------------------- Simple Output ---------------------------------------
Booting...
Connected to: test1, Local IP=192.168.51.32
weatherQueryURL=http://api.openweathermap.org/data/2.5/weather?lon=-1.000&lat=50.000&units=metric&lang=en&appid=f058fe1cad2afe8e2ddc5d063a64cecb
httpResponseCode=200
Asking for getString() [takes 2mins, but should take ~1ms]...
Time=121029ms
Got
---
1f2
{"coord":{"lon":-1,"lat":50},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04d"}],"base":"stations","main":{"temp":9.24,"feels_like":6.05,"temp_min":9.24,"temp_max":9.24,"pressure":1029,"humidity":79,"sea_level":1029,"grnd_level":1029},"visibility":10000,"wind":{"speed":6.72,"deg":60,"gust":7.38},"clouds":{"all":100},"dt":1744295754,"sys":{"country":"FR","sunrise":1744262467,"sunset":1744310941},"timezone":3600,"id":3016536,"name":"Gatteville-le-Phare","cod":200}
0
--------------------------------------- Verbose Output ---------------------------------------
( 277.6 KB)
Allocated Bytes : 34508 B ( 33.7 KB)
Minimum Free Bytes: 278256 B ( 271.7 KB)
Largest Free Block: 110580 B ( 108.0 KB)
------------------------------------------
Flash Info:
------------------------------------------
Chip Size : 4194304 B (4 MB)
Block Size : 65536 B ( 64.0 KB)
Sector Size : 4096 B ( 4.0 KB)
Page Size : 256 B ( 0.2 KB)
Bus Speed : 80 MHz
Bus Mode : QIO
------------------------------------------
Partitions Info:
------------------------------------------
nvs : addr: 0x00009000, size: 20.0 KB, type: DATA, subtype: NVS
otadata : addr: 0x0000E000, size: 8.0 KB, type: DATA, subtype: OTA
app0 : addr: 0x00010000, size: 1280.0 KB, type: APP, subtype: OTA_0
app1 : addr: 0x00150000, size: 1280.0 KB, type: APP, subtype: OTA_1
spiffs : addr: 0x00290000, size: 1408.0 KB, type: DATA, subtype: SPIFFS
coredump : addr: 0x003F0000, size: 64.0 KB, type: DATA, subtype: COREDUMP
------------------------------------------
Software Info:
------------------------------------------
Compile Date/Time : Apr 10 2025 15:51:57
Compile Host OS : windows
ESP-IDF Version : v5.4-690-gd4aa25a38e-dirty
Arduino Version : 3.2.0
------------------------------------------
Board Info:
------------------------------------------
Arduino Board : ESP32_DEV
Arduino Variant : esp32
Arduino FQBN : esp32:ESP32:esp32:JTAGAdapter=default,PSRAM=disabled,PartitionScheme=default,CPUFreq=240,FlashMode=qio,FlashFreq=80,FlashSize=4M,UploadSpeed=921600,LoopCore=1,EventsCore=1,DebugLevel=verbose,EraseFlash=none,ZigbeeMode=default
============ Before Setup End ============
[ 734][V][esp32-hal-uart.c:552] uartBegin(): UART0 baud(115200) Mode(800001c) rxPin(3) txPin(1)
[ 743][V][esp32-hal-uart.c:656] uartBegin(): UART0 not installed. Starting installation
[ 753][V][esp32-hal-uart.c:666] uartBegin(): UART0 RX FIFO full threshold set to 120 (value requested: 120 || FIFO Max = 128)
[ 766][V][esp32-hal-uart.c:687] uartBegin(): Setting UART0 to use REF_TICK clock
[ 776][V][esp32-hal-uart.c:738] uartBegin(): UART0 initialization done.
[ 791][V][esp32-hal-uart.c:785] uartSetRxFIFOFull(): UART0 RX FIFO Full value set to 120 from a requested value of 120
Booting...
[ 3817][V][NetworkEvents.cpp:117] _checkForEvent(): Network Event: 101 - WIFI_READY
[ 3892][V][STA.cpp:186] _onStaEvent(): STA Started
[ 3897][V][NetworkEvents.cpp:117] _checkForEvent(): Network Event: 110 - STA_START
[ 3905][V][STA.cpp:110] _onStaArduinoEvent(): Arduino STA Event: 110 - STA_START
[ 3955][V][STA.cpp:206] _onStaEvent(): STA Connected: SSID: sg1, BSSID: ca:27:c2:4b:90:70, Channel: 8, Auth: WPA2_PSK
[ 3966][V][NetworkEvents.cpp:117] _checkForEvent(): Network Event: 112 - STA_CONNECTED
[ 3974][V][STA.cpp:110] _onStaArduinoEvent(): Arduino STA Event: 112 - STA_CONNECTED
[ 4978][V][NetworkInterface.cpp:78] _onIpEvent(): sta Got New IP: 192.168.51.32 MASK: 255.255.255.0 GW: 192.168.51.153
[ 4989][V][NetworkEvents.cpp:117] _checkForEvent(): Network Event: 115 - STA_GOT_IP
[ 4996][V][STA.cpp:110] _onStaArduinoEvent(): Arduino STA Event: 115 - STA_GOT_IP
[ 5004][V][STA.cpp:171] _onStaArduinoEvent(): STA IP: 192.168.51.32, MASK: 255.255.255.0, GW: 192.168.51.153
Connected to: test1, Local IP=192.168.51.32
weatherQueryURL=http://api.openweathermap.org/data/2.5/weather?lon=-1.000&lat=50.000&units=metric&lang=en&appid=f058fe1cad2afe8e2ddc5d063a64cecb
[ 5026][V][HTTPClient.cpp:242] beginInternal(): url: http://api.openweathermap.org/data/2.5/weather?lon=-1.000&lat=50.000&units=metric&lang=en&appid=f058fe1cad2afe8e2ddc5d063a64cecb
[ 5047][D][HTTPClient.cpp:293] beginInternal(): protocol: http, host: api.openweathermap.org port: 80 url: /data/2.5/weather?lon=-1.000&lat=50.000&units=metric&lang=en&appid=f058fe1cad2afe8e2ddc5d063a64cecb
[ 5065][D][HTTPClient.cpp:574] sendRequest(): request type: 'GET' redirCount: 0
[ 5073][D][NetworkManager.cpp:83] hostByName(): Clearing DNS cache
[ 5359][D][NetworkManager.cpp:127] hostByName(): DNS found IPv4 5.9.68.251
[ 5394][D][HTTPClient.cpp:1112] connect(): connected to api.openweathermap.org:80
[ 5474][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: 'HTTP/1.1 200 OK'
[ 5481][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: 'Server: openresty'
[ 5488][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: 'Date: Thu, 10 Apr 2025 14:53:07 GMT'
[ 5497][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: 'Content-Type: application/json; charset=utf-8'
[ 5507][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: 'X-Cache-Key: /data/2.5/weather?lang=en&lat=50&lon=-1&units=metric'
[ 5519][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: 'Access-Control-Allow-Origin: *'
[ 5528][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: 'Access-Control-Allow-Credentials: true'
[ 5537][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: 'Access-Control-Allow-Methods: GET, POST'
[ 5546][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: 'Connection: close'
[ 5553][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: ''
[ 5559][D][HTTPClient.cpp:1257] handleHeaderResponse(): code: 200
[ 5565][D][HTTPClient.cpp:618] sendRequest(): sendRequest code=200
httpResponseCode=200
Asking for getString() [takes 2mins, but should take ~1ms]...
[126603][D][NetworkClient.cpp:576] connected(): Disconnected: RES: 0, ERR: 128
[126611][V][HTTPClient.cpp:1399] writeToStreamDataBlock(): connection closed or file end (written: 510).
[126620][D][HTTPClient.cpp:393] disconnect(): tcp is closed
Time=121054ms
Got
---
1f2
{"coord":{"lon":-1,"lat":50},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04d"}],"base":"stations","main":{"temp":9.24,"feels_like":6.05,"temp_min":9.24,"temp_max":9.24,"pressure":1029,"humidity":79,"sea_level":1029,"grnd_level":1029},"visibility":10000,"wind":{"speed":6.72,"deg":60,"gust":7.38},"clouds":{"all":100},"dt":1744296787,"sys":{"country":"FR","sunrise":1744262467,"sunset":1744310941},"timezone":3600,"id":3016536,"name":"Gatteville-le-Phare","cod":200}
0
[126666][D][HTTPClient.cpp:393] disconnect(): tcp is closed
=========== After Setup Start ============
INTERNAL Memory Info:
------------------------------------------
Total Size : 326036 B ( 318.4 KB)
Free Bytes : 229784 B ( 224.4 KB)
Allocated Bytes : 86004 B ( 84.0 KB)
Minimum Free Bytes: 217112 B ( 212.0 KB)
Largest Free Block: 110580 B ( 108.0 KB)
------------------------------------------
GPIO Info:
------------------------------------------
GPIO : BUS_TYPE[bus/unit][chan]
--------------------------------------
1 : UART_TX[0]
3 : UART_RX[0]
============ After Setup End =============