Home ESP32 SSL connectivity fails (at least while accessing Azure Storage Account files >5kB)

Library/API/IoT binding

nanoFramework.System.Net.Http.Client

Visual Studio version

VS2022

.NET nanoFramework extension version

2022.3.0.18

Target name(s)

ESP32 (ESP32-D0WD-V3 (revision 3))

Firmware version

1.8.1.15

Device capabilities

System Information
HAL build info: nanoCLR running @ ESP32 built with ESP-IDF 6407ecb
  Target:   ESP32
  Platform: ESP32

Firmware build Info:
  Date:        Jan 31 2023
  Type:        MinSizeRel build, chip rev. 3, without support for PSRAM
  CLR Version: 1.8.0.947
  Compiler:    GNU ARM GCC v8.4.0

OEM Product codes (vendor, model, SKU): 0, 0, 0

Serial Numbers (module, system):
  00000000000000000000000000000000
  0000000000000000

Target capabilities:
  Has nanoBooter: NO
  IFU capable: NO
  Has proprietary bootloader: YES

AppDomains:

Assemblies:
  nf_ssl, 1.0.0.0
  nanoFramework.Runtime.Native, 1.6.6.0
  System.Net.Http, 1.5.82.0
  nanoFramework.ResourceManager, 1.2.13.0
  System.Threading, 1.1.19.33722
  nanoFramework.System.Collections, 1.5.18.0
  System.Device.Wifi, 1.5.51.0
  System.Net, 1.10.52.0
  System.IO.Streams, 1.1.38.0
  System.IO.FileSystem, 1.1.23.0
  nanoFramework.Runtime.Events, 1.11.6.0
  nanoFramework.System.Text, 1.2.37.0
  mscorlib, 1.14.3.0

Native Assemblies:
  mscorlib v100.5.0.18, checksum 0xCCE8376E
  nanoFramework.Runtime.Native v100.0.9.0, checksum 0x109F6F22
  nanoFramework.Hardware.Esp32 v100.0.7.3, checksum 0xBE7FF253
  nanoFramework.Hardware.Esp32.Rmt v100.0.3.0, checksum 0x0A915860
  nanoFramework.Device.OneWire v100.0.4.0, checksum 0xB95C43B4
  nanoFramework.Networking.Sntp v100.0.4.4, checksum 0xE2D9BDED
  nanoFramework.ResourceManager v100.0.0.1, checksum 0xDCD7DF4D
  nanoFramework.System.Collections v100.0.1.0, checksum 0x2DC2B090
  nanoFramework.System.Text v100.0.0.1, checksum 0x8E6EB73D
  nanoFramework.Runtime.Events v100.0.8.0, checksum 0x0EAB00C9
  EventSink v1.0.0.0, checksum 0xF32F4C3E
  System.IO.FileSystem v1.0.0.0, checksum 0x3AB74021
  System.Math v100.0.5.4, checksum 0x46092CB1
  System.Net v100.1.5.0, checksum 0x5BAB8CB3
  System.Device.Adc v100.0.0.0, checksum 0xE5B80F0B
  System.Device.Dac v100.0.0.6, checksum 0x02B3E860
  System.Device.Gpio v100.1.0.6, checksum 0x097E7BC5
  System.Device.I2c v100.0.0.1, checksum 0xFA806D33
  System.Device.I2s v100.0.0.1, checksum 0x478490FE
  System.Device.Pwm v100.1.0.4, checksum 0xABF532C3
  System.IO.Ports v100.1.6.1, checksum 0xB798CE30
  System.Device.Spi v100.1.2.0, checksum 0x3F6E2A7E
  System.Runtime.Serialization v100.0.0.0, checksum 0x0A066871
  System.Device.Wifi v100.0.6.4, checksum 0x1C1D3214
  Windows.Storage v100.0.2.0, checksum 0x954A4192



++++++++++++++++++++++++++++++++
++        Memory Map          ++
++++++++++++++++++++++++++++++++
  Type     Start       Size
++++++++++++++++++++++++++++++++
  RAM   0x3ffe49ac  0x0001b000
  FLASH 0x00000000  0x00400000



+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++                   Flash Sector Map                        ++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  Region     Start      Blocks   Bytes/Block    Usage
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
      0    0x00010000       1      0x1A0000     nanoCLR
      1    0x001B0000       1      0x1F0000     Deployment
      2    0x003C0000       1      0x040000     Configuration


+++++++++++++++++++++++++++++++++++++++++++++++++++
++              Storage Usage Map                ++
+++++++++++++++++++++++++++++++++++++++++++++++++++
  Start        Size (kB)           Usage
+++++++++++++++++++++++++++++++++++++++++++++++++++
  0x003C0000    0x040000 (256kB)    Configuration
  0x00010000    0x1A0000 (1664kB)   nanoCLR
  0x001B0000    0x1F0000 (1984kB)   Deployment



Deployment Map
Empty

Description

While trying to implement OTA updates for ESP32 according to samples, I stumbled upon a problem fetching files which where around 5.7kB in size. Upon further analysis I found out, that ESP32 has problems with the SSL connectivity, not sure why exactly though.

How to reproduce

Clone https://github.com/BWalti/nf-ssl
Open Solution
Connect nanoFramework device based on ESP32
Configure it's network connectivity (WiFi) OR configure SSID and password variables appropriately
Debug / Start the application
Check the output / Exception

Free memory = 81588
Trying to fetch file with name: System.Device.Gpio.pe
    ++++ Exception System.Net.Sockets.SocketException - 0x00000000 (1) ++++
    ++++ Message: 
    ++++ System.Net.InputNetworkStreamWrapper::Read_HTTP_Line [IP: 015a] ++++
    ++++ System.Net.HttpWebRequest::ParseHTTPResponse [IP: 002e] ++++
    ++++ System.Net.HttpWebRequest::GetResponse [IP: 0040] ++++
    ++++ System.Net.Http.HttpClientHandler::Send [IP: 00a7] ++++
    ++++ System.Net.Http.HttpMessageInvoker::Send [IP: 0013] ++++
    ++++ System.Net.Http.HttpClient::Send [IP: 006f] ++++
    ++++ System.Net.Http.HttpClient::Get [IP: 000c] ++++
    ++++ Program::<Main>$ [IP: 00e7] ++++
    ++++ Exception System.Net.WebException - 0x00000000 (1) ++++
    ++++ Message: GetResponse() failed
    ++++ System.Net.HttpWebRequest::GetResponse [IP: 00da] ++++
    ++++ System.Net.Http.HttpClientHandler::Send [IP: 00a7] ++++
    ++++ System.Net.Http.HttpMessageInvoker::Send [IP: 0013] ++++
    ++++ System.Net.Http.HttpClient::Send [IP: 006f] ++++
    ++++ System.Net.Http.HttpClient::Get [IP: 000c] ++++
    ++++ Program::<Main>$ [IP: 00e7] ++++
    ++++ Exception System.Net.Http.HttpRequestException - 0x00000000 (1) ++++
    ++++ Message: An error occurred while sending the request
    ++++ System.Net.Http.HttpClientHandler::Send [IP: 00bf] ++++
    ++++ System.Net.Http.HttpMessageInvoker::Send [IP: 0013] ++++
    ++++ System.Net.Http.HttpClient::Send [IP: 006f] ++++
    ++++ System.Net.Http.HttpClient::Get [IP: 000c] ++++
    ++++ Program::<Main>$ [IP: 00e7] ++++
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Could not fetch file with name: System.Device.Gpio.pe
System.Net.WebException: GetResponse() failed
An error occurred while sending the request
0
GetResponse() failed
System.Net.Sockets.SocketException: Exception was thrown: System.Net.Sockets.SocketException
10053

Expected behaviour

The application should be able to fetch the file named System.Device.Gpio.pe.

Screenshots

No response

Sample project or code

https://github.com/BWalti/nf-ssl

Aditional information

We already started to discuss this issue on Discord: https://discord.com/channels/478725473862549535/1063036621177499688

As you can see there, the same application works perfectly well on ST_STM32F769I_DISCOVERY according to @josesimoes

It seems that the response stream contains 0 bytes and therefore the exception is risen. What I tried to find out: if the "sending" of the HTTP request succeeded or not. But that's just how far I've gotten.

Feb 28 '23 09:02 BWalti

@BWalti can you please check if this is still happening? I've made a number of tiny improvements in the memory management and we're now on the latest IDF 4.4.5. If this was on ESP32 code (which I believe it is) maybe this got fixed upstream.

Sep 01 '23 11:09 josesimoes

Hello @josesimoes - I just did a nanoff --update to ESP32_REV3-1.8.1.491 and even upgraded the HTTP package in my repro git repo (https://github.com/BWalti/nf-ssl) - still no success!

The same error code as above: 10053 fetching the same files via http (non secure) succeeds.

Sep 04 '23 09:09 BWalti

Could it be an issue with the CA root cert you're using?

Sep 04 '23 11:09 josesimoes

hm.. according to this: I'd say I should have the "right" CA in my project (if I remember correctly I was actually using one of the sample projects). On the left you'd see the thumbprint from the "CA in the project's resources" and on the right the CA the browser's CA upon accessing that storage account.

Should I also add the intermediate cert? Or should the root CA suffice?

Sep 05 '23 07:09 BWalti

Agreed that the thumbprints are the same. It could be the format that the is stored with. Please check this sample here and compare with your project.

Sep 05 '23 08:09 josesimoes

I just changed (https://github.com/BWalti/nf-ssl/commit/ea708f9c827c3b30d90b2d4651b9ac77d03fce53) to the same PEM file as in the AzureSDK sample (using "text" file .pem and loading the cert using "GetString" instead of "GetBytes") - still I got the same results..

Sep 05 '23 08:09 BWalti

@BWalti are you sure it's the right certificate? The date when connecting to the blob storage is not before end of July this year and not after 2024:

The ones used in Azure IoT have a longer duration. So you want to double check this part. Also note that the sample here uses as well the Blob storage and still works fine: https://github.com/nanoframework/Samples/tree/main/samples/AzureSDK/AzureEdgeOta

Sep 05 '23 14:09 Ellerbach

hm.. I believe I started my little project with exactly that project as blueprint, because I actually wanted to implement the OTA functionality. But I'll give it another try (hopefully the next few days).

Sep 05 '23 14:09 BWalti

Quick feedback: Strangely enough I pulled in the .pem file from the OTA project, loaded it the same way but still: in my project the connectivity fails, in OTA sample project it doesn't. So there must be another difference...

I'm trying to figure it out, but didn't find it yet and haven't had the time for a more detailed analysis yet.

Sep 14 '23 05:09 BWalti

@Ellerbach : I tried it the other way round: using the official OTA Sample just re-coding to fetch the file from my storage account using HTTPS but without SAS token fails the same way as my sample project does. So it's not the certificate per se. Interestingly though if I configured a storage account to be used for the OTA sample, it worked. maybe yet again because of "file size differences". Have to check this in more depth next time..

Oct 24 '23 14:10 BWalti

rom my storage account using HTTPS but without SAS token fails the same way as my sample project does. So it's not the certificate per se

Does the HTTPS place uses the same root cert? Place all the root CA certs you need in a single PEM file and pass that one, on the native side, it will figure it out which one is the proper one to use.

Have to check this in more depth next time..

Thanks for reporting already the progress and yes, please report anything so we can make the docs or fix any specific bug.

Oct 25 '23 11:10 Ellerbach

I'm no security expert, thus: how would I get all the root CA certs for my storage account? Is there a tool for that? Additionally, with "all root CAs", do you mean "intermediate" CAs as well?

The storage account for which I'd like to get the correct PEM file would be: https://bwaltilasertag.blob.core.windows.net/piot/

Oct 25 '23 13:10 BWalti

I just tried https://www.ssllabs.com/ssltest/analyze.html?d=bwaltilasertag.blob.core.windows.net and fetched the whole chain and put that into the pem file. Still the same behavior:

    ++++ Exception System.Net.Sockets.SocketException - 0x00000000 (1) ++++
    ++++ Message: 
    ++++ System.Net.InputNetworkStreamWrapper::Read_HTTP_Line [IP: 015a] ++++
    ++++ System.Net.HttpWebRequest::ParseHTTPResponse [IP: 002e] ++++
    ++++ System.Net.HttpWebRequest::GetResponse [IP: 0040] ++++
    ++++ System.Net.Http.HttpClientHandler::Send [IP: 00a7] ++++
    ++++ System.Net.Http.HttpMessageInvoker::Send [IP: 0013] ++++
    ++++ System.Net.Http.HttpClient::Send [IP: 006f] ++++
    ++++ System.Net.Http.HttpClient::Get [IP: 000c] ++++
    ++++ Program::<Main>$ [IP: 0106] ++++
    ++++ Exception System.Net.WebException - 0x00000000 (1) ++++
    ++++ Message: GetResponse() failed
    ++++ System.Net.HttpWebRequest::GetResponse [IP: 00da] ++++
    ++++ System.Net.Http.HttpClientHandler::Send [IP: 00a7] ++++
    ++++ System.Net.Http.HttpMessageInvoker::Send [IP: 0013] ++++
    ++++ System.Net.Http.HttpClient::Send [IP: 006f] ++++
    ++++ System.Net.Http.HttpClient::Get [IP: 000c] ++++
    ++++ Program::<Main>$ [IP: 0106] ++++
    ++++ Exception System.Net.Http.HttpRequestException - 0x00000000 (1) ++++
    ++++ Message: An error occurred while sending the request
    ++++ System.Net.Http.HttpClientHandler::Send [IP: 00bf] ++++
    ++++ System.Net.Http.HttpMessageInvoker::Send [IP: 0013] ++++
    ++++ System.Net.Http.HttpClient::Send [IP: 006f] ++++
    ++++ System.Net.Http.HttpClient::Get [IP: 000c] ++++
    ++++ Program::<Main>$ [IP: 0106] ++++
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll

Oct 25 '23 13:10 BWalti

@Ellerbach : have you any input? or could you maybe try to run the repro https://github.com/BWalti/nf-ssl on an ESP32?

Nov 07 '23 14:11 BWalti

@BWalti not sure what's happening. Indeed, I'll look at this deeply as soon as I have a bit of time. On the cert approval chain, here it is, you can check them in good browsers:

For you not to be blocked in the development and the time of checking what's happening, you can bypass the cert check.

Nov 08 '23 10:11 Ellerbach

Can you try this (you need openssl installed properly):

openssl s_client -connect bwaltilasertag.blob.core.windows.net:443 -showcerts

This will show you the certs used and the chain. You should be able to play with this. In all cases, that should definitely work!

Nov 20 '23 13:11 Ellerbach

@BWalti are still seeing this issue with the latest fw version?

Jun 12 '24 08:06 josesimoes

Hey José - last time I checked, I still had the same issue if I recall correctly. Didn't get the chance to re-try lately. But we can also close this issue for now and I'll re-open it, when I get to this point again.

I also used the openssl s_client command mentioned above, couldn't detect any difference, but will give it another shot, hopefully soon. Have a bit much to do at work lately :)

Jun 14 '24 05:06 BWalti

OK! Good luck for whatever you need it and do reopen if the issue is still there. 😃

Jun 14 '24 09:06 josesimoes

Home Home copied to clipboard

ESP32 SSL connectivity fails (at least while accessing Azure Storage Account files >5kB)

Library/API/IoT binding

Visual Studio version

.NET nanoFramework extension version

Target name(s)

Firmware version

Device capabilities

Description

How to reproduce

Expected behaviour

Screenshots

Sample project or code

Aditional information

Home
Home copied to clipboard