plugins icon indicating copy to clipboard operation
plugins copied to clipboard

security/etpro-telemetry: widget empty when "sensor_status" is not "ACTIVE"

Open allenlook opened this issue 3 months ago • 13 comments

A clear and concise description of what the bug is, including last known working version (if any).

Tip: to validate your setup was working with the previous version, use opnsense-revert (https://docs.opnsense.org/manual/opnsense_tools.html#opnsense-revert)

To Reproduce

Steps to reproduce the behavior:

  1. Check telemetry widget on 25.7.3_3, it is working.
  2. Upgrade to 25.7.3_7.
  3. Check telemetry widget, it displays "Failed to load widget".

Expected behavior

I expected the widget to continue to function after the upgrade, as it always has through 24.x to 25.x

Environment

Software version used and hardware type if relevant, e.g.:

OPNsense 25.7.3_7 (amd64). Intel® N100

allenlook avatar Sep 17 '25 19:09 allenlook

Thank you for creating an issue. Since the ticket doesn't seem to be using one of our templates, we're marking this issue as low priority until further notice.

For more information about the policies for this repository, please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.

The easiest option to gain traction is to close this ticket and open a new one using one of our templates.

OPNsense-bot avatar Sep 17 '25 20:09 OPNsense-bot

@allenlook thanks for the report! when you call "https://your.firewall/api/diagnostics/proofpoint_et/status" in the browser does it load correctly? And if so how long does the load of the API page take?

Cheers, Franco

fichtner avatar Sep 18 '25 07:09 fichtner

It immediately returns this response:

{"status":"failed","response":"HTTPSConnectionPool(host='opnsense.emergingthreats.net', port=443): Max retries exceeded with url: /api/v1/sensorinfo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1016)')))"}

allenlook avatar Sep 18 '25 12:09 allenlook

Thanks, there seems to be a CA chain issue with opnsense.emergingthreats.net not including the intermediate:

# echo | openssl s_client -no_ign_eof "opnsense.emergingthreats.net:443"

CONNECTED(00000003)
depth=0 C = US, ST = California, O = "Proofpoint, Inc.", CN = *.emergingthreats.net
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = California, O = "Proofpoint, Inc.", CN = *.emergingthreats.net
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 C = US, ST = California, O = "Proofpoint, Inc.", CN = *.emergingthreats.net
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, O = "Proofpoint, Inc.", CN = *.emergingthreats.net
   i:C = GB, O = Sectigo Limited, CN = Sectigo Public Server Authentication CA OV R36
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jul 26 00:00:00 2025 GMT; NotAfter: Jul 26 23:59:59 2026 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIIKjCCBpKgAwIBAgIQDwSpLVPmwxeLKjuxso7tyjANBgkqhkiG9w0BAQsFADBg
MQswCQYDVQQGEwJHQjEYMBYGA1UEChMPU2VjdGlnbyBMaW1pdGVkMTcwNQYDVQQD
Ey5TZWN0aWdvIFB1YmxpYyBTZXJ2ZXIgQXV0aGVudGljYXRpb24gQ0EgT1YgUjM2
MB4XDTI1MDcyNjAwMDAwMFoXDTI2MDcyNjIzNTk1OVowXTELMAkGA1UEBhMCVVMx
EzARBgNVBAgTCkNhbGlmb3JuaWExGTAXBgNVBAoTEFByb29mcG9pbnQsIEluYy4x
HjAcBgNVBAMMFSouZW1lcmdpbmd0aHJlYXRzLm5ldDCCAiIwDQYJKoZIhvcNAQEB
BQADggIPADCCAgoCggIBANqn1neqkkcpb8UBdTHLKvlrmmHHNQUK8UXSJDcSqRn9
uilQCvwLfmTFO4UpeK1Ildr/iVsn2Uzo2zcDPjCcBO+Fku+XoHfxDPrtSAc889yz
H8MB98FHJbZC7h6rQ75qX5TCYu5084/mOaeACBgEkytEKZ7ibednxOm0pOvnaA4a
tOQoqHBjnY3g3Pmkg+/52FWU/UdTO3x6iChT/7Kh6yg3jcHby4tw44XwoIrF5wZx
JL2f5SjxIBZrJPnQaTKirAcGYZPqKT5WrLgm6PdjEhlbLokptoqyOatLpzYumqtr
4KXuN8YAUz4CbnxhXB6z10tsDiPWKrV+pm6Nlq2NMTOzz0oJz+dHLYGdH+n40Ctl
38mCujejImCDzCLjSLKFJMFgHr/l3pMYAiYtBpY+rMOnIoIUKRvnR/wZFVasSTVY
N883FIL6QQ3bxztIRz9yir0n7kRqnhkAazY/pmpeIDLJMogJGIhnkj9VTQSyUYEK
scbzPgHm/nELeGrPhyRimCGD25Zy/Fy2hcRV3epJvatw3VNWaT0n6sh9qcp87SjE
tKgI4cUkDMlt/6/NUmoLNODdcVx0cwo+reWy9KXiCG061VAW8JIxtYzFxaVKcdOD
MrBQut0Gp04PhONdITYSHruTvY/bmawPApoMECI/kBydjYmWkJ2+0BEVcAzdOupR
AgMBAAGjggNhMIIDXTAfBgNVHSMEGDAWgBTjZnS7cGiNLF1ODqZKj5s3IpyCkjAd
BgNVHQ4EFgQUH7+CNFCQHcz01zal6wUPpYu1djAwDgYDVR0PAQH/BAQDAgWgMAwG
A1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMEoGA1Ud
IARDMEEwNQYMKwYBBAGyMQECAQMEMCUwIwYIKwYBBQUHAgEWF2h0dHBzOi8vc2Vj
dGlnby5jb20vQ1BTMAgGBmeBDAECAjBUBgNVHR8ETTBLMEmgR6BFhkNodHRwOi8v
Y3JsLnNlY3RpZ28uY29tL1NlY3RpZ29QdWJsaWNTZXJ2ZXJBdXRoZW50aWNhdGlv
bkNBT1ZSMzYuY3JsMIGEBggrBgEFBQcBAQR4MHYwTwYIKwYBBQUHMAKGQ2h0dHA6
Ly9jcnQuc2VjdGlnby5jb20vU2VjdGlnb1B1YmxpY1NlcnZlckF1dGhlbnRpY2F0
aW9uQ0FPVlIzNi5jcnQwIwYIKwYBBQUHMAGGF2h0dHA6Ly9vY3NwLnNlY3RpZ28u
Y29tMIIBfAYKKwYBBAHWeQIEAgSCAWwEggFoAWYAdQDYCVU7lE96/8gWGW+UT4Wr
sPj8XodVJg8V0S5yu0VLFAAAAZhEQ3AiAAAEAwBGMEQCIARTUC4SRbLHdIF6n8gx
j5ARSvTIPNaq/rsBa2weZVfLAiAfCDNM7M0MReHsYfxRt7W7YHbM7+n80sOSZzUT
xmEbXQB1AKyrMHBs6+yEMfQT0vSRXxEeQiRDsfKmjE88KzunHgLDAAABmERDcAMA
AAQDAEYwRAIgQB1cqpoD6L/2xI7IwsjBNy8An3I2eCyd6IzM1NsvWDMCIEw/Y6bK
slaRZbGrd6e8j4Btci+G8i2tBEjveFK2+gvOAHYA1219ENGn9XfCx+lf1wC/+YLJ
M1pl4dCzAXMXwMjFaXcAAAGYRENv/QAABAMARzBFAiEAzSHWWEjmv+Ci00r6YqJf
dwvBTk77khRWaMMotZwjaLICIChrje0XWeWwCvCDnO1FxWrcTRFIdts+pesRP97V
dZSoMDUGA1UdEQQuMCyCFSouZW1lcmdpbmd0aHJlYXRzLm5ldIITZW1lcmdpbmd0
aHJlYXRzLm5ldDANBgkqhkiG9w0BAQsFAAOCAYEAbB7/smHPD9kzfBx0PB71ABtl
H0ezmaJBbBtUCGhucabjH9PDs7wbLBaUWKXLDFZ5Pfv9PxoSEQXsYMtx+meaZ1Ih
MzxuaKYifz9luE16Jo8kMef/IOVGmgDpfMIOw/DdUKh4ArUN5SfPmBbMnO6193c0
8yEr6QxarJA5hlWXeQwaCBqYv3FRsdxZK99SZxQ2GItLhQaIqvu0nnS4cS5NR7dv
8QBqWnbq7wpxDb2QBy4ABe/A/RoivxRsKbK+pTNky+Y44ZXtHTF6+/gNhzMqgQFs
IjVdoUl9Mk+oi4Ciy7Lr+23CqLLxQ+VI91rOFZEDAW0dSgxNan86jLpHrzdcQlsq
HNzqei/+ITcw+vL/jBBd7pwjO/3w/nnuHqvWZ8A3MCAToLvpEcEl5wS7hCgyGLvN
L7l1XMh3U8akIxwaz+LE6kfxzd7GJ1kvNeNIl6t3mOylbpK3FmOwmt3DwDzdmdPy
zmAtV2i1Jq3v20WIIiyhJlrwT9hTFcPjPpi+wE99
-----END CERTIFICATE-----
subject=C = US, ST = California, O = "Proofpoint, Inc.", CN = *.emergingthreats.net
issuer=C = GB, O = Sectigo Limited, CN = Sectigo Public Server Authentication CA OV R36
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2890 bytes and written 398 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
DONE

Loading the intermediate into the trust store of OPNsense (and enabling storing intermediates in the settings) fixes this but the server end should probably have the intermediate for better interoperability.

fichtner avatar Sep 18 '25 13:09 fichtner

Looks fixed from my end. Now we can go back to https://github.com/opnsense/plugins/issues/4943#issuecomment-3306314947

fichtner avatar Sep 19 '25 13:09 fichtner

After 2 seconds, it returns:

{"sensorId":"removed-out-of-caution","sensor_status":"DORMANT","last_heartbeat":"2025-09-19T13:26:28+00:00","last_rule_download":"2025-09-17T06:00:12+00:00","event_received":"2025-09-19T13:53:04+00:00","created":"2024-09-26T21:15:11+00:00","disable_date":"2026-09-19T13:53:04+00:00","status":"ok"}

The widget still displays just hyphens in each of the data fields, though.

allenlook avatar Sep 19 '25 13:09 allenlook

Shortly after I posted that, I restarted the Intrusion Detection service as a lark, and then the firewall went completely belly up - a continuous scroll of init and dump error messages. I had to use the power button to shut it down and reboot it. Hopefully that's not a "new thing", as it's never done that before.

allenlook avatar Sep 19 '25 14:09 allenlook

Perhaps some instability with IPS mode use?

I haven't tested the response you posted on my end yet but it looks rather normal and validates as proper JSON.

Cheers, Franco

fichtner avatar Sep 19 '25 15:09 fichtner

API return looks good, yes. But even after the reboot the widget does not display the dates or the "OK" status.

The Suricata log is FULL of errors from the time of the crash, though. It shows that the engine restarted, and then within 3 seconds it started posting about 20,000 of these, until I shut it off.

[299151] <Error> -- igc0^: error reading netmap data via polling: No error: 0 [299176] <Error> -- igc0^: error reading netmap data via polling: No buffer space available

allenlook avatar Sep 19 '25 15:09 allenlook

Yes, it's IPS mode. Mileage may vary with hardware, NICs and traffic being pushed.

I tracked it down and it has always been like this when the service is not "ACTIVE" (currently returning "DORMANT" which is probably in relation to the server side SSL issue and not heartbeats being sent over).

https://github.com/opnsense/plugins/blob/25b4d659576a3d050042ac6d74f4317a17b22f9f/security/etpro-telemetry/src/opnsense/www/js/widgets/ETProTelemetry.js#L44-L54

I'll make this a cleanup ticket but need to discuss what to with the widget code since this isn't my plugin.

Cheers, Franco

fichtner avatar Sep 19 '25 15:09 fichtner

@allenlook thanks for the report! when you call "https://your.firewall/api/diagnostics/proofpoint_et/status" in the browser does it load correctly? And if so how long does the load of the API page take?

Cheers, Franco

I personally get nothing, a blank html page.

lemaximedu66 avatar Sep 22 '25 16:09 lemaximedu66

Don't forget to replace "your.firewall" with the IP address and port of your firewall, if you have not created an Alias otherwise.

allenlook avatar Sep 22 '25 17:09 allenlook

Don't forget to replace "your.firewall" with the IP address and port of your firewall, if you have not created an Alias otherwise.

Don't worry, I did, I'm not that dumb, also, even in the dev tools there is a blank html document...

lemaximedu66 avatar Sep 22 '25 17:09 lemaximedu66