Failure after TLS issue

[Ottergoose]

Currently running Running ADS-B Feeder Image v1.1.9-beta.1(beta) with containers from 2024-01-25T18:40:21-06:00, running on Raspberry Pi 3 Model B Plus Rev 1.3. adsb-im-raspberrypi64-v1.1.4.img; I assume it has the latest version of this software.

I noticed FlightAware said I wasn't feeding data for a few hours, and checked my logs; it looks like a TLS issue tripped things up, and it was unable to recover until I rebooted. After this happened, no data was going to FlightAware, and my CPU usage bumped way up.

07:29:43 AM [piaware] 2024/02/04 07:29:43 TLS alert (read): bad record mac 07:29:43 AM [piaware] 2024/02/04 07:29:43 TLS error: 0 07:29:43 AM [piaware] 2024/02/04 07:29:43 Lost connection to adept server at piaware.flightaware.com/1200: error reading "sock559fd35b60": software caused connection abort

Hope this helps, thank you for your work!

[the-jeffski]

I am seeing the same - have enabled the autoheal container to trigger restarts for now

[wiedehopf]

https://github.com/flightaware/piaware/pull/87

Ok so ... i'm not sure but possibly we could just make a watchdog for this container that watches the piaware process. Apparently there is some bug they can't fix so they added a watchdog .... The container doesn't run systemd so the watchdog would just have to check piaware log output i suppose.

How regular is the log output?

[the-jeffski]

I'll aim to grab a log output - just upgraded the container a couple of hours back so the current log is gone. I'm running autoheal at the moment to restart the container as the only fix but it occasionally doesn't work then I find I'm offline for 6 hrs. Auto heal log gives an idea of how often: `2024-06-11 08:50:15+0100 [WARNING] [piaware (8fb95a0ab823)] Container restarting with 10s timeout

2024-06-11 08:50:19+0100 [ INFO] [piaware (8fb95a0ab823)] Container restart was successful

2024-06-11 11:50:25+0100 [WARNING] [piaware (8fb95a0ab823)] Container is unhealthy with 3 failures

2024-06-11 11:50:25+0100 [WARNING] [piaware (8fb95a0ab823)] Container last output: [1] No connection to Flightaware, NOT OK.

54227 dump1090 messages sent in past hour, OK.

Webserver listening on port 80, OK.

Webserver listening on port 8080, OK.

2024-06-11 11:50:25+0100 [WARNING] [piaware (8fb95a0ab823)] Container restarting with 10s timeout

2024-06-11 11:50:28+0100 [ INFO] [piaware (8fb95a0ab823)] Container restart was successful

2024-06-11 13:20:35+0100 [WARNING] [piaware (8fb95a0ab823)] Container is unhealthy with 3 failures

2024-06-11 13:20:35+0100 [WARNING] [piaware (8fb95a0ab823)] Container last output: [1] No connection to Flightaware, NOT OK.

57745 dump1090 messages sent in past hour, OK.

Webserver listening on port 80, OK.

Webserver listening on port 8080, OK.

2024-06-11 13:20:35+0100 [WARNING] [piaware (8fb95a0ab823)] Container restarting with 10s timeout

2024-06-11 13:20:38+0100 [ INFO] [piaware (8fb95a0ab823)] Container restart was successful

2024-06-11 14:20:45+0100 [WARNING] [piaware (8fb95a0ab823)] Container is unhealthy with 3 failures

2024-06-11 14:20:45+0100 [WARNING] [piaware (8fb95a0ab823)] Container last output: [1] No connection to Flightaware, NOT OK.

28553 dump1090 messages sent in past hour, OK.

Webserver listening on port 80, OK.

Webserver listening on port 8080, OK.

2024-06-11 14:20:45+0100 [WARNING] [piaware (8fb95a0ab823)] Container restarting with 10s timeout

2024-06-11 14:20:48+0100 [ INFO] [piaware (8fb95a0ab823)] Container restart was successful

2024-06-11 15:50:55+0100 [WARNING] [piaware (8fb95a0ab823)] Container is unhealthy with 3 failures

2024-06-11 15:50:55+0100 [WARNING] [piaware (8fb95a0ab823)] Container last output: [1] No connection to Flightaware, NOT OK.

56664 dump1090 messages sent in past hour, OK.

Webserver listening on port 80, OK.

Webserver listening on port 8080, OK.

2024-06-11 15:50:55+0100 [WARNING] [piaware (8fb95a0ab823)] Container restarting with 10s timeout

2024-06-11 15:50:58+0100 [ INFO] [piaware (8fb95a0ab823)] Container restart was successful

2024-06-11 17:11:05+0100 [WARNING] [piaware (8fb95a0ab823)] Container is unhealthy with 3 failures

2024-06-11 17:11:05+0100 [WARNING] [piaware (8fb95a0ab823)] Container last output: [1] No connection to Flightaware, NOT OK.

50810 dump1090 messages sent in past hour, OK.

Webserver listening on port 80, OK.

Webserver listening on port 8080, OK.

2024-06-11 17:11:05+0100 [WARNING] [piaware (8fb95a0ab823)] Container restarting with 10s timeout

2024-06-11 17:11:09+0100 [ INFO] [piaware (8fb95a0ab823)] Container restart was successful

2024-06-11 17:51:15+0100 [WARNING] [piaware (8fb95a0ab823)] Container is unhealthy with 3 failures

2024-06-11 17:51:15+0100 [WARNING] [piaware (8fb95a0ab823)] Container last output: [1] No connection to Flightaware, NOT OK.

9632 dump1090 messages sent in past hour, OK.

Webserver listening on port 80, OK.

Webserver listening on port 8080, OK.

2024-06-11 17:51:15+0100 [WARNING] [piaware (8fb95a0ab823)] Container restarting with 10s timeout

2024-06-11 17:51:18+0100 [ INFO] [piaware (8fb95a0ab823)] Container restart was successful

2024-06-11 19:11:25+0100 [WARNING] [piaware (8fb95a0ab823)] Container is unhealthy with 3 failures

2024-06-11 19:11:25+0100 [WARNING] [piaware (8fb95a0ab823)] Container last output: [1] No connection to Flightaware, NOT OK.

52173 dump1090 messages sent in past hour, OK.

Webserver listening on port 80, OK.

Webserver listening on port 8080, OK.

2024-06-11 19:11:25+0100 [WARNING] [piaware (8fb95a0ab823)] Container restarting with 10s timeout

2024-06-11 19:11:34+0100 [ INFO] [piaware (8fb95a0ab823)] Container restart was successful`

[the-jeffski]

Grabbed the error in the log from today:

[piaware] 2024/06/12 16:59:00 piaware has successfully sent several msgs to FlightAware! [piaware] 2024/06/12 17:03:34 4866 msgs recv'd from dump1090 (4866 in last 5m); 4866 msgs sent to FlightAware [piaware] 2024/06/12 17:05:33 TLS alert (read): bad record mac [piaware] 2024/06/12 17:05:33 TLS error: 0 [piaware] 2024/06/12 17:05:33 Lost connection to adept server at piaware.flightaware.com/1200: error reading "sock55564e94af50": software caused connection abort

You only get the one error output then it dies. Restart of container starts a fresh log