firmware
firmware copied to clipboard
Published
20 hours ago •
meshtastic
meshtastic
[Bug]: 1.3.40 and 1.3.41 crashes with backtrace and keeps rebooting
Category
Other
Hardware
T-Beam
Firmware Version
1.3.40.e87ecc2
Description
After doing a fresh install and setting the basics the node keeps crashing and rebooting with the backtrace as seen below.
Relevant log output
Rebooting...
ets Jun 8 2016 00:22:57
rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:1044
load:0x40078000,len:8896
load:0x40080400,len:5828
entry 0x400806ac
[D][esp32-hal-psram.c:47] psramInit(): PSRAM enabled
▒▒H??:??:?? 0
//\ E S H T /\ S T / C
??:??:?? 0 booted, wake cause 0 (boot count 1), reset_reason=reset
??:??:?? 0 Filesystem files:
??:??:?? 0 /prefs/channels.proto (59 Bytes)
??:??:?? 0 /prefs/config.proto (121 Bytes)
??:??:?? 0 /prefs/db.proto (152 Bytes)
??:??:?? 0 /prefs/module.proto (18 Bytes)
??:??:?? 0 I2C device found at address 0x34
??:??:?? 0 axp192 PMU found
??:??:?? 0 I2C device found at address 0x3c
??:??:?? 0 0x7 subtype probed in 2 tries
??:??:?? 0 ssd1306 display found
??:??:?? 0 2 I2C devices found
??:??:?? 0 Meshtastic hwvendor=4, swver=1.3.40.e87ecc2-d
??:??:?? 0 Setting random seed 3511673233
??:??:?? 0 Total heap: 239372
??:??:?? 0 Free heap: 207236
??:??:?? 0 Total PSRAM: 4194252
??:??:?? 0 Free PSRAM: 4194252
??:??:?? 0 NVS: UsedEntries 238, FreeEntries 392, AllEntries 630, NameSpaces 8
??:??:?? 0 Setup Preferences in Flash Storage
??:??:?? 0 Number of Device Reboots: 61
??:??:?? 0 Initializing NodeDB
??:??:?? 0 Loading /prefs/db.proto
??:??:?? 0 Loaded saved devicestate version 16
??:??:?? 0 Loading /prefs/config.proto
??:??:?? 0 Loaded saved config version 16
??:??:?? 0 Loading /prefs/module.proto
??:??:?? 0 Loaded saved moduleConfig version 16
??:??:?? 0 Loading /prefs/channels.proto
??:??:?? 0 Loaded saved channelFile version 16
??:??:?? 0 Number of Device Reboots: 61
??:??:?? 0 Expanding short PSK #1
??:??:?? 0 Wanted region 3, using EU868
??:??:?? 0 region=3, NODENUM=0xbdf0c808, dbsize=1
??:??:?? 0 Saving /prefs/db.proto
??:??:?? 0 Saving /prefs/config.proto
??:??:?? 0 Saving /prefs/module.proto
??:??:?? 0 Saving /prefs/channels.proto
chip id detect 0x3
Detect CHIP :AXP192
OUTPUT Register 0x5f
??:??:?? 1 AXP192 Begin PASS
??:??:?? 1 DCDC1: ENABLE
??:??:?? 1 DCDC2: ENABLE
??:??:?? 1 LDO2: ENABLE
??:??:?? 1 LDO3: ENABLE
??:??:?? 1 DCDC3: ENABLE
??:??:?? 1 Exten: ENABLE
??:??:?? 1 ----------------------------------------
??:??:?? 1 DCDC1: ENABLE
??:??:?? 1 DCDC2: ENABLE
??:??:?? 1 LDO2: ENABLE
??:??:?? 1 LDO3: ENABLE
??:??:?? 1 DCDC3: ENABLE
??:??:?? 1 Exten: ENABLE
SRC REG:0xc4
Charging enable is enable
Charging target-voltage : 0x2
end when the charge current is lower than 10% of the set value
Charge current : 700.00 mA
??:??:?? 1 Battery: usbPower=1, isCharging=0, batMv=0, batPct=0
??:??:?? 1 Read RTC time as 34
??:??:?? 1 Using MSL altitude model
??:??:?? 1 WANT GPS=1
??:??:?? 1 GxGSA NOT available
??:??:?? 1 External Notification Module Disabled
[E][vfs_api.cpp:64] open(): /littlefs/prefs/oem.proto does not exist
??:??:?? 1 No /prefs/oem.proto preferences found
??:??:?? 1 Turning on screen
??:??:?? 1 Module wants a UI Frame
??:??:?? 1 Starting meshradio init...
??:??:?? 1 Set radio: region=EU868, name=LongF, config=0, ch=0, power=27
??:??:?? 1 Radio myRegion->freqStart / myRegion->freqEnd: 869.400024 -> 869.650024 (0.250000 mhz)
??:??:?? 1 Radio myRegion->numChannels: 1
??:??:?? 1 Radio channel_num: 0
??:??:?? 1 Radio frequency: 869.525024
??:??:?? 1 Slot time: 42 msec
??:??:?? 1 Set radio: final power level=20
??:??:?? 1 RF95 init result -2
??:??:?? 1 Frequency set to 869.525024
??:??:?? 1 Bandwidth set to 250.000000
??:??:?? 1 Power output set to 20
??:??:?? 1 Current limit set to 100.000000
??:??:?? 1 Current limit set result -16
??:??:?? 1 Warning: Failed to find RF95 radio
??:??:?? 1 Starting meshradio init...
??:??:?? 1 Set radio: region=EU868, name=LongF, config=0, ch=0, power=27
??:??:?? 1 Radio myRegion->freqStart / myRegion->freqEnd: 869.400024 -> 869.650024 (0.250000 mhz)
??:??:?? 1 Radio myRegion->numChannels: 1
??:??:?? 1 Radio channel_num: 0
??:??:?? 1 Radio frequency: 869.525024
??:??:?? 1 Slot time: 42 msec
??:??:?? 1 Set radio: final power level=22
??:??:?? 1 SX126x init result 0
??:??:?? 1 Frequency set to 869.525024
??:??:?? 1 Bandwidth set to 250.000000
??:??:?? 1 Power output set to 22
??:??:?? 1 Current limit set to 140.000000
??:??:?? 1 Current limit set result 0
??:??:?? 1 SX1262 Radio init succeeded, using SX1262 radio
??:??:?? 1 WiFi ... Client Mode
??:??:?? 1 Waiting for SSL Cert to be generated.
??:??:?? 1 Retrieved Private Key: 1190 Bytes
??:??:?? 1 Retrieved Certificate: 780 Bytes
??:??:?? 1 SSL Cert Ready!
[D][WiFiGeneric.cpp:374] _eventCallback(): Event: 0 - WIFI_READY
[D][WiFiGeneric.cpp:374] _eventCallback(): Event: 2 - STA_START
??:??:?? 1 JOINING WIFI soon: ssid=XXXXXXX
??:??:?? 1 (bw=250, sf=11, cr=4/8) packet symLen=8 ms, payloadSize=237, time 3188 ms
??:??:?? 1 myNodeInfo.bitrate = 74.341286 bytes / sec
??:??:?? 1 PowerFSM init, USB power=1
??:??:?? 1 Enter state: BOOT
??:??:?? 1 Setting CPU to 240mhz because WiFi is in use.
??:??:?? 1 [Power] Battery: usbPower=1, isCharging=0, batMv=0, batPct=0
??:??:?? 1 [Screen] Screen: Started...
??:??:?? 2 [DeviceTelemetryModule] Device Telemetry: Initializing
??:??:?? 2 [DeviceTelemetryModule] -----------------------------------------
??:??:?? 2 [DeviceTelemetryModule] Device Telemetry: Read data
??:??:?? 2 [DeviceTelemetryModule] Telemetry->time: 34
??:??:?? 2 [DeviceTelemetryModule] Telemetry->air_util_tx: 0.054222
??:??:?? 2 [DeviceTelemetryModule] Telemetry->battery_level: 0
??:??:?? 2 [DeviceTelemetryModule] Telemetry->channel_utilization: 0.000000
??:??:?? 2 [DeviceTelemetryModule] Telemetry->voltage: 0.000000
??:??:?? 2 [DeviceTelemetryModule] Initial packet id 705520221, numPacketId 4294967295
??:??:?? 2 [DeviceTelemetryModule] Device Telemetry: Sending packet to mesh
??:??:?? 2 [DeviceTelemetryModule] Update DB node 0xbdf0c808, rx_time=0
??:??:?? 2 [DeviceTelemetryModule] handleReceived(LOCAL) (id=0x2a0d625f Fr0x08 To0xff, WantAck0, HopLim3 Ch0x0 Portnum=67)
??:??:?? 2 [DeviceTelemetryModule] No modules interested in portnum=67, src=LOCAL
??:??:?? 2 [DeviceTelemetryModule] Add packet record (id=0x2a0d625f Fr0x08 To0xff, WantAck0, HopLim3 Ch0x0 Portnum=67)
??:??:?? 2 [DeviceTelemetryModule] Should encrypt MQTT?: 1
??:??:?? 2 [DeviceTelemetryModule] Expanding short PSK #1
??:??:?? 2 [DeviceTelemetryModule] Using AES128 key!
??:??:?? 2 [DeviceTelemetryModule] ESP32 crypt fr=bdf0c808, num=2a0d625f, numBytes=16!
??:??:?? 2 [DeviceTelemetryModule] enqueuing for send (id=0x2a0d625f Fr0x08 To0xff, WantAck0, HopLim3 Ch0x6e encrypted)
??:??:?? 2 [DeviceTelemetryModule] txGood=0,rxGood=0,rxBad=0
??:??:?? 2 [DeviceTelemetryModule] Using channel 0 (hash 0x6e)
??:??:?? 2 [DeviceTelemetryModule] Expanding short PSK #1
??:??:?? 2 [DeviceTelemetryModule] Using AES128 key!
??:??:?? 2 [DeviceTelemetryModule] ESP32 crypt fr=bdf0c808, num=2a0d625f, numBytes=16!
??:??:?? 2 [DeviceTelemetryModule] decoded message (id=0x2a0d625f Fr0x08 To0xff, WantAck0, HopLim3 Ch0x0 Portnum=67 priority=64)
??:??:?? 2 [DeviceTelemetryModule] updateTelemetry LOCAL
??:??:?? 2 [DeviceTelemetryModule] Node status update: 1 online, 1 total
??:??:?? 2 [SerialModule] Serial Module Disabled
??:??:?? 2 [StoreForwardModule] Store & Forward Module - Disabled
??:??:?? 2 [RangeTestModule] Range Test Module - Disabled
??:??:?? 2 [WifiConnect] ... Reconnecting to WiFi access point
??:??:?? 2 [GPS] publishing pos@0:2, hasVal=0, GPSlock=0
??:??:?? 2 [GPS] No GPS lock
??:??:?? 2 [GPS] onGPSChanged() pos@0, time=0, lat=0, lon=0, alt=0
??:??:?? 2 [GPS] updatePosition LOCAL pos@0, time=0, latI=0, lonI=0, alt=0
??:??:?? 2 [GPS] Node status update: 1 online, 1 total
??:??:?? 2 [RadioIf] Starting low level send (id=0x2a0d625f Fr0x08 To0xff, WantAck0, HopLim3 Ch0x6e encrypted priority=64)
??:??:?? 2 [RadioIf] (bw=250, sf=11, cr=4/8) packet symLen=8 ms, payloadSize=32, time 747 ms
??:??:?? 2 [RadioIf] AirTime - Packet transmitted : 747ms
[D][WiFiGeneric.cpp:374] _eventCallback(): Event: 4 - STA_CONNECTED
??:??:?? 2 ************ [WiFi-event] event: 4 ************
??:??:?? 2 Connected to access point
[D][WiFiGeneric.cpp:374] _eventCallback(): Event: 7 - STA_GOT_IP
[D][WiFiGeneric.cpp:418] _eventCallback(): STA IP: 192.168.188.27, MASK: 255.255.255.0, GW: 192.168.188.1
?:?? 2 ************ [WiFi-event] event: 7 ************
??:??:?? 2 Obtained IP address: 192.168.188.27
... Starting network services
??:??:?? 2 mDNS responder started
??:??:?? 2 mDNS Host: Meshtastic.local
??:??:?? 2 Starting NTP time client
??:??:?? 2 Initializing Web Server ...
??:??:?? 2 Starting Secure Web Server...
??:??:?? 2 [PowerFSM] Starting Insecure Web Server...
??:??:?? 2 Web Servers Ready! :-)
??:??:?? 2 API server listening on TCP port 4403
??:??:?? 2 [WebServerThread] Connecting to MQTT server mqtt.meshtastic.org, port: 1883, username: meshdev, password: large4cats
??:??:?? 2 MQTT connected
??:??:?? 2 published 1
??:??:?? 2 Subscribing to msh/1/c/LongF/#
??:??:?? 2 [Button] Subscribing to msh/1/json/LongF/#
??:??:?? 3 [RadioIf] Completed sending (id=0x2a0d625f Fr0x08 To0xff, WantAck0, HopLim3 Ch0x6e encrypted priority=64)
??:??:?? 4 [PowerFSM] Enter state: POWER
??:??:?? 4 [Screen] Screen: Powered...
??:??:?? 6 [Screen] Done with boot screen...
??:??:?? 6 [Screen] showing standard frames
??:??:?? 6 [Screen] Showing 0 module frames
??:??:?? 6 [Screen] Total frame count: 83
??:??:?? 6 [Screen] Added modules. numframes: 0
??:??:?? 6 [Screen] Finished building frames. numframes: 3
??:??:?? 7 [Screen] Setting fast framerate
??:??:?? 7 [Screen] Setting idle framerate
??:??:?? 21 [Power] Battery: usbPower=1, isCharging=0, batMv=0, batPct=0
??:??:?? 31 [NodeInfoModule] Sending our nodeinfo to mesh (wantReplies=1)
??:??:?? 31 [NodeInfoModule] sending owner !bdf0c808/Tobias/Tbs
??:??:?? 31 [NodeInfoModule] Update DB node 0xbdf0c808, rx_time=0
??:??:?? 31 [NodeInfoModule] handleReceived(LOCAL) (id=0x2a0d6260 Fr0x08 To0xff, WantAck0, HopLim3 Ch0x0 Portnum=4 WANTRESP priority=10)
??:??:?? 31 [NodeInfoModule] No modules interested in portnum=4, src=LOCAL
??:??:?? 31 [NodeInfoModule] Add packet record (id=0x2a0d6260 Fr0x08 To0xff, WantAck0, HopLim3 Ch0x0 Portnum=4 WANTRESP priority=10)
??:??:?? 31 [NodeInfoModule] Should encrypt MQTT?: 1
??:??:?? 31 [NodeInfoModule] Expanding short PSK #1
??:??:?? 31 [NodeInfoModule] Using AES128 key!
??:??:?? 31 [NodeInfoModule] ESP32 crypt fr=bdf0c808, num=2a0d6260, numBytes=40!
??:??:?? 31 [NodeInfoModule] publish msh/1/c/LongF/!bdf0c808, 83 bytes
??:??:?? 31 [NodeInfoModule] enqueuing for send (id=0x2a0d6260 Fr0x08 To0xff, WantAck0, HopLim3 Ch0x6e encrypted priority=10)
??:??:?? 31 [NodeInfoModule] txGood=1,rxGood=0,rxBad=0
??:??:?? 31 [RadioIf] Starting low level send (id=0x2a0d6260 Fr0x08 To0xff, WantAck0, HopLim3 Ch0x6e encrypted priority=10)
??:??:?? 31 [RadioIf] (bw=250, sf=11, cr=4/8) packet symLen=8 ms, payloadSize=56, time 1034 ms
??:??:?? 31 [RadioIf] AirTime - Packet transmitted : 1034ms
Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled.
Core 1 register dump:
PC : 0x40001277 PS : 0x00060030 A0 : 0x800f2c95 A1 : 0x3ffd5600
A2 : 0x00000000 A3 : 0x3ffc419c A4 : 0x00000000 A5 : 0x00002c40
A6 : 0x00000000 A7 : 0x3ffceaac A8 : 0x80125de1 A9 : 0x3ffd5630
A10 : 0x00000001 A11 : 0x0000006e A12 : 0x00000001 A13 : 0x00000038
A14 : 0x0000006d A15 : 0x00000000 SAR : 0x0000000a EXCCAUSE: 0x0000001c
EXCVADDR: 0x00000000 LBEG : 0x4000c2e0 LEND : 0x4000c2f6 LCOUNT : 0xffffffff
ELF file SHA256: 0000000000000000
Backtrace: 0x40001277:0x3ffd5600 0x400f2c92:0x3ffd5610 0x400f2d2d:0x3ffd56d0 0x401f3287:0x3ffd56f0 0x400fe415:0x3ffd5710 0x400f28f5:0x3ffd5750 0x400d572e:0x3ffd5770 0x400f79b5:0x3ffd5790 0x400dd9a0:0x3ffd57c0 0x4012cc65:0x3ffd57e0
Rebooting...
Looks like it's choking on some mqtt messages. Not entirely sure yet, but I wonder if it could be older versioned message protobufs coming through the public MQTT server.
Here's the decoded trace:
0x400f2c92: MQTT::onPublish(char*, unsigned char*, unsigned int) at /home/runner/work/Meshtastic-device/Meshtastic-device/src/mqtt/MQTT.cpp:70
0x400f2d2d: MQTT::mqttCallback(char*, unsigned char*, unsigned int) at /home/runner/work/Meshtastic-device/Meshtastic-device/src/mqtt/MQTT.cpp:23
0x401f3287: std::_Function_handler::_M_invoke(std::_Any_data const&, char*&&, unsigned char*&&, unsigned int&&) at /home/runner/.platformio/packages/toolchain-xtensa32/xtensa-esp32-elf/include/c++/5.2.0/functional:1871
0x400fe415: std::function::operator()(char*, unsigned char*, unsigned int) const at /home/runner/.platformio/packages/toolchain-xtensa32/xtensa-esp32-elf/include/c++/5.2.0/functional:2271
0x400fe415: PubSubClient::loop() at /home/runner/work/Meshtastic-device/Meshtastic-device/.pio/libdeps/tbeam/PubSubClient/src/PubSubClient.cpp:416
0x400f28f5: MQTT::runOnce() at /home/runner/work/Meshtastic-device/Meshtastic-device/src/mqtt/MQTT.cpp:199
0x400d572e: concurrency::OSThread::run() at /home/runner/work/Meshtastic-device/Meshtastic-device/src/concurrency/OSThread.cpp:78
0x400f79b5: ThreadController::runOrDelay() at /home/runner/work/Meshtastic-device/Meshtastic-device/.pio/libdeps/tbeam/Thread/ThreadController.cpp:59
0x400dd9a0: loop() at /home/runner/work/Meshtastic-device/Meshtastic-device/src/main.cpp:481
0x4012cc65: loopTask(void*) at /home/runner/.platformio/packages/framework-arduinoespressif32/cores/esp32/main.cpp:19
This is not happening with our 1.3 only dev MQTT broker
Just use the public MQTT broker, it is constantly happening there. ;-)
From looking at other Bugs this might be the same underlying issue as #1407 and #1229.
I think it is also a good idea to add a malformed message testcase, if possible.
Still present in 1.3.41:
Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled.
Core 1 register dump:
PC : 0x40001277 PS : 0x00060a30 A0 : 0x800f5c99 A1 : 0x3ffd5550
A2 : 0x00000000 A3 : 0x3ffc418c A4 : 0x00000000 A5 : 0x0000c180
A6 : 0x00000000 A7 : 0x3ffd37c0 A8 : 0x80128735 A9 : 0x3ffd5580
A10 : 0x00000001 A11 : 0x00000074 A12 : 0x00000001 A13 : 0x00000038
A14 : 0x00000073 A15 : 0x00000000 SAR : 0x0000000a EXCCAUSE: 0x0000001c
EXCVADDR: 0x00000000 LBEG : 0x4000c2e0 LEND : 0x4000c2f6 LCOUNT : 0xffffffff
ELF file SHA256: 0000000000000000
Backtrace: 0x40001277:0x3ffd5550 0x400f5c96:0x3ffd5560 0x400f5d31:0x3ffd5620 0x401f5dab:0x3ffd5640 0x401014d5:0x3ffd5660 0x400f58f9:0x3ffd56a0 0x400d832a:0x3ffd56c0 0x400fa9f9:0x3ffd56e0 0x400e08a4:0x3ffd5710 0x4012f5b9:0x3ffd5730
Rebooting...
This has now been released in 1.3.43, the public MQTT should be able to handle both 1.2 and 1.3