mosquitto icon indicating copy to clipboard operation
mosquitto copied to clipboard
verified publisher icon eclipse

sometimes appear coredump in libmosquitto.so

Open cpf871744029 opened this issue 3 years ago • 2 comments

mosquitto version:v1.5.4

describe:the server running mosquitto server reboot, client will reconnect server , this problem sometimes happen after server reboot a few seconds, and in mosquitto client log ,we find "Warning: Received PUBREC from storage device1 ctl module 1 for an unknown packet identifier 39."

coredump stack:

Program terminated with signal 11, Segmentation fault.
#0  0x00007f65c6ab859e in malloc_consolidate () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-222.el7.x86_64 libgcc-4.8.5-28.el7_5.1.x86_64 libstdc++-4.8.5-28.el7_5.1.x86_64
(gdb) bt
#0  0x00007f65c6ab859e in malloc_consolidate () from /lib64/libc.so.6
#1  0x00007f65c6aba275 in _int_malloc () from /lib64/libc.so.6
#2  0x00007f65c6abd84c in malloc () from /lib64/libc.so.6
#3  0x00007f65c8c51b02 in packet__read (mosq=0x7f65880009e0) at packet_mosq.c:484
#4  0x00007f65c8c50044 in mosquitto_loop_read (mosq=0x7f65880009e0, max_packets=3) at loop.c:374
#5  0x00007f65c8c504da in mosquitto_loop (mosq=0x7f65880009e0, timeout=<optimized out>, max_packets=1) at loop.c:151
#6  0x00007f65c8c505e6 in mosquitto_loop_forever (mosq=0x7f65880009e0, timeout=-1, max_packets=1) at loop.c:213
#7  0x0000000000411420 in ezmqtt_run (p_data=0x0) at ezmqtt/ezmqtt.c:109
#8  0x00007f65c97d5e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f65c6b36bad in clone () from /lib64/libc.so.6

mosquitto log:

[2022 05 18 12:41:20:732015] TID:2941 [bfums_mos.cpp:1116:bfums_mos_log_cb] [INFO] [010040] mosquitto:Client storage device1 ctl module 1 sending PUBCOMP (Mid: 224)
[2022 05 18 12:41:20:732231] TID:2941 [bfums_mos.cpp:1116:bfums_mos_log_cb] [INFO] [010040] mosquitto:Client storage device1 ctl module 1 received PUBREL (Mid: 225)
[2022 05 18 12:41:20:732262] TID:2941 [ezmqtt/ezmqtt.c:72:ezmqtt_on_message] [INFO] [000000] recv topic err_insert
[2022 05 18 12:41:20:732280] TID:2941 [bfums_mos.cpp:1116:bfums_mos_log_cb] [INFO] [010040] mosquitto:Client storage device1 ctl module 1 sending PUBCOMP (Mid: 225)
[2022 05 18 12:41:20:732303] TID:2941 [bfums_mos.cpp:1116:bfums_mos_log_cb] [INFO] [010040] mosquitto:Client storage device1 ctl module 1 received PUBREL (Mid: 226)
[2022 05 18 12:41:20:732318] TID:2941 [ezmqtt/ezmqtt.c:72:ezmqtt_on_message] [INFO] [000000] recv topic err_insert
[2022 05 18 12:41:20:732333] TID:2941 [bfums_mos.cpp:1116:bfums_mos_log_cb] [INFO] [010040] mosquitto:Client storage device1 ctl module 1 sending PUBCOMP (Mid: 226)
[2022 05 18 12:41:21:147475] TID:2941 [bfums_mos.cpp:1116:bfums_mos_log_cb] [INFO] [010040] mosquitto:Client storage device1 ctl module 1 received PUBLISH (d0, q2, r0, m227, 'err_insert', ... (1044 bytes))
[2022 05 18 12:41:21:147549] TID:2941 [bfums_mos.cpp:1116:bfums_mos_log_cb] [INFO] [010040] mosquitto:Client storage device1 ctl module 1 sending PUBREC (Mid: 227)
[2022 05 18 12:41:21:147854] TID:2941 [bfums_mos.cpp:1116:bfums_mos_log_cb] [INFO] [010040] mosquitto:Client storage device1 ctl module 1 received PUBLISH (d0, q2, r0, m228, 'err_insert', ... (1044 bytes))
[2022 05 18 12:41:21:147895] TID:2941 [bfums_mos.cpp:1116:bfums_mos_log_cb] [INFO] [010040] mosquitto:Client storage device1 ctl module 1 sending PUBREC (Mid: 228)
[2022 05 18 12:41:21:148133] TID:2941 [bfums_mos.cpp:1116:bfums_mos_log_cb] [INFO] [010040] mosquitto:Client storage device1 ctl module 1 received PUBREL (Mid: 227)
[2022 05 18 12:41:21:148201] TID:2941 [ezmqtt/ezmqtt.c:72:ezmqtt_on_message] [INFO] [000000] recv topic err_insert
[2022 05 18 12:41:21:148244] TID:2941 [bfums_mos.cpp:1116:bfums_mos_log_cb] [INFO] [010040] mosquitto:Client storage device1 ctl module 1 sending PUBCOMP (Mid: 227)
[2022 05 18 12:41:21:148504] TID:2941 [bfums_mos.cpp:1116:bfums_mos_log_cb] [INFO] [010040] mosquitto:Client storage device1 ctl module 1 received PUBREL (Mid: 228)
[2022 05 18 12:41:21:148545] TID:2941 [ezmqtt/ezmqtt.c:72:ezmqtt_on_message] [INFO] [000000] recv topic err_insert
[2022 05 18 12:41:21:148569] TID:2941 [bfums_mos.cpp:1116:bfums_mos_log_cb] [INFO] [010040] mosquitto:Client storage device1 ctl module 1 sending PUBCOMP (Mid: 228)
[2022 05 18 12:41:21:410064] TID:2607 [bfums_mos.cpp:1116:bfums_mos_log_cb] [INFO] [010040] mosquitto:Client storage device1 ctl module 1 sending PUBLISH (d0, q2, r0, m39, 'upload_disk_info', ... (1292 bytes))
[2022 05 18 12:41:21:410093] TID:2607 [ezmqtt/ezmqtt.c:0152:ezmqtt_publish] [INFO] [000000] publish topic upload_disk_info, i_messagelen = 1292
[2022 05 18 12:41:21:410425] TID:2941 [bfums_mos.cpp:1116:bfums_mos_log_cb] [INFO] [010040] mosquitto:Client storage device1 ctl module 1 received PUBREC (Mid: 39)
[2022 05 18 12:41:21:410446] TID:2941 [bfums_mos.cpp:1116:bfums_mos_log_cb] [INFO] [010040] mosquitto:Warning: Received PUBREC from storage device1 ctl module 1 for an unknown packet identifier 39.
[2022 05 18 12:41:21:410454] TID:2941 [bfums_mos.cpp:1116:bfums_mos_log_cb] [INFO] [010040] mosquitto:Client storage device1 ctl module 1 sending PUBREL (Mid: 39)
[2022 05 18 12:41:21:410620] TID:2941 [bfums_mos.cpp:1116:bfums_mos_log_cb] [INFO] [010040] mosquitto:Client storage device1 ctl module 1 received PUBCOMP (Mid: 39)

code:

int ezmqtt_init(ezmqtt_config_t *p_cfg, int i_retry_cnt)
{
   int i_ret = 0;
   int i = 0;

    pthread_mutex_lock(&g_ezmqtt_lock);
    if (g_i_init)
    {
        pthread_mutex_unlock(&g_ezmqtt_lock);
        return 0;
    }
   i_ret = mosquitto_lib_init();
   if (i_ret != MOSQ_ERR_SUCCESS)
   {
       pthread_mutex_unlock(&g_ezmqtt_lock);
      return EZMQTT_E_INIT;
   }

   memcpy(&g_st_ezmqt_cfg, p_cfg, sizeof(ezmqtt_config_t));

   g_p_ezmqtt_mosq = mosquitto_new(g_st_ezmqt_cfg.p_id
      , g_st_ezmqt_cfg.b_clean_session, &g_st_ezmqt_cfg);
   if (g_p_ezmqtt_mosq == NULL)
   {
      mosquitto_lib_cleanup();
      pthread_mutex_unlock(&g_ezmqtt_lock);
      return EZMQTT_E_NEW;
   }

   mosquitto_threaded_set(g_p_ezmqtt_mosq, true);
   mosquitto_log_callback_set(g_p_ezmqtt_mosq, ezmqtt_on_log);
   mosquitto_message_callback_set(g_p_ezmqtt_mosq, ezmqtt_on_message);
   mosquitto_connect_callback_set(g_p_ezmqtt_mosq, ezmqtt_on_connect);

    while (1)
    {
       i_ret = mosquitto_connect(g_p_ezmqtt_mosq, g_st_ezmqt_cfg.a_host
          , g_st_ezmqt_cfg.i_port, g_st_ezmqt_cfg.i_keepalive);
        if (i_ret == MOSQ_ERR_SUCCESS)
        {
            break;
        }
        i++;
        if (i == i_retry_cnt)
        {
            break;
        }
        usleep(1000000);
    }
   if (i_ret != MOSQ_ERR_SUCCESS)
   {
      mosquitto_destroy(g_p_ezmqtt_mosq);
      g_p_ezmqtt_mosq = NULL;
      mosquitto_lib_cleanup();
      pthread_mutex_unlock(&g_ezmqtt_lock);
      return EZMQTT_E_CONN;
   }

   g_i_init = 1;
    pthread_mutex_unlock(&g_ezmqtt_lock);
   return EZMQTT_OK;
}

cpf871744029 avatar May 20 '22 09:05 cpf871744029

Thanks for the details you've provided - you haven't include the ezmqtt_run part of the code where the crash is occurring though.

Warning: Received PUBREC from storage device1 ctl module 1 for an unknown packet identifier 39. is not a particular problem on its own, by the way.

Final though - mosquitto 1.5.4 is very old now, I'd suggest updating to 2.0.14.

ralight avatar May 20 '22 14:05 ralight

  1. I use pthread_create(pid, NULL, ezmqtt_run, NULL) create a pthread

void *ezmqtt_run(void *p_data) { mosquitto_loop_forever(g_p_ezmqtt_mosq, -1 ,1) }

  1. 2.0.14 can not compile in os centos 6.10, compile command: make WITH_UUID=no WITH_TLS=no WITH_CJSON=no

error info: typedef 'mosquitto_property' multidefined in mosquitto_broker.h and persist.h

cpf871744029 avatar May 23 '22 03:05 cpf871744029