pinba-engine does not compile under osx (homebrew)

[neanton]

Here is a message I get when I'm trying to build pinba.

ha_pinba.cc:2687:15: error: no matching function for call to 'JudyLNext'
                                ppvalue = JudyLNext(D->tag.name_index, &str_hash, NULL);
                                          ^~~~~~~~~
/usr/local/Cellar/judy/1.0.5/include/Judy.h:263:17: note: candidate function not viable: no known conversion from 'uint64_t *' (aka 'unsigned long long *') to 'Word_t *' (aka 'unsigned long *') for 2nd argument
extern PPvoid_t JudyLNext(       Pcvoid_t  PArray, Word_t * PIndex,  P_JE);
                ^
1 error generated.

I have following source packages:

mysql-engine-pinba 1.1.0
mysql 5.6.24
judy 1.0.5
libevent 2.0.22
protobuf 2.6.1

Seems like changing type of str_hash to Word_t solves the issue but I'm not sure it's a correct way of fixing this. Maybe related with 64-bit platform?

[nhatthm]

+1, I have the same error, cannot compile pinba 1.1.0

[tony2001]

Does this patch help: https://gist.github.com/17ee3f9cde526a4eae28 ?

[nhatthm]

Yes it does, thanks!

[tony2001]

Are you using OS X too, by any chance?

[neanton]

I can confirm it compiles now, but mysql service fails to stop. Same happens with Percona MySQL. The last line I see in MySQL log file is:

Shutting down plugin 'PINBA'

Here is the output of lldb backtrace:

(lldb) bt
* thread #1: tid = 0x2b6946, 0x00007fff9419d136 libsystem_kernel.dylib`__psynch_cvwait + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007fff9419d136 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff93fc6e0c libsystem_pthread.dylib`_pthread_cond_wait + 693
    frame #2: 0x000000010e30f276 mysqld`mysqld_main(int, char**) + 9909
    frame #3: 0x00007fff91c225c9 libdyld.dylib`start + 1

I can provide additional information if required.

[tony2001]

Ok, what do you get with thread info in gdb? You should see quite a lot of threads and some of them are apparently stuck somewhere, so the main thread is waiting for them to finish and quit.

[neanton]

Here is the output:

(gdb) info threads
  Id   Target Id         Frame 
  27   Thread 0x3103 of process 83964 0x00007fff9419d48a in __semwait_signal () from /usr/lib/system/libsystem_kernel.dylib
  26   Thread 0x3003 of process 83964 0x00007fff9419d75a in __sigwait () from /usr/lib/system/libsystem_kernel.dylib
  25   Thread 0x2f03 of process 83964 0x00007fff9419e21a in kevent () from /usr/lib/system/libsystem_kernel.dylib
  24   Thread 0x2e03 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  23   Thread 0x2d03 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  22   Thread 0x2c03 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  21   Thread 0x2b03 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  20   Thread 0x2a03 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  19   Thread 0x2903 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  18   Thread 0x2803 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  17   Thread 0x2703 of process 83964 0x00007fff9419d3fa in select$DARWIN_EXTSN () from /usr/lib/system/libsystem_kernel.dylib
  16   Thread 0x2603 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  15   Thread 0x2503 of process 83964 0x00007fff9419d3fa in select$DARWIN_EXTSN () from /usr/lib/system/libsystem_kernel.dylib
  14   Thread 0x2403 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  13   Thread 0x2303 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  12   Thread 0x2203 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  11   Thread 0x2103 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  10   Thread 0x2003 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  9    Thread 0x1f03 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  8    Thread 0x1e03 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  7    Thread 0x1d03 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  6    Thread 0x1c03 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  5    Thread 0x1b03 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  4    Thread 0x1a03 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  3    Thread 0x1903 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
  2    Thread 0x1803 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib
* 1    Thread 0x1703 of process 83964 0x00007fff9419d136 in __psynch_cvwait () from /usr/lib/system/libsystem_kernel.dylib

Thread 25 shows somethig related to pinba:

(gdb) thread 25
[Switching to thread 25 (Thread 0x2f03 of process 83964)]
#0  0x00007fff9419e21a in kevent () from /usr/lib/system/libsystem_kernel.dylib
(gdb) bt
#0  0x00007fff9419e21a in kevent () from /usr/lib/system/libsystem_kernel.dylib
#1  0x000000011f1d7fa5 in kq_dispatch () from /usr/local/lib/libevent-2.0.5.dylib
#2  0x000000011f1c73e6 in event_base_loop () from /usr/local/lib/libevent-2.0.5.dylib
#3  0x000000011f1b35cd in pinba_collector_main(void*) () from /usr/local/Cellar/mysql/5.6.24/lib/plugin/libpinba_engine.so
#4  0x00007fff93fc6268 in const_upshift () from /usr/lib/system/libsystem_pthread.dylib
#5  0x0000000000002703 in ?? ()
#6  0x000000014d550000 in ?? ()
#7  0x000000014d54ff50 in ?? ()
#8  0x00007fff93fc61e5 in const_upshift () from /usr/lib/system/libsystem_pthread.dylib
#9  0x0000000000000000 in ?? ()

[tony2001]

Okay, so pthread_cancel() doesn't seem to break the eventloop on OS X. I've seen a similar problem on FreeBSD iirc. I'll try look to it up and see if there's something that can be done.

[tony2001]

If you're using this MySQL instance for Pinba only (which I do recommend), then you can safely kill it with kill, since Pinba doesn't write any data on the disk.

[neanton]

Okay. Thanks. Ping me later if you'll need some additional info/tests.

[tony2001]

Yes, it seems that this problem is inherited throughout all *BSD family. Please try this patch: https://gist.github.com/tony2001/9ba9067be681d8febb8a

[neanton]

Does not help. Please see, I've done a backtrace on all running threads on server shutdown. https://gist.github.com/neanton/e4f7e1e2ddcd37214ecb

Maybe I should recompile MySQL/pinba-engine with debugging support to help finding out the issue?

[tony2001]

Yes, recompiling MySQL and pinba with debug symbols would help to pinpoint the place where it's stuck, but I'm fairly sure I know where it is. The problem is that libevent uses kevent() syscall on *BSD and kevent() is not interrupted by pthread_cancel()/pthread_kill(). I'll try tomorrow to do a more complex patch to fix it.

[tony2001]

Please try & see if branch devel_kevent_workaround fixes it for you.

[neanton]

Not sure if I've compiled it properly, but it does not work yet. Here is the output I was able to get recompiling MySQL and Pinba with debug support. https://gist.github.com/neanton/1942884e8bd7075095fd

BTW, INSTALL PLUGIN pinba... also hangs current cli server connection.

[nhatthm]

Hi, can we have the fix (for compiling) in master branch and release 1.1.1 first? Then we can keep investigating and fix the issue with shutting down later?

[vincentbernat]

I also have the same compilation problem on Linux i386:

  ha_pinba.cc: In member function 'int ha_pinba::read_next_row(unsigned char*, uint, bool)':
  ha_pinba.cc:2687:59: error: cannot convert 'uint64_t* {aka long long unsigned int*}' to 'Word_t* {aka long unsigned int*}' for argument '2' to 'void** JudyLNext(Pcvoid_t, Word_t*, PJError_t)'
       ppvalue = JudyLNext(D->tag.name_index, &str_hash, NULL);

The initial patch proposed by @tony2001 is not available anymore. Does changing the type of str_hash to Word_t would work as expected?

[nhatthm]

Here is the patch

diff --git a/src/ha_pinba.cc b/src/ha_pinba.cc
index 8c71010..85193bb 100644
--- a/src/ha_pinba.cc
+++ b/src/ha_pinba.cc
@@ -2684,7 +2684,7 @@ int ha_pinba::read_next_row(unsigned char *buf, uint active_index, bool by_key)

                str_hash = this_index[active_index].ival;

-               ppvalue = JudyLNext(D->tag.name_index, &str_hash, NULL);
+               ppvalue = JudyLNext(D->tag.name_index, (Word_t *)&str_hash, NULL);
                if (!ppvalue) {
                    ret = HA_ERR_END_OF_FILE;
                    goto failure;

It works for me

[vincentbernat]

It would break on big endian architectures, I think. The value pointed for JudyLNext will be different from the value stored a few lines later in the index:

this_index[active_index].ival = str_hash;

Maybe this is OK because this is consistent with what is done everywhere but since this is the only place where we do that, I think this is not.

[juricast]

Hello,

I am sending a patch for this package. This was already suggested by vincentbernat here.

With this patch package was successfully built on mips, mipsel, i386 on Debian. Package is also built on 64 bit archs without an issue when this patch is used.

I have changed a type of str_hash in file src/ha_pinba.cc. Patch:

--- pinba-engine-mysql-1.1.0.orig/src/ha_pinba.cc
+++ pinba-engine-mysql-1.1.0/src/ha_pinba.cc
@@ -2680,7 +2680,7 @@ int ha_pinba::read_next_row(unsigned cha
                                PPvoid_t ppvalue;
                                char name[PINBA_MAX_LINE_LEN] = {0};
                                pinba_tag *tag;
-                               uint64_t str_hash;
+                               Word_t str_hash;
                                str_hash = this_index[active_index].ival;

Loking at a pinba-engine-mysql-1.1.0/src/ha_pinba.h one can notice that ival is of type size_t:

typedef struct pinba_index_st { /* {{{ */
        union {
                size_t ival;
                struct {
                        unsigned char *val;
                        uint len;
                } str;
        };
        struct {
                unsigned char *val;
                uint len;
        } subindex;
        size_t position;
} pinba_index_st;
/* }}} */

However if we replace "uint64_t str_hash;" with "size_t str_hash;" on 32bit archs we have following problem:

ha_pinba.cc:2687:59: error: invalid conversion from 'size_t* {aka unsigned int*}' to 'Word_t* {aka long unsigned int*}' [-fpermissive]

Taking a look at a Word_t one can find that: A Word_t is a typedef unsigned long int in Judy.h and must be the same size as sizeof(void *) I.E. a pointer.

Same size as size_t but on 32bit one is "unsigned int" (size_t) and the other "long unsigned int" (Word_t) While on 64bit they are both "long unsigned int".

Taking this into consideration I have proposed a patch that I think it is safer than previously proposed patch that include type casting, which will probably fail during run-time on big-endian.

Could you please consider including this patch?

Thank you!

Regards, Jurica

[latinovic]

The patch that Jurica had proposed worked for me. I was able to build pinba-engine-mysql on Debian for mips, mipsel, i386. This patch was proposed here as well: https://bugs.debian.org/793511 Any concerns for including it?