Pinba engine memory leak?

[bvis]

Hi, almost every week we need to restart the "mysqld" service because it gets out of memory when the Pinba engine is under heavy use.

You can see what happens in our cacti report: What we are doing now is a manual restart when we observe the system goes out of memory, but of course this is something not sustainable.

Our configuration is:

mysql> show variables like '%pinba%'; +------------------------------+---------+ | Variable_name | Value | +------------------------------+---------+ | pinba_address | | | pinba_port | 30002 | | pinba_request_pool_size | 1000000 | | pinba_show_protobuf_errors | OFF | | pinba_stats_gathering_period | 10000 | | pinba_stats_history | 300 | | pinba_tag_report_timeout | 600 | | pinba_temp_pool_size | 10000 | +------------------------------+---------+ 8 rows in set (0.00 sec)

[[email protected] ~]# rpm -qa |grep mysql mysql55pinba-plugin-pinba-5.5.15-1 mysql55pinba-libs-5.5.15-1 mysql55pinba-server-5.5.15-1 mysql55pinba-5.5.15-1

Server version: 5.5.15-log

[tony2001]

Do you use any timers?

[bvis]

Yes

[tony2001]

I'd recommend using jemalloc or hoard memory allocator (they tend to reduce memory usage at least twice for me) and add some more memory then. Depending on the way it's used Pinba can consume different amounts of memory - one of the most heavily loaded instances here uses ~15Gb for 3mln records with ~40 timers each.

[bvis]

Yes, but the problem here is that after a week the system degrades as much as being completely unstable, and it's not able to save more info in the Engine. Couldn't this be related with a memory leak? Your proposal is related with increase the efficiency of the system, that's good, of course.

But would this change remove the apparent memory leak?

[tony2001]

Judging from your images, the server just starts swapping because Pinba allocates a lot of memory. I suppose that happens exactly because of the default memory manager and this particular problem can be solved using jemalloc/hoard.

Couldn't this be related with a memory leak?

It's possible there are some leaks, but unlikely - I have several heavy loaded Pinba instances running for months and they consume a constant (but high) amount of memory.

But would this change remove the apparent memory leak?

It's not that apparent to me yet.

[bvis]

Ok, I'll give a try to Hoard. Thanks for your help. I'll go back here once after a week with hoard.

[tony2001]

Sure, let me know of the result.

[bvis]

HI again,

After 1 week working with HOARD here you have the result:

With this info:

ldd /usr/lib64/mysql/plugin/libpinba_engine.so.0.0.0 linux-vdso.so.1 => (0x00007fffa0fff000) libm.so.6 => /lib64/libm.so.6 (0x00007f493aae3000) libprotobuf-lite.so.7 => /usr/lib64/libprotobuf-lite.so.7 (0x00007f493a8b7000) libevent-1.4.so.2 => /usr/lib64/libevent-1.4.so.2 (0x00007f493a69c000) libJudy.so.1 => /usr/lib64/libJudy.so.1 (0x00007f493a44e000) libhoard.so => /usr/lib64/libhoard.so (0x00007f493a1f3000) libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f4939eed000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f4939cd7000) libc.so.6 => /lib64/libc.so.6 (0x00007f4939946000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f493972a000) libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f4939511000) librt.so.1 => /lib64/librt.so.1 (0x00007f4939308000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f49390ee000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f4938eea000) /lib64/ld-linux-x86-64.so.2 (0x0000003b6f600000)

[[email protected] ~]# rpm -qf /usr/lib64/mysql/plugin/libpinba_engine.so.0.0.0 mysql55pinba-server-5.5.15-6.x86_64 mysql55pinba-plugin-pinba-5.5.15-6.x86_64

With the same engine variables:

[pinba] > show variables like '%pinba%';
+------------------------------+---------+
| Variable_name                | Value   |
+------------------------------+---------+
| pinba_address                |         |
| pinba_port                   | 30002   |
| pinba_request_pool_size      | 1000000 |
| pinba_show_protobuf_errors   | OFF     |
| pinba_stats_gathering_period | 10000   |
| pinba_stats_history          | 300     |
| pinba_tag_report_timeout     | 600     |
| pinba_temp_pool_size         | 10000   |
+------------------------------+---------+
8 rows in set (0.01 sec)

[tony2001]

How did you run it with Hoard? I can see almost no difference in your diagrams and it should be much more considerable imo.

How much timers are usually in timer table? (select count() from timer) How much tags are used in timers? (select count() from timertag)

For example, one of Pinba instances here consumes ~16Gb while storing about 60mln timers (that's about 40 per request) with ~2 tags each and that's expected - I have to store all those timers and the values of their tags somewhere.

[bvis]

Hi, we don't execute anything special, but we compiled it with the flag "--with-hoard", and we have this information:

The file /proc/[pid]/maps is a way to see the memory regions mapped by a process.

7fa2e1717000-7fa2e1748000 r-xp 00000000 fd:00 668265 /usr/lib64/libhoard.so
7fa2e1748000-7fa2e1948000 ---p 00031000 fd:00 668265 /usr/lib64/libhoard.so
7fa2e1948000-7fa2e194a000 rw-p 00031000 fd:00 668265 /usr/lib64/libhoard.so

And:

[pinba] > select count(*) from timer;
+----------+
| count(*) |
+----------+
|  3489332 |
+----------+
1 row in set (0.01 sec)
[pinba] > select count(*) from timertag;
+----------+
| count(*) |
+----------+
|  7303430 |
+----------+
1 row in set (0.00 sec)