TableMapEventData instance increase always

[hope-onely]

In my project,I found TableMapEventData instance increase always,like this:

jmap -histo:live 92694 |head -n 100

num #instances #bytes class name

1: 5225216 962730608 [I 2: 10513406 537535568 [C 3: 5230996 329332816 [B 4: 10513240 252317760 java.lang.String 5: 5224967 208998680 com.github.shyiko.mysql.binlog.event.TableMapEventData 6: 5229496 167343872 java.util.HashMap$Entry

I found this code case this problem,which in com\github\shyiko\mysql\binlog\event\deserialization\EventDeserializer.java.

public Event nextEvent(ByteArrayInputStream inputStream) throws IOException { if (inputStream.peek() == -1) { return null; } EventHeader eventHeader = eventHeaderDeserializer.deserialize(inputStream); EventDataDeserializer eventDataDeserializer = getEventDataDeserializer(eventHeader.getEventType()); if (eventHeader.getEventType() == EventType.TABLE_MAP && tableMapEventDataDeserializer != null) { eventDataDeserializer = tableMapEventDataDeserializer; } EventData eventData = deserializeEventData(inputStream, eventHeader, eventDataDeserializer); if (eventHeader.getEventType() == EventType.TABLE_MAP) { TableMapEventData tableMapEvent; if (eventData instanceof EventDataWrapper) { EventDataWrapper eventDataWrapper = (EventDataWrapper) eventData; tableMapEvent = (TableMapEventData) eventDataWrapper.getInternal(); if (tableMapEventDataDeserializer != null) { eventData = eventDataWrapper.getExternal(); } } else { tableMapEvent = (TableMapEventData) eventData; } tableMapEventByTableId.put(tableMapEvent.getTableId(), tableMapEvent); } return new Event(eventHeader, eventData); }

if table cache not big enough will case this problem,reason the table id is not union mapping to the table in some time. so i think TableMapEventData object need remove when used complete.

[shyiko]

Hi @hope-onely. tableId stays the same as long as table is not deleted. It seems like you have something that DROP/CREATE|ed 5 224 967 tables.

A quick fix is to provide custom EventDeserializer::tableMapEventByTableId (you can grab the code from here) but instead of

final Map<Long, TableMapEventData> tableMapEventByTableId = new HashMap<Long, TableMapEventData>();

use

final Map<Long, TableMapEventData> tableMapEventByTableId = new LinkedHashMap<Long, TableMapEventData>(0, 1, true) {

    @Override
    protected boolean removeEldestEntry(Map.Entry<Long, TableMapEventData> eldest) {
        return size() > 1;
    }
};

(or similar).

I'll keep this ticket open as a reminder for 1.0.0.

[darnaut]

tableId stays the same as long as table is not deleted.

Not only that. It changes whenever the table is added to MySQL's table cache. For example:

conn1> BEGIN;
conn1> INSERT INTO t1 VALUES (1);
conn2> FLUSH TABLES;
conn1> INSERT INTO t1 VALUES (2);
conn1> COMMIT;

Produces the following events:

+----------------+-------------------------------------+
| Event_type     | Info                                |
+----------------+-------------------------------------+
| Query          | BEGIN                               |
| Table_map      | table_id: 226 (test.t2)             |
| Write_rows     | table_id: 226 flags: STMT_END_F     |
| Table_map      | table_id: 227 (test.t2)             |
| Write_rows     | table_id: 227 flags: STMT_END_F     |
| Xid            | COMMIT /* xid=54 */                 |
+----------------+-------------------------------------+

The ids are assigned whenever a table is added to MySQL’s internal table cache. If a table is kicked out of the table cache and is later accessed again (that is, table cache becomes full scenario, or flush tables, or table definition is altered), it will have a different id. Inside an event group, a table map/id is only valid until an event with the STMT_END_F flag is seen.

[shyiko]

@darnaut Thank you! I wasn't aware of that. One more reason to remove tableMapEventByTableId. :bow:

[az3]

I've made some tests and I think the tableId in TABLE_MAP event only changes if there is an alteration in the table structure in that database. That is, mysql STOP/START events effect those numbers. I think there is another cache for those tableIds.

So you should delete your cache if you see a ROTATE event in case the tableId will point to a different table name. I've come across that case, and that was very painful to debug.

update mytable1 set num=1 where id=1; -- binlog will mark mytable1 with tableId=2033
create table atest (id int);
create table ztest (id int);
-- stop and start db
update mytable1 set num=1 where id=1; -- binlog will mark mytable1 with tableId=125 , and a completely unrelated mytable2 might be tagged with tableId 2033 miraculously and if you're very unlucky.

Those tableId values are different from the TABLE_ID value shown in INFORMATION_SCHEMA.INNODB_SYS_TABLES as they are completely different. I think the TABLE_ID for INNODB will stay the same as long as table is not deleted. But not the tableId in TABLE_MAP events.

You are free to investigate log_event.h , I couldn't read that thoroughly.

[fabiocatalao]

I'm catching all TableMapEventData events and I'm using an in-memory structure where my app saves tableId and corresponding table information. Every time I receive a new TableMapEventData event I refresh my data.

[SuyuZhuang]

@fabiocatalao What if an old data_id message is not consumed after you refresh the cache? Or we can say all messages are consumed in sequence?