fluent-plugin-mysql-replicator
fluent-plugin-mysql-replicator copied to clipboard
Fluentd input plugin to track insert/update/delete event from MySQL databases.
fluent-plugin-mysql-replicator ![Build Status](https://travis-ci.org/y-ken/fluent-plugin-mysql-replicator.png?branch=master)
Overview
Fluentd input plugin to track insert/update/delete event from MySQL database server.
Not only that, it could multiple table replication into single or multi Elasticsearch/Solr.
It's comming support replicate to another RDB/noSQL.
Requirements
fluent-plugin-mysql-replicator | fluentd | ruby |
---|---|---|
>= 0.6.1 | >= v0.14.x | >= 2.1 |
<= 0.6.1 | >= v0.12.x | >= 1.9 |
Dependency
before use, install dependent library as:
# for RHEL/CentOS
$ sudo yum group install "Development Tools"
# for Ubuntu/Debian
$ sudo apt-get install build-essential
Installation
install with gem or fluent-gem command as:
# for system installed fluentd
$ gem install fluent-plugin-mysql-replicator -v 1.0.3
# for td-agent2
$ sudo td-agent-gem install fluent-plugin-mysql-replicator -v 0.6.1
# for td-agent3
$ sudo td-agent-gem install fluent-plugin-mysql-replicator -v 1.0.3
Included plugins
- Input Plugin: mysql_replicator
- Input Plugin: mysql_replicator_multi
- Output Plugin: mysql_replicator_elasticsearch
- Output Plugin: mysql_replicator_solr (experimental)
Output example
It is a example when detecting insert/update/delete events.
sample query
$ mysql -e "create database myweb"
$ mysql myweb -e "create table search_test(id int auto_increment, text text, PRIMARY KEY (id))"
$ sleep 10
$ mysql myweb -e "insert into search_test(text) values('aaa')"
$ sleep 10
$ mysql myweb -e "update search_test set text='bbb' where text = 'aaa'"
$ sleep 10
$ mysql myweb -e "delete from search_test where text='bbb'"
result
$ tail -f /var/log/td-agent/td-agent.log
2013-11-25 18:22:25 +0900 replicator.myweb.search_test.insert.id: {"id":"1","text":"aaa"}
2013-11-25 18:22:35 +0900 replicator.myweb.search_test.update.id: {"id":"1","text":"bbb"}
2013-11-25 18:22:45 +0900 replicator.myweb.search_test.delete.id: {"id":"1"}
Tutorial
mysql_replicator
It is easy to try it on this plugin quickly.
For more detail are described at Tutorial-mysql_replicator.md
Features
- Table (or view table) synchronization supported.
- Replicate small record under a millons table.
- It is recommend to use insert only table.
- Nested documents are supported with placeholder which accessing to temporary table created at the each loop.
Examples
mysql_replicator_multi
It replicates a millions of records and/or multiple tables with multiple threads.
This architecture is storing hash table in MySQL management table instead of ruby internal memory.
See tutorial at Tutorial-mysql_replicator_multi.md
Features
- table (or view table) synchronization supported.
- Multiple table synchronization supported and its DSN stored in MySQL management table.
- Using MySQL database as hash table cache to support replicate over a millions table.
- It is recommend to make whole copy of tables.
- Nested documents are supported with placeholder which accessing to temporary table created at the each loop.
Examples
Articles
-
MySQLテーブルへの更新/削除イベントを逐次取得するFluentdプラグイン「fluent-plugin-mysql-replicator」をリリースしました - Y-Ken Studio
http://y-ken.hatenablog.com/entry/fluent-plugin-mysql-replicator-has-released -
MySQLユーザ視点での小さく始めるElasticsearch
http://www.slideshare.net/y-ken/introducing-elasticsearch-for-mysql-users -
MySQLからelasticsearchへ、レコードをネスト構造化しつつ同期出来る fluent-plugin-mysql-replicator v0.4.0 を公開しました - Y-Ken Studio
http://y-ken.hatenablog.com/entry/fluent-plugin-mysql-repicator-v0.4.0
TODO
Pull requests are very welcome like below!!
- more documents
- more tests with mock.
- support string type of primary_key.
- support reload setting on demand.
Copyright
Copyright © 2013- Kentaro Yoshida (@yoshi_ken)
License
Apache License, Version 2.0