= Scribe input plugin for Fluentd[http://fluentd.org/]

== Overview

This is a plugin for fluentd[http://fluentd.org] data collector. This plugin adds the Scribe[https://github.com/facebook/scribe] compatible interface to fluentd.

== What's Scribe?

Scribe[https://github.com/facebook/scribe] is a server for aggregating log data streamed in real time from a large number of servers, developed at Facebook.

It uses Thrift[http://thrift.apache.org/], a cross-language RPC framework, to communicate between clients and servers.

== What's Scribe plugin for fluent?

The Scribe plugin for fluentd, which enables fluentd to talk the Scribe protocol. Scribe protocol is defined as follows, in Thrift-IDL format:

enum ResultCode { OK, TRY_LATER }

struct LogEntry { 1: string category, 2: string message }

service scribe extends fb303.FacebookService { ResultCode Log(1: list<LogEntry> messages); }

The category field is used as fluentd 'tag'.

== How to use?

fluent-plugin-scribe contains both input and output.

=== Scribe Input

Please add the following configurations to fluent.conf. This allows your Scribe client to import logs through port 1463.

Scribe input

These options are supported.

• port: port number (default: 1463)
• bind: bind address (default: 0.0.0.0)
• server_type: server architecture either in 'simple', 'threaded', 'thread_pool', 'nonblocking' (default: nonblocking)
• is_framed: use framed protocol or not (default: true)
• add_prefix: prefix string, added to the tag (default: nil)
• msg_format: format of the messages either in 'text', 'json', 'url_param' (default: text)

=== Scribe Output

Please add the following configurations to fluent.conf. This allows fluentd to output its logs into another Scribe server. Note that fluentd conveys semi-structured data while Scribe conveys unstructured data, thus, 'field_ref' parameter is prepared to specify which field will be transferred.

Scribe output

<match *> type scribe host scribed-host.local port 1463 field_ref message

These options are supported.

• host: host name or address (default: localhost)
• port: port number (default: 1463)
• field_ref: field name which sent as scribe log message (default: message)
• timeout: thrift protocol timeout (default: 30)
• format_to_json: if true/yes, format entire record as json, and send as message (default: false)
• remove_prefix: prefix string, removed from the tag (default: nil)

== For Developers

To run fluentd with this plugin on chaging,

$ bundle # (or 'bundle update') $ bundle exec fluentd -v -v -v -c example.conf

Then please execute the sample client.

$ bundle exec bin/fluent-scribe-remote

== Contributors

• {Satoshi Tagomori}[https://github.com/tagomoris]
• {Sadayuki Furuhashi}[https://github.com/frsyuki]
• {Ken Robertson}[https://github.com/krobertson]

== Copyright

Copyright:: Copyright (c) 2011 Treasure Data, Inc. License:: Apache License, Version 2.0