tcpdp
tcpdp copied to clipboard
tcpdp is TCP dump tool with custom dumper and structured logger written in Go.
tcpdp
tcpdp is TCP dump tool with custom dumper and structured logger written in Go.
tcpdp
has 3 modes:
- TCP Proxy server mode
- Probe mode ( using libpcap )
- Read pcap file mode
Usage
tcpdp proxy
: TCP proxy server mode
$ tcpdp proxy -l localhost:12345 -r localhost:1234 -d hex # hex.Dump()
$ tcpdp proxy -l localhost:55432 -r db.internal.example.com:5432 -d pg # Dump query of PostgreSQL
$ tcpdp proxy -l localhost:33306 -r db.example.com:3306 -d mysql # Dump query of MySQL
With server-starter
https://github.com/lestrrat-go/server-starter
$ start_server --port 33306 -- tcpdp proxy -s -r db.example.com:3306 -d mysql
With config file
$ tcpdp proxy -c config.toml
tcpdp probe
: Probe mode (like tcpdump)
$ tcpdp probe -i lo0 -t localhost:3306 -d mysql # is almost the same setting as 'tcpdump -i lo0 host 127.0.0.1 and tcp port 3306'
$ tcpdp probe -i eth0 -t 3306 -d hex # is almost the same setting as 'tcpdump -i eth0 tcp port 3306'
tcpdp read
: Read pcap file mode
$ tcpdump -i eth0 host 127.0.0.1 and tcp port 3306 -w mysql.pcap
$ tcpdp read mysql.pcap -d mysql -t 3306 -f ltsv
tcpdp config
Create config
$ tcpdp config > myconfig.toml
Show current config
$ tcpdp config
config format
[tcpdp]
pidfile = "/var/run/tcpdp.pid"
dumper = "mysql"
[probe]
target = "db.example.com:3306"
interface = "en0"
bufferSize = "2MB"
immediateMode = false
snapshotLength = "auto"
internalBufferLength = 10000
filter = ""
[proxy]
useServerStarter = false
listenAddr = "localhost:3306"
remoteAddr = "db.example.com:3306"
[log]
dir = "/var/log/tcpdp"
enable = true
enableInternal = true
stdout = true
format = "ltsv"
rotateEnable = true
rotationTime = "daily"
rotationCount = 7
# You can execute arbitrary commands after rotate
# $1 = prev filename
# $2 = current filename
rotationHook = "/path/to/after_rotate.sh"
fileName = "tcpdp.log"
[dumpLog]
dir = "/var/log/dump"
enable = true
stdout = false
format = "json"
rotateEnable = true
rotationTime = "hourly"
rotationCount = 24
fileName = "dump.log"
Installation
$ go get github.com/k1LoW/tcpdp
Architecture
tcpdp proxy connection diagram
client_addr
^
| tcpdp
+----------|---------------+
| v |
| proxy_listen_addr |
| + ^ |
| | | +--------+ |
| |<----+ dumper | |
| | |<--+ | |
| | | +--------+ |
| v + |
| proxy_client_addr |
| ^ |
+----------|---------------+
|
v
remote_addr
tcpdp probe connection diagram
server
+--------------------------+
| |
| +---+---+
| <--------------| eth0 |----------->
| interface +---+---+
| /target ^ |
| | |
| tcpdp | |
| +--------+ | |
| | dumper +------+ |
| +--------+ |
+--------------------------+
tcpdp read diagram
tcpdp
+--------+ STDIN +--------+ STDOUT
| *.pcap +------>+ dumper +-------->
+--------+ +--------+
tcpdp.log ( tcpdp proxy
or tcpdp probe
)
key | description | mode |
---|---|---|
ts | timestamp | proxy / probe / read |
level | log level | proxy / probe |
msg | log message | proxy / probe |
error | error info | proxy / probe |
caller | error caller | proxy / probe |
conn_id | TCP connection ID by tcpdp | proxy / probe |
target | probe target | proxy / probe |
dumper | dumper type | proxy / probe |
use_server_starter | use server_starter | proxy |
conn_seq_num | TCP comunication sequence number by tcpdp | proxy |
client_addr | client address | tcpdp.log, hex, mysql, pg |
remote_addr | remote address | proxy |
proxy_listen_addr | listen address | proxy |
direction | client to remote: -> / remote to client: <- |
proxy |
interface | probe target interface | probe |
mtu | interface MTU (Maximum Transmission Unit) | probe |
mss | TCP connection MSS (Max Segment Size) | probe |
probe_target_addr | probe target address | probe |
filter | BPF (Berkeley Packet Filter) | probe |
buffer_size | libpcap buffer_size | probe |
immediate_mode | libpcap immediate_mode | probe |
snapshot_length | libpcap snapshot length | probe |
internal_buffer_length | tcpdp internal packet buffer length | probe |
Dumper
mysql
MySQL query dumper
NOTICE: MySQL query dumper require --target
option when tcpdp proxy
tcpdp probe
key | description | mode |
---|---|---|
ts | timestamp | proxy / probe / read |
conn_id | TCP connection ID by tcpdp | proxy / probe / read |
conn_seq_num | TCP comunication sequence number by tcpdp | proxy |
client_addr | client address | proxy |
proxy_listen_addr | listen address | proxy |
proxy_client_addr | proxy client address | proxy |
remote_addr | remote address | proxy |
direction | client to remote: -> / remote to client: <- |
proxy |
interface | probe target interface | probe |
src_addr | src address | probe / read |
dst_addr | dst address | probe / read |
probe_target_addr | probe target address | probe |
proxy_protocol_src_addr | proxy protocol src address | probe / proxy /read |
proxy_protocol_dst_addr | proxy protocol dst address | probe / proxy /read |
query | SQL query | proxy / probe / read |
stmt_id | statement id | proxy / probe / read |
stmt_prepare_query | prepared statement query | proxy / probe / read |
stmt_execute_values | prepared statement execute values | proxy / probe / read |
character_set | character set | proxy / probe / read |
username | username | proxy / probe / read |
database | database | proxy / probe / read |
seq_num | sequence number by MySQL | proxy / probe / read |
command_id | command_id for MySQL | proxy / probe / read |
pg
PostgreSQL query dumper
NOTICE: PostgreSQL query dumper require --target
option tcpdp proxy
tcpdp probe
key | description | mode |
---|---|---|
ts | timestamp | proxy / probe / read |
conn_id | TCP connection ID by tcpdp | proxy / probe / read |
conn_seq_num | TCP comunication sequence number by tcpdp | proxy |
client_addr | client address | proxy |
proxy_listen_addr | listen address | proxy |
proxy_client_addr | proxy client address | proxy |
remote_addr | remote address | proxy |
direction | client to remote: -> / remote to client: <- |
proxy |
interface | probe target interface | probe |
src_addr | src address | probe / read |
dst_addr | dst address | probe / read |
probe_target_addr | probe target address | probe |
proxy_protocol_src_addr | proxy protocol src address | probe / proxy /read |
proxy_protocol_dst_addr | proxy protocol dst address | probe / proxy /read |
query | SQL query | proxy / probe / read |
portal_name | portal Name | proxy / probe / read |
stmt_name | prepared statement name | proxy / probe / read |
parse_query | prepared statement query | proxy / probe / read |
bind_values | prepared statement bind(execute) values | proxy / probe / read |
username | username | proxy / probe / read |
database | database | proxy / probe / read |
message_type | message type for PostgreSQL | proxy / probe / read |
hex
key | description | mode |
---|---|---|
ts | timestamp | proxy / probe / read |
conn_id | TCP connection ID by tcpdp | proxy / probe / read |
conn_seq_num | TCP comunication sequence number by tcpdp | proxy |
client_addr | client address | proxy |
proxy_listen_addr | listen address | proxy |
proxy_client_addr | proxy client address | proxy |
remote_addr | remote address | proxy |
direction | client to remote: -> / remote to client: <- |
proxy |
interface | probe target interface | probe |
src_addr | src address | probe / read |
dst_addr | dst address | probe / read |
probe_target_addr | probe target address | probe |
proxy_protocol_src_addr | proxy protocol src address | probe / proxy /read |
proxy_protocol_dst_addr | proxy protocol dst address | probe / proxy /read |
bytes | bytes string by hex.Dump | proxy / probe / read |
ascii | ascii string by hex.Dump | proxy / probe / read |
References
- https://github.com/jpillora/go-tcp-proxy
- https://github.com/dmmlabo/tcpserver_go