teip icon indicating copy to clipboard operation
teip copied to clipboard

Published 20 hours ago • verified publisher icon greymd

フィールドをコマンドに渡す際に、位置を指定して差込むような機能がほしい

Open blacknon opened this issue 4 years ago • 2 comments

日本語ですみません！ すごいいいツールですね！個人マシンに早速入れさせてもらいました！

使ってて欲しいなと思った機能として、フィールドをコマンドにわたす際に標準入力ではなく xargs -I@みたいな形でフィールドを渡せたら便利そうだなぁと思いました。

$ # syslogの日付をdateに渡して変換する例
$ head syslog
Jun 27 15:25:01 BS-PUB-DEVELOP CRON[6002]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jun 27 15:31:32 BS-PUB-DEVELOP systemd[1]: Starting Cleanup of Temporary Directories...
Jun 27 15:31:32 BS-PUB-DEVELOP systemd-tmpfiles[6004]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
Jun 27 15:31:32 BS-PUB-DEVELOP systemd[1]: Started Cleanup of Temporary Directories.
Jun 27 15:35:01 BS-PUB-DEVELOP CRON[6009]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jun 27 15:36:00 BS-PUB-DEVELOP systemd[1]: Starting Daily apt download activities...
Jun 27 15:36:36 BS-PUB-DEVELOP systemd[1]: Started Daily apt download activities.
Jun 27 15:45:01 BS-PUB-DEVELOP CRON[6279]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jun 27 15:55:01 BS-PUB-DEVELOP CRON[6282]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jun 27 16:05:02 BS-PUB-DEVELOP CRON[6286]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

$ # v1.2.0でやる場合
$ cat syslog | teip -og '^([^ ]+ ){3}' -- gdate -f- "+%Y-%m-%d %H:%M:%S "                                                                                                                                                            [2020/06/27 18:02:11 (土) JST]
2020-06-27 15:25:01 BS-PUB-DEVELOP CRON[6002]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 15:31:32 BS-PUB-DEVELOP systemd[1]: Starting Cleanup of Temporary Directories...
2020-06-27 15:31:32 BS-PUB-DEVELOP systemd-tmpfiles[6004]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
2020-06-27 15:31:32 BS-PUB-DEVELOP systemd[1]: Started Cleanup of Temporary Directories.
2020-06-27 15:35:01 BS-PUB-DEVELOP CRON[6009]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 15:36:00 BS-PUB-DEVELOP systemd[1]: Starting Daily apt download activities...
2020-06-27 15:36:36 BS-PUB-DEVELOP systemd[1]: Started Daily apt download activities.
2020-06-27 15:45:01 BS-PUB-DEVELOP CRON[6279]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 15:55:01 BS-PUB-DEVELOP CRON[6282]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:05:02 BS-PUB-DEVELOP CRON[6286]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:15:01 BS-PUB-DEVELOP CRON[6289]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:17:01 BS-PUB-DEVELOP CRON[6292]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
2020-06-27 16:25:01 BS-PUB-DEVELOP CRON[6296]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:35:01 BS-PUB-DEVELOP CRON[6299]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:45:01 BS-PUB-DEVELOP CRON[6303]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:55:01 BS-PUB-DEVELOP CRON[6306]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 17:05:01 BS-PUB-DEVELOP CRON[6309]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 17:13:14 BS-PUB-DEVELOP systemd[1]: Started Session 1391 of user blacknon.

$ # こんな感じで指定できると嬉しいかも？
$ cat syslog | teip -og '^([^ ]+ ){3}' -I{} -- gdate -d {} "+%Y-%m-%d %H:%M:%S "                                                                                                                                                            [2020/06/27 18:02:11 (土) JST]
2020-06-27 15:25:01 BS-PUB-DEVELOP CRON[6002]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 15:31:32 BS-PUB-DEVELOP systemd[1]: Starting Cleanup of Temporary Directories...
2020-06-27 15:31:32 BS-PUB-DEVELOP systemd-tmpfiles[6004]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
2020-06-27 15:31:32 BS-PUB-DEVELOP systemd[1]: Started Cleanup of Temporary Directories.
2020-06-27 15:35:01 BS-PUB-DEVELOP CRON[6009]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 15:36:00 BS-PUB-DEVELOP systemd[1]: Starting Daily apt download activities...
2020-06-27 15:36:36 BS-PUB-DEVELOP systemd[1]: Started Daily apt download activities.
2020-06-27 15:45:01 BS-PUB-DEVELOP CRON[6279]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 15:55:01 BS-PUB-DEVELOP CRON[6282]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:05:02 BS-PUB-DEVELOP CRON[6286]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:15:01 BS-PUB-DEVELOP CRON[6289]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:17:01 BS-PUB-DEVELOP CRON[6292]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
2020-06-27 16:25:01 BS-PUB-DEVELOP CRON[6296]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:35:01 BS-PUB-DEVELOP CRON[6299]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:45:01 BS-PUB-DEVELOP CRON[6303]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:55:01 BS-PUB-DEVELOP CRON[6306]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 17:05:01 BS-PUB-DEVELOP CRON[6309]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 17:13:14 BS-PUB-DEVELOP systemd[1]: Started Session 1391 of user blacknon.

(このとき、通常だとヒットする箇所ごとにコマンドに渡してると思うんですが、%1 %2 %3...みたいな形で、該当するフィールドを結合して渡せるオプションもあると正規表現で書かないで済みそうなのでとっつきやすいかも思ったんですが、実質的には1個の機能のような気もしてるので、ここで触れさせてください)

$ # さらにこんな感じで指定できると嬉しいかも？
$ cat syslog | teip -f1-3 -I -- gdate -d '\1 \2 \3' "+%Y-%m-%d %H:%M:%S "                                                                                                                                                            [2020/06/27 18:02:11 (土) JST]
2020-06-27 15:25:01 BS-PUB-DEVELOP CRON[6002]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 15:31:32 BS-PUB-DEVELOP systemd[1]: Starting Cleanup of Temporary Directories...
2020-06-27 15:31:32 BS-PUB-DEVELOP systemd-tmpfiles[6004]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
2020-06-27 15:31:32 BS-PUB-DEVELOP systemd[1]: Started Cleanup of Temporary Directories.
2020-06-27 15:35:01 BS-PUB-DEVELOP CRON[6009]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 15:36:00 BS-PUB-DEVELOP systemd[1]: Starting Daily apt download activities...
2020-06-27 15:36:36 BS-PUB-DEVELOP systemd[1]: Started Daily apt download activities.
2020-06-27 15:45:01 BS-PUB-DEVELOP CRON[6279]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 15:55:01 BS-PUB-DEVELOP CRON[6282]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:05:02 BS-PUB-DEVELOP CRON[6286]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:15:01 BS-PUB-DEVELOP CRON[6289]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:17:01 BS-PUB-DEVELOP CRON[6292]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
2020-06-27 16:25:01 BS-PUB-DEVELOP CRON[6296]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:35:01 BS-PUB-DEVELOP CRON[6299]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:45:01 BS-PUB-DEVELOP CRON[6303]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 16:55:01 BS-PUB-DEVELOP CRON[6306]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 17:05:01 BS-PUB-DEVELOP CRON[6309]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2020-06-27 17:13:14 BS-PUB-DEVELOP systemd[1]: Started Session 1391 of user blacknon.

blacknon avatar Jun 27 '20 09:06 blacknon

ご提案ありがとうございます。

おそらくこの機能を実装すると毎行forkが発生してしまうのでパフォーマンスは期待できなくなってしまいますね。 内部的に毎行forkするオプション（-sオプション）が強制的に併用されてしまいますが、とはいえ利便性を上げる機能だと思うので検討してみます。

greymd avatar Jun 28 '20 10:06 greymd 

$ cat syslog | teip -f1-3 -I -- gdate -d '\1 \2 \3' "+%Y-%m-%d %H:%M:%S "

こちらの機能の提案もありがとうございますー。 残念ながら、フィールドを\1,\2のように、後方参照できるようにする機能は実装するつもりはないです。 teipがデータの加工まで始めるとまたややこしい話になるので。。

とはいえ、-f1-3で3つの分裂した選択範囲（トークン）がでてきてしまうのは確かに不便に思う場面はあると思います。 私も薄々それは感じていて、現状のcut互換の範囲指定ではなく、独自の範囲指定ルールのようなものを考えています。 https://github.com/greymd/teip/issues/8

-f1...3で別々のトークン、-f1..3で一つのトークン、のような使い分けができたらいいなーと。

greymd avatar Jun 28 '20 10:06 greymd

フィールドをコマンドに渡す際に位置を指定して差し込む機能ですが v2.3.0 で -I オプションとしてリリースしました。 使い方は README.md をご参照。 -s オプションが強制的に有効になるので、パフォーマンスとしては高くなく、巨大なファイルの扱いには不向きな点にご注意ください。

また、該当するフィールドを結合して渡せるオプションについては、別 Issue #8 で引き続き検討しますので、この Issue については Close にします。

元ファイル

$ zcat test_secure.gz | head
May 26 03:19:26 localhost sshd[17872]: Received disconnect from 192.0.2.152 port 29864:11:  [preauth]
May 26 03:19:26 localhost sshd[17872]: Disconnected from 192.0.2.78 port 29864 [preauth]
May 26 03:21:10 localhost sshd[17927]: Invalid user amavis1 from 192.0.2.148 port 53364
May 26 03:21:10 localhost sshd[17927]: input_userauth_request: invalid user amavis1 [preauth]
May 26 03:21:10 localhost sshd[17927]: Received disconnect from 192.0.2.189 port 53364:11: Bye Bye [preauth]
May 26 03:21:10 localhost sshd[17927]: Disconnected from 192.0.2.201 port 53364 [preauth]
May 26 03:32:03 localhost sshd[18464]: Invalid user postgres from 192.0.2.111 port 48360
May 26 03:32:03 localhost sshd[18464]: input_userauth_request: invalid user postgres [preauth]
May 26 03:32:03 localhost sshd[18464]: Received disconnect from 192.0.2.85 port 48360:11: Normal Shutdown, Thank you for playing [preauth]
May 26 03:32:03 localhost sshd[18464]: Disconnected from 192.0.2.199 port 48360 [preauth]

マスクする

$ zcat test_secure.gz | head | teip -og '^([^ ]+ ){3}'
[May 26 03:19:26 ]localhost sshd[17872]: Received disconnect from 192.0.2.152 port 29864:11:  [preauth]
[May 26 03:19:26 ]localhost sshd[17872]: Disconnected from 192.0.2.78 port 29864 [preauth]
[May 26 03:21:10 ]localhost sshd[17927]: Invalid user amavis1 from 192.0.2.148 port 53364
[May 26 03:21:10 ]localhost sshd[17927]: input_userauth_request: invalid user amavis1 [preauth]
[May 26 03:21:10 ]localhost sshd[17927]: Received disconnect from 192.0.2.189 port 53364:11: Bye Bye [preauth]
[May 26 03:21:10 ]localhost sshd[17927]: Disconnected from 192.0.2.201 port 53364 [preauth]
[May 26 03:32:03 ]localhost sshd[18464]: Invalid user postgres from 192.0.2.111 port 48360
[May 26 03:32:03 ]localhost sshd[18464]: input_userauth_request: invalid user postgres [preauth]
[May 26 03:32:03 ]localhost sshd[18464]: Received disconnect from 192.0.2.85 port 48360:11: Normal Shutdown, Thank you for playing [preauth]
[May 26 03:32:03 ]localhost sshd[18464]: Disconnected from 192.0.2.199 port 48360 [preauth]

変換

$ zcat test_secure.gz | head | teip -og '^([^ ]+ ){3}' -I{} -- date -d {} "+%Y-%m-%d %H:%M:%S "
2023-05-26 03:19:26 localhost sshd[17872]: Received disconnect from 192.0.2.152 port 29864:11:  [preauth]
2023-05-26 03:19:26 localhost sshd[17872]: Disconnected from 192.0.2.78 port 29864 [preauth]
2023-05-26 03:21:10 localhost sshd[17927]: Invalid user amavis1 from 192.0.2.148 port 53364
2023-05-26 03:21:10 localhost sshd[17927]: input_userauth_request: invalid user amavis1 [preauth]
2023-05-26 03:21:10 localhost sshd[17927]: Received disconnect from 192.0.2.189 port 53364:11: Bye Bye [preauth]
2023-05-26 03:21:10 localhost sshd[17927]: Disconnected from 192.0.2.201 port 53364 [preauth]
2023-05-26 03:32:03 localhost sshd[18464]: Invalid user postgres from 192.0.2.111 port 48360
2023-05-26 03:32:03 localhost sshd[18464]: input_userauth_request: invalid user postgres [preauth]
2023-05-26 03:32:03 localhost sshd[18464]: Received disconnect from 192.0.2.85 port 48360:11: Normal Shutdown, Thank you for playing [preauth]
2023-05-26 03:32:03 localhost sshd[18464]: Disconnected from 192.0.2.199 port 48360 [preauth]

greymd avatar Mar 11 '23 14:03 greymd