Consider publishing tailer separately

[sgtsquiggs]

Hi,

I'm currently using a chunk of mtail as a replacement for hpcloud/tail. I was previously pulling google/mtail/[tailer,watcher,logline], but with the latest changes on master this isn't possible anymore. I agree with the move to internal and I can always maintain my own copy and make updates but it would be very useful to have the tail implementation in mtail exposed as a standalone library. It's outside of the scope of mtail to expose a library, but maybe useful to the golang community as a whole :)

[jaqx0r]

Do you use all of tailer and watcher? I'm not really a fan of them at the moment, but if you can tell me what you like then we can figure out something.

I expected someone to want to do this, so I'm glad you asked.

On Fri, 21 Dec 2018 at 06:48, Matthew Crenshaw [email protected] wrote:

Hi,

I'm currently using a chunk of mtail as a replacement for hpcloud/tail https://github.com/hpcloud/tail. I was previously pulling google/mtail/[tailer,watcher,logline], but with the latest changes on master this isn't possible anymore. I agree with the move to internal and I can always maintain my own copy and make updates but it would be very useful to have the tail implementation in mtail exposed as a standalone library. It's outside of the scope of mtail to expose a library, but maybe useful to the golang community as a whole :)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/google/mtail/issues/199, or mute the thread https://github.com/notifications/unsubscribe-auth/AC5b-wagTDNbivFdSmdvvFZOLpkxak5oks5u6-mdgaJpZM4Zczvc .

[sgtsquiggs]

My usage is very roughly like:

func (t *Tail) Start() error {
	t.Lock()
	defer t.Unlock()
	t.lines := make(chan *logline.LogLine)
	t.watcher := watcher.NewLogWatcher(pollDuration, !poll)
	if err != nil {
		defer close(t.lines)
		return err
	}
	var opts []func(*tailer.Tailer) error
	if t.FromBeginning {
		opts = append(opts, tailer.OneShot)
	}
	t.tailer, err := tailer.New(t.lines, afero.NewOsFs(), t.watcher, opts...)
	if err != nil {
		defer t.watcher.Close()
		defer close(t.lines)
		return err
	}
	t.wg.Add(1)
	go t.processLines()
	return t.tailNew()
}

func (t *Tail) Stop() error {
	t.Lock()
	defer t.Unlock()

	err := t.tailer.Close()
	t.wg.Wait()
	return err
}

// never errors but that's ok for now
func (t *Tail) tailFiles() error {
	for _, filepath := t.Files {
		// snip globbing stuff i'm doing myself
		for file := range globbed {
			if _, ok := t.files[file]; ok {
				continue
			}
			err := t.tailer.TailPath(file)
			if err != nil {
				t.log.Error(err)
				continue
			}
			t.log.Debug("start tailing %s", file)
			t.files[file] = true
		}
	}
	return nil
}

func (t *Tail) processLines() {
	defer t.wg.Done()

	for line = range t.lines {
		// do stuff with line.Line depending on line.Filename
	}

	for _, file := range t.files {
		t.log.Debug("stop tailing %s", file)
	}
}

Other methods add/remove files to be tailed on the fly

[jaqx0r]

Ok so you use the same core functions as mail does. I think I'm happy to move it out of internal and let you depend on it directly.

I don't want to require use of fsnotify though, so the watcher will change, and I don't want to force glog on anyone so that'll change too.

I don't think I'm comfortable with the obligation of making it a public API but I'm sure it'll work out.

On Sat., 22 Dec. 2018, 06:31 Matthew Crenshaw, [email protected] wrote:

My usage is very roughly like:

func (t *Tail) Start() error { t.Lock() defer t.Unlock() t.lines := make(chan *logline.LogLine) t.watcher := watcher.NewLogWatcher(pollDuration, !poll) if err != nil { defer close(t.lines) return err } var opts []func(*tailer.Tailer) error if t.FromBeginning { opts = append(opts, tailer.OneShot) } t.tailer, err := tailer.New(t.lines, afero.NewOsFs(), t.watcher, opts...) if err != nil { defer t.watcher.Close() defer close(t.lines) return err } t.wg.Add(1) go t.processLines() return t.tailNew() } func (t *Tail) Stop() error { t.Lock() defer t.Unlock()

err := t.tailer.Close() t.wg.Wait() return err } // never errors but that's ok for nowfunc (t *Tail) tailFiles() error { for _, filepath := t.Files { // snip globbing stuff i'm doing myself for file := range globbed { if _, ok := t.files[file]; ok { continue } err := t.tailer.TailPath(file) if err != nil { t.log.Error(err) continue } t.log.Debug("start tailing %s", file) t.files[file] = true } } return nil } func (t *Tail) processLines() { defer t.wg.Done()

for line = range t.lines { // do stuff with line.Line depending on line.Filename }

for _, file := range t.files { t.log.Debug("stop tailing %s", file) } }

Other methods add/remove files to be tailed on the fly

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/google/mtail/issues/199#issuecomment-449478473, or mute the thread https://github.com/notifications/unsubscribe-auth/AC5b-64vEvkRT1F-VsQ97ox_1fREKCzyks5u7TcPgaJpZM4Zczvc .

[sgtsquiggs]

Hi @jaqx0r, I started work on separating out the lib here: https://github.com/sgtsquiggs/mtail

Other than removing unrelated mtail code: I removed the dependency on glog. It should still work with glog though :smile:

Please let me know if I'm missing anything license-wise. I need to add a README.

[jnovack]

As another use-case, I've been dying for a simple-evcorr replacement. I grew up on perl, so it's hard for me to bad-mouth it; but in 2019, using perl is not cute anymore.

I love the simple-evcorr idea (and I use it exclusively on my syslog servers for alerting), but it's very primitive and archaic, and mtail has a lot of the core pieces of a replacement.