Streaming Lexer

[gprossliner]

Hi guys!

I'm about to check some implementations for a scanner in go. Because I need to process log files, I can't feed the data at once. I would need a streaming implementation, eventually based on channels.

tokens := make(chan Token)  # consumer
l := lexer.NewLexer()
l.Output(tokens); # 

# new data ([]byte) is feed into scanner, maybe emitting new tokens to Output channel
l.Input <- data  

Do you see if this project could be used / ported for this usecase?