multiple files?

[gelisam]

Would it make sense to support multiple input files? Today I had a use case in which that would have been helpful. I had used hawk to take the union of a few apache group files, and now I for each of the original files I wanted to compute which groups had gained users due to the union operation. I wanted to reuse the hawk code I had written to create the union, but I since I couldn't easily give hawk two two files it needed to access, it was easier to do the work using other tools.

[melrief]

Yes, I think we should really support multiple input streams. I also used tools to process multiple streams in the past.

A lot of time ago I though about a good system to support multiple stream while maintaining the same syntax. My preliminary idea was to change the type of the user function based on the number of streams: with one stream you must supply a function ByteString -> ByteString while with 2 streams you must supply a function with signature ByteString -> ByteString -> ByteString. The problem that I found when I tried this is that it requires changes at the core of hawk and a lot of work behind. But it should be possible to adapt apply and map mode to this feature.