Support HTTP streamer for Sumo Logic

[harlow]

Sumo Logic has a "Hosted Collector" but its via HTTP and not Syslog: https://service.sumologic.com/help/Setting_up_a_Hosted_Collector.htm

If its something that Logspout team would consider as PR for the project I wouldn't mind writing the code for it (just need a little direction):

We have syslog streamers defined here: https://github.com/gliderlabs/logspout/blob/master/router/streamers.go#L43-L54

However, then there is an Exported HttpStreamer but that seems to be writing to HTTP Response object and not a remote HTTP endpoint. https://github.com/gliderlabs/logspout/blob/master/router/streamers.go#L116

A new httpStreamer function might be a bit confusing (as far as naming is concerned). Any suggestions?

[progrium]

We're about to merge the adapter branch into master which has a new API for creating what were previously called "streamers". If you work against that branch now, the easier it will be to pull it in. Depending on specific or uncommon its use might be, we may recommend to have it live as a separate module repo.

[harlow]

@progrium glad I asked! Will make sure to work off of that branch. Thanks

[progrium]

Adapters now merged into master

[raychaser]

hi there. i am actually with Sumo Logic, and i had a rough and tumble implementation pre-refactored Logspout. @progrium you might remember me emailing you about this. just getting back into some hacking with the goal of making an HTTP adapter and noticed this open issue. @harlow do you have any code already? happy to discuss live in #gliderlabs.

[progrium]

Ping me in IRC if anybody needs help working with what's in master. It should be straightforward looking at other adapters, but definitely want to learn any pain points, etc.

On Thu, Apr 9, 2015 at 7:56 PM, raychaser [email protected] wrote:

hi there. i am actually with Sumo Logic, and i had a rough and tumble implementation pre-refactored Logspout. @progrium https://github.com/progrium you might remember me emailing you about this. just getting back into some hacking with the goal of making an HTTP adapter and noticed this open issue. @harlow https://github.com/harlow do you have any code already? happy to discuss live in #gliderlabs.

— Reply to this email directly or view it on GitHub https://github.com/gliderlabs/logspout/issues/56#issuecomment-91391669.

Jeff Lindsay http://progrium.com

[andyshinn]

I'm working in a new environment where Sumo Logic is the log collection service. Was there a branch with some code somewhere that I could test? Anything I could do to help get this off the ground?

[raychaser]

hey Andy, here's what I had done previously: https://github.com/raychaser/logspout-http. i frankly haven't touched this in a while. please let me know what you find. busy Monday morning for me right now, but would love help, and to help see this through!

[benton]

I recently wrote a Logspout adaptor for Amazon Cloudwatch Logs that performs the necessary batching of logspout events into the HTTP-request-sized chunks, based on a maximum batch size (or on a timeout that flushes all batches). While this batching logic is pretty generic, the POSTing part is pretty idiosyncratic to the AWS APIs -- my adaptor does not present any kind of generic HTTP interface.

Now my employer is expressing interest in a similar Logspout adaptor for SumoLogic. So I may take a crack at this soon. Any tips or pointers to example code for consuming the latest version of the Sumo API will be appreciated.

Thanks, -b

[michaelshobbs]

@progrium @andyshinn did we get anywhere with this?

[andyshinn]

Nope, though I'm happy to sponsor work on this feature (bountysource?).

[mrserverless]

I'm interested in this feature and just came across this issue. It appears SumoLogic has recently added syslog source: https://help.sumologic.com/Send_Data/Sources/Syslog_Source Will test it out and confirm.

[igrayson]

@yunspace have you had a chance to test it out?

[od0]

@yunspace @igrayson Tried to set this up just now. It looks like Sumo Logic doesn't yet support syslog for their hosted collectors (only for installed collectors).

[taiidani]

I just came upon this while trying to find an alternative to https://hub.docker.com/r/sumologic/collector/ (collects way too much; doesn't seem configurable). We've used Logspout to great success with Papertrail and I was wondering if there's been any progress on it supporting SumoLogic?

[raychaser]

hey, so it turns out that we released what we are calling "Cloud Syslog" now after all. please have a look at the docs:

https://help.sumologic.com/Send_Data/Sources/02Sources_for_Hosted_Collectors/Cloud_Syslog_Source

this will allow you to just send Syslog directly to Sumo without a need for a collector, or streaming to HTTP.

@taiidani it would be great to hear if this works for you, either here or on http://slack.sumologic.com/. would also be great to hear some more details on why the collector container didn't work for you.

[taiidani]

@raychaser We actually ended up taking the sumologic collector and turning off its Stats collection. That seems to have worked for us. But good to know that Sumo supports aggregating through a syslog endpoint!

[mashayev]

Hey, Any updates here? is logspout support https://help.sumologic.com/Send_Data/Sources/02Sources_for_Hosted_Collectors/Cloud_Syslog_Source?

[mashayev]

Hey, I could sent the logs to sumologic with Cloud syslog collector. The only change was to add environment variable SYSLOG_STRUCTURED_DATA which includes the token provided by sumo collector. I've implemented it with secret, it look like that:

- name: SYSLOG_STRUCTURED_DATA
valueFrom:
  secretKeyRef:
    name: sumologic
    key: sumologic_token