Re-org the whole damn thing

[coolacid]

Need to reorg the whole thing.

• Snippets -- These are single bits that do cool things
• Examples -- These are examples to use specific inputs/filters/codecs/outputs
• Configs -- These are example configs to work with a specific log type
• Tuturials -- A set of configurations that are a step by step example of how to do something.

[shurane]

+1.

I'm thinking of collecting example logs from

1. IRC
2. logstash-users mailing list: https://groups.google.com/forum/#!forum/logstash-users
3. logstash-cookbook repo: https://github.com/logstash/cookbook
4. https://github.com/jamtur01/logstashbook-code

and creating a git repo with all the ones I find. It would be a good idea to congregate them in a single place.

Do you have an idea on how to make this more actionable and into bite-sized steps? I'm willing to help out muchos since I still find documentation of logstash a bit lacking. More working out of the box examples would be better.

Having a collection of recordings from asciinema could be a good idea too: https://asciinema.org/

[coolacid]

Well, first step is to take what's here and re-classify and beautify the readmes. Add another classification of TODO where we have example logs that need either a config, example or tuturials to work with.

Some examples:

https://github.com/coolacid/GettingStartedWithELK/tree/master/Configs/Weblogs

• Would be a Tutorial, it has a set number of steps to go from basic to more advanced.

I just need to find time to organize the whole thing. I'm pretty busy until mid Aug but should be more focused then.

[coolacid]

I've started screwing with the whole thing.

Something that needs to be noted - some of the "configs" should be made into tutorials or examples, the configs should literally be JUST the filter commands wrapped in a type conditional.

[shurane]

Not sure I follow, do you have some before-and-after examples to show what you are describing?

[shurane]

By the way, you don't even have to copy the files into /tmp unless you specifically want to use the File input. You can just use exec { command => "cat weblogs.txt" interval => 60 } as in here

[shurane]

@coolacid so... I don't think there's enough of a distinction between the words for Configs, Examples, Snippets, and Tutorials. In fact, all those words mean so very similar things. I think we should be more explicit. I think Examples and Snippets are similar to each other. It's just that Snippets cover more advanced examples.

Some other things I think should be added:

1. Standardize on using either dashes always or underscores always. I prefer all lower case snake casing (as with C), but I really prefer consistency more than that.

2. Common errors and pitfalls, like accessing fields that do not exist or TCP connections that cannot be made.

3. We should have a default logstash.conf. Maybe something like this is a good base:

   input {
       generator {
           message => '127.0.0.1 - - [08/Aug/2009:00:01:01 -0000] "GET /elasticsearch/logstash/tree/82fdc11f2e35b52b05f773be52ba3c0e8072c509 HTTP/1.1" 200 1420 "https://github.com/elasticsearch/logstash/tree/82fdc11f2e35b52b05f773be52ba3c0e8072c509" "curl/7.37.1"'
           type => "generated"
           count => 1
       }
   }
   filter {
       grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
       date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] }
   }
   
   output {
       stdout { codec => rubydebug }
   }
   

   And then from there, have each example change as few things from the base logstash.conf as possible. This is to ensure that users will always have a usable example and won't require a leap of faith on understanding semantics on a particular thing on Logstash.

4. In a Makefile probably, we should provide an easy way to download multiple versions of Logstash to test the conf files with, at least the ones listed here: https://gist.github.com/shurane/92b122ebd0a1ed665397. This is in the interest of easy access.

5. Scrap together the aforementioned logs from the different sources, each into a digestable example form. It would be a good idea to get real world logstash configs that other people are using, whether in ##logstash or [email protected] or so on.

Also, Tut**u**rial is a typo, it should be Tut**o**rial.

I'm totally willing to do a lot of this work, just need to make some time to work through these.

[coolacid]

So, im on vacation. Let me revisit when I get back.