Analytics Dashboard

[jordanschalm]

Todo: [x] Update main README with info about analytics [x] Update sample config with dashboard config [x] Add mitigation for HELO spamming

[jordanschalm]

Up to date with more-backends-refactoring branch and merge conflicts resolved

[flashmob]

now working on this first, will resolve the conflicts

[flashmob]

@jordanschalm

Made the following changes:

• Added a Stop() function, this required adding channels for stopping and using a non-blocking version of ListenAndServe that returned a Closer (see http.go). Could you validate that all that needs to be stopped is being stopped?

Also, started writing tests. It would be good if we could add more tests for this.

Now trying to figure out this problem, when pointing the browser to http:

INFO[0020] Logging analytics data                        clients=0 ram=3010784
2017/03/23 00:32:13 http: panic serving 10.0.2.2:34084: runtime error: invalid memory address or nil pointer dereference
goroutine 7 [running]:
net/http.(*conn).serve.func1(0xc4200aa460)
	/usr/local/go/src/net/http/server.go:1721 +0xd0
panic(0x873f60, 0xd0cc80)
	/usr/local/go/src/runtime/panic.go:489 +0x2cf
net/http.serveFile(0xd21ce0, 0xc42021c1c0, 0xc42009e900, 0x0, 0x0, 0xc42023e0d4, 0x1, 0xc42023e101)
	/usr/local/go/src/net/http/fs.go:525 +0x8b
net/http.(*fileHandler).ServeHTTP(0xc420337820, 0xd21ce0, 0xc42021c1c0, 0xc42009e900)
	/usr/local/go/src/net/http/fs.go:689 +0xb4
github.com/flashmob/go-guerrilla/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc420339680, 0xd21ce0, 0xc42021c1c0, 0xc42009e900)
	/home/flashmob/projects/golang/src/github.com/flashmob/go-guerrilla/vendor/github.com/gorilla/mux/mux.go:114 +0x10c
net/http.serverHandler.ServeHTTP(0xc420334790, 0xd21ce0, 0xc42021c1c0, 0xc42009e600)
	/usr/local/go/src/net/http/server.go:2568 +0x92
net/http.(*conn).serve(0xc4200aa460, 0xd224e0, 0xc42006a200)
	/usr/local/go/src/net/http/server.go:1825 +0x612
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2668 +0x2ce

it was doing this before the changes..

My dashboard config:

"dashboard": {
    "is_enabled": true,
    "listen_interface": ":8081",
    "tick_interval": "5s",
    "max_window": "24h",
    "ranking_aggregation_interval": "6h"
  }

The 8081 port is forwarded via ssh

[jordanschalm]

That might be because the statik package doesn't contain the correct JS/HTML files? I haven't encountered that before. Have you run the build.sh file in the root?

[flashmob]

thanks; didn't do that - will try that soon!

[flashmob]

It seems like the statik package is missing from the repo. Could you pls commit when you get a chance? Thanks.

[jordanschalm]

@flashmob So I deliberately didn't include the statik package because it's a build artifact, and it's pretty big (~7MB). If it's committed each time a change is made to the dashboard, that'll increase the repo size pretty quickly, but it's an option.

In order to use the dashboard you need to:

• install dependencies npm install
• build front-end into HTTP-serveable bundle file npm run build
• make statik package

This is a somewhat long process so it certainly shouldn't be part of building guerrillad, since that is very fast right now. However, it does have to happen at least once after the package is installed, assuming the statik package is not committed.

Another option would be to add a one-time build step to the Makefile called dashboard (actually already there, but doesn't quite work due to me not knowing Make very well, that's why I added the build.sh script instead) that builds everything required for the dashboard. This has the benefit of not needing to commit build artifacts, but adds a step to the installation process.

Installation process would then be like:

git clone <url>
make dependencies
make dashboard
make guerrillad

[ghost]

Is this ready to be committed to Master ? Dashboard is useful for me.

[flashmob]

This branch (dashboard) has been updated with the latest master.

If you require the dashboard, you may git checkout this branch and build according to this doc https://github.com/flashmob/go-guerrilla/wiki/Dashboard

[flashmob]

Todo in the future:

• Break up this branch by putting the front-end in a separate repo (and parts of the backend that services the front-end). Throw away the logrus hook, as we won't need it anymore.
• Merge the other half of this branch to master. This means merging all the event logging code to master.
• In the separate repo, implement scanning of the log file and processing the events, incorporating the existing code.
• Add more types of events to the log (garbage collector, amount of data transferred, etc..) to be enabled via config