How we can help make Erlbrake better?

[benarent]

Hi Guys!

Firstly, thanks for building an excellant Erlang notifier for Airbrake. If you're ever in San Francisco; we owe all contributors a beer / coffee or two.

I've been talking to few customers and many are asking for 'First Class' Erlang notifier. This may mean improving the notifier, updating the API or changes to the Airbrake UI.

I would like to raise this 'issue' as an open discussion and as a place in which we can discuss what can be done to improve ErlBrake and Airbrake.

Ben - Product Guy at http://airbrake.io

[kenpratt]

Hey Ben!

Thanks for getting in touch. I'm open to any arrangement -- if you'd like to take over development of erlbrake in-house, that's fine with me, or else if you'd like to join in on this project, that's fine too. Or if you have a list of changes you'd like to see, you can enter as issues and we can address them as time permits.

Have those customers mentioned any specific improvements they'd like to see to make erlbrake first class?

Thanks,

-Ken

[ddossot]

Cool to read there's interest in erlbrake in particular and Erlang in general :)

One thing I was wondering is if we could find a way to handle deploys through erlbrake. But as Ken said, it's probably better to work on actual users' needs instead of making them up.

[benarent]

Hi Guys, I sent you a mail. I'll leave this ticket open as another Airbrake user may reply.

[benarent]

@ddossot and @kenpratt . This is some of the feedback from the Erlang users. I've just dumped most of the e-mail in. Please let me know if you want more information of I can sync him up.

The main problem we have is that the notifier doesn't do a great job of extracting error information from the Erlang error report and so we end up with a lot of different errors that Airbrake considers the same. Because of this problem we have not been notified when problems occurred because Airbrake thought the errors were duplicates.

For example, here is the title of one of the errors generated by our Erlang application:

"Worker <0.19839.1> crashed: {{try_clause,\n {error,\n {'EXIT',\n {{{badmatch,\n {error,\n {{badmatch,{error,enotconn}},\n [{ssh_connection_handler,init_ssh,5,\n [{file,\"ssh_connection_handler.erl\"}, \n {line,637}]},\n {ssh_connection_handler,init,1,\n ["

This isn't very useful at a glance and is highly likely to be misclassified as a duplicate of an unrelated error. This is a side-effect of tying erlbrake into Erlang's error_logger process, so I am not sure if there is really a reasonable solution to the problem. We can get better looking error notifications when we create them ourselves and call the notifier directly, but the benefit to using Airbrake is that you get notified when things you weren't expecting to go wrong do so.

[ddossot]

This is a side-effect of tying erlbrake into Erlang's error_logger process

This is a correct assessment.

When a crash occurs, Erlang calls this function in erlbrake_error_logger:

handle_event({error, _Gleader, {_Pid, Format, Data}}, State)

I was expecting that Data would contain a rich structure (leaving it to Format to build the error report) that we could mine for context on the crash but it turns out it contains just a string :(

Also we report all errors caught through the error_logger as initiating from erlbrake_error_logger as the source file.

Thus, so far, the only way I can think of to improve things would be to parse these string-based error reports (like the one above) in order to mine them for root cause, source file, line number and the actual stack trace. That's ugly so hopefully someone else has a better idea, maybe one of Airbrake's Erlang users?

Maybe a first step would be to try to locate file and line from the error report and use that when sending to Airbrake? That still won't give nifty error reports but @benarent would that be enough to help de-duplicating errors?

[ypaq]

Lager uses error_logger handlers in a similar manner as erlbrake does (unfortunately Lager actually removes all error handlers from error_logger on startup which could result in incompatibilities. This should be kept in mind since Lager is quite popular).

@ddossot's proposal to parse the data part seems legit, Lager does the same in the module error_logger_lager_h.

It might make sense to write a Lager backend for Airbrake.

[ddossot]

Interestingly pull request #10 comes quite timely :)

[betawaffle]

My pleasure!

Lager (and by strange coincidence, my PR) has figured out how to extract the valuable format juice from the Data parameter for standard erlang-generated error messages.

Also, there's the crash reports and supervisor reports, which are very standardized and reliable (although, not well documented).

They also are very good at extracting the stack trace from most Reasons, and reformat the error message for consumption by a human.

[benarent]

@betawaffle Great stuff, please let me know if anything else is needed on the API end to get Pull Request into the master of this project.

[betawaffle]

@benarent Well, there are probably plenty of nice to have things, but none required for #10. It stays within the confines of the current API.

I'd have to think more about it, but I could come up with some feature requests if you guys are interested in more explicitly supporting Erlang projects.

Off the top of my head:

• Having remaining rate limit headers, similar to what Twitter has for the REST API, so that I can implement a proper rate limiter on the client side.
• Having a place for miscellaneous context data, so that I don't have to build a fake request. cgi-vars works fine but it's kinda awkward, and shows up quoted in the UI.

[ypaq]

It would also make sense to include the Erlang node in the environment.

[betawaffle]

@ypaq I use that as the URL right now.

[ypaq]

@betawaffle You mean URL under request?

[betawaffle]

@ypaq Yep. It's a temporary solution until Airbrake has better support.

[ypaq]

BTW: started a lager backend but am not happy with the preformatted data from lager. There are efforts to add formatters into lager but are not merged into the master yet (see: https://github.com/basho/lager/pull/36).

https://github.com/ypaq/lager_erlbrake_backend

[kenpratt]

Pull request #10 has been merged into trunk. See tag 0.2 if you need the pre-merge code. Thanks Andrew!

[ddossot]

YTMND!

[ypaq]

Hey Guys!

I've sent a pull request for some improvements to erlbrake. Furthermore, I've finished the lager_erlbrake_backend (see https://github.com/ypaq/lager_erlbrake_backend).

The benefits to people that are using erlbrake already are minor, though. The benefits come for lager users that don't use Airbrake yet, they can hook in the lager_erlbrake_backend and are good to go.

Implementing this I realized it is quite a stretch to match the Airbrake XSD for Erlang applications. Also the rate limitations on the API are not helpful when the Erlang VM produces lots of errors in case of a crash. The noise factor is pretty high when crash reports go to Airbrake. I'd recommend to send only clearly defined emergency notifications to Airbrake.

Furthermore, the lager version required for the lager_erlbrake_backend is not official yet. It will come with the next lager release.

So far I've only tested this. No production experience yet.

Your feedback is more than welcome.

Cheers

• Tilman

[benarent]

@ypaq Thanks for the pull request. I'll leave it with @ddossot and @kenpratt.

We've recently been doing a lot of work to both the API and displaying errors, and I would love to make Erlang Rock on Airbrake. Our new API endpoint accepts JSON, and we've also considered 'twitter-like' rate limiting response in the API.

@ypaq hit me up on [email protected] and I'll setup a test environment for you. The API XSD Gate-keeper is a tough man.. but can be bribed.

[ypaq]

@benarent Thanks! If your JSON endpoint is more flexible in terms of fields display and relevance for notice classification I don't need an XSD. I find it too clunky for this purpose. I'd be much rather interested in how the JSON API looks like.

[benarent]

@betawaffle We've added a rate limit to our new JSON API.

X-Ratelimit-Limit:1000
X-Ratelimit-Remaining:996

and this is the scheme.

http://collect.airbrake.io/api/v3/projects/[PROJECT_ID]/notices?key=[API_KEY]

POST http://collect.airbrake.io/api/v3/projects/PROJECT_ID/notices?key=API_KEY
Content-Type: application/json

{
  "notifier": {
    "name":    "notifier name",
    "version": "notifier version",
    "url":     "notifier url"
  },
  "errors": [
    {
      "type":      "error1",
      "message":   "message1",
      "backtrace": [
        {
          "file": "backtrace file",
          "line": 10,
          "function": "backtrace function"
        }
      ]
    },
    {
      "type":      "error2",
      "message":   "message2",
      "backtrace": [
        {
          "file": "backtrace file",
          "line": 10,
          "function": "backtrace function"
        }
      ]
    }
  ],
  "context": { // Optional.
    "os": "Linux 3.5.0-21-generic #32-Ubuntu SMP Tue Dec 11 18:51:59 UTC 2012 x86_64",
    "language": "Ruby 1.9.1",
    "environment": "production",

    "version": "1.1.1",
    "url:": "http://some-site.com/example",
    "rootDirectory": "/home/app-root-directory",

    "userId": "12345",
    "userName": "root",
    "userEmail": "[email protected]",
  },
  "environment": {}, // Optional.
  "session": {}, // Optional.
  "params": {} // Optional.
}

If you would like early access, please hit me up [email protected]

[betawaffle]

Sweet! I'll let you know before I have some time to make a prototype for this. It may be a while :(

[benarent]

@betawaffle No worries, I just wanted to put it here in case anyone wants to do more dev on it.