Bucky clients, memcache, mysql, bugfixes

[kmcminn]

We've taken a liking to bucky at EA. I've made several changes to it, mainly refactoring the clients to be threaded and adding a mysql and memcache client. I've tweaked optparse a bit so that config options work as expected and fixed a few bugs we encountered in testing.

[davisp]

@kmcminn

Just going through this PR and while I think the general idea is sound I'm not a fan of the current implementation. What I think I'd like to see is that that the alternative clients for memached and mysql get pulled in dynamically. What this means is that we should factor out a trivial base Client class that each client will extend. Then the config file can pull in whatever it wants as a client and add it to a list. The basics of a queue per client type with multiple threads per client should be part of the core package. We'll just add some extra stuff to the config so that custom clients can be used.

Though we'll also have to add a few other things. Things like merging the config file and handing that to each client class's constuctor and the like. It should be relatively easy but there might be a few gotchya's in there for specifying defaults for non-core classes and the like.

There's also a few technical bits that I'll want changed. The init file and the config examples should be in individual commits. We might also pull out the README changes into a final commit as well. I've got other comments that are more code review so I'll respond with line comments for each of those.

Thanks for the work, I think this is definitely a good feature to have. And apologies again for being so daft with my configuration this repo. I should've gotten to this a long time ago.

[kmcminn]

pretty much agree with most your notes in the code review.

I'll finish another client for bucky (opentsdb) this week and will update my fork and work through the cr and go from there.

[davisp]

Sounds good. I'm a bit busy with handling some client tests at work this week but hopefully I can fit some time in to take a crack at what's kicking around in my head. I have the pkg_resources stuff for collectd converters. I'm not entirely certain if I like that because it requires an installed package to work. I think if we abuse the config file we can pull in plugins or even define them right there in the config for use which seems a lot more user friendly.

I'm still a little bit unsure of the configuration for these bits but I'm sure its nothing we can't handle. I have some stuff in Gunicorn that might be applicable but it seems a bit heavy for this.

Make sure and ping me if you have push anything new.

[kmcminn]

i'm reworking this PR using the multiprocessing branch. might make sense to do a fresh PR once its merged into master.

[davisp]

@kmcminn sounds like a plan. Ping me when you want me to review anything.

[trbs]

@kmcminn are you still using Bucky and interested to get your improvements into the current mainline ?

[kmcminn]

I'm not using it currently but would like to see the work not lost.

[kmcminn]

I'd like to propose doing the the proxy sources (memcache, opentsdb, mysql, etc) one at time to shrink the pull down. my work needs refactoring before being ready for the mainline obviously.