nathanmarz
Add `nimbus.classpath` and `worker.classpath` config options.
This addresses Issue #260 with a slight addition of adding an option for adding to the nimbus classpath as well. This is useful for us as we have our own custom schedule which also has some HBase dependencies, so I found it pretty easy to add both at the same time.
I technically created this change off of the 0.8.3 branch, because at the time, I couldn't get master to build via lein. The changes are small enough that the merge should apply cleanly regardless, but if you want me to rebase, let me know.
There's no point to adding a feature for adding to the nimbus classpath, as those jars can just be put in the lib/ directory. So let's remove that from this pull request.
Along with those changes, can you send me a contributor agreement? This is necessary to provide a legal basis for merging in external contributions. Instructions here, should only take a couple minutes: https://github.com/nathanmarz/storm/wiki/Contributing-to-Storm
@nathanmarz So in our case, there are a lot of redundant JARs that need to get copied into the storm lib/ directory. To give a little context, we have a scheduler that specifically tries to enforce locality of Bolts and Spouts based on the HBase region mapping, thus we need an HBase client in the scheduler. This thus requires all of HBase and its dependencies to be copied to lib/, which is quite annoying. Plus, it is specifically, the HBase config files that I want to avoid copying because they are very reliant on the actual proper placement in their classpath.
In a scenario where the classpath consists of paths on shared storage, it is desirable to use a configuration stored locally to avoid conflicts (ui.port, etc.). Passing the --config option to storm will specify a custom config, but that config also must live somewhere on the classpath (Utils#findResources).
It would be nice to be able to specify a classpath for the different daemons. Or is there a better way this is typically done?
@nathanmarz I just got my work to sign off on the contributor agreement, you should have received a copy today from SEOmoz. I'll send a personal one as well this evening.
Did you get the contributor agreements already? Also, how do you feel about leaving the nimbus.classpath after my explanation for why?
Just checking in. I haven't seen any progress, and I was hoping for an update. I think I've covered the contributor agreement, but let me know if you haven't received it. Also, I hope my explanation for why the nimbus.classpath config option is desirable makes sense as well.
+1 ; we would like a clean way to add jars into a configurable classpath. We deploy storm via a git repository, and having working copy changes within [storm-directory]/lib can potentially disrupt deployments.
I vote to support adding to the daemon classpaths as well. Putting stuff in the storm lib/ directory makes it difficult to cleanly upgrade both storm and our own project (that's introducing those dependencies anyway) in the future. I'd really like the ability to point the appropriate daemons at a separate lib/ directory for our product to be included along with storm's jars.
