Storm example doesn't run using newer library versions

[kahunamoore]

Updating the dependencies in the project.clj to include the latest versions of all the libraries used by this example results in runtime errors.

I have worked through several errors and fixed them but have now run into an error I don't know what the right way to fix it is. So I'm creating this issue to ask questions and discuss how to update this example.

The current error I am working on is:

Exception in thread "main" java.lang.IllegalStateException: var: #'clara.rules.compiler/create-get-alphas-fn is not public, compiling:(clara/rules/storm.clj:55:23)

It looks like the refactoring of the engine namespace into compiler etc. has resulted in this function being private to the namespace. Obviously I could modify the clara sources to make it public but I suspect there is a recommended alternative. There must have been a reason that function was hidden.

Thoughts?

[rbrush]

I should have a chance to take a look at it later today. It's probably just a casualty of some refactoring, since I hadn't been maintaining the Storm support for a while. I'll post back here when I track it down.

[kahunamoore]

I really don't want to make more work for you - I should be able to figure it out on my own. I was just looking for a tip as to where I should look for a suitable replacement.

I'm curious about Clojure's private function's. How does one handle a situation like this where a function is private to a namspace but you want a separate library (e.g. storm support) to have access to said fn... In C++ we have the friend designator. Is there á way to accomplish a similar thing in clojure?

[rbrush]

Yes, there isn't a "friend" equivalent in Clojure. But I don't mind just making this function public. It's already in a namespace that indicates it's not for general consumption, so making it accessible for special cases like Storm is fine.

On Wed, Jul 30, 2014 at 11:10 AM, Alan Moore [email protected] wrote:

I really don't want to make more work for you - I should be able to figure it out on my own. I was just looking for a tip as to where I should look for a suitable replacement.

I'm curious about Clojure's private function's. How does one handle a situation like this where a function is private to a namspace but you want a separate library (e.g. storm support) to have access to said fn... In C++ we have the friend designator. Is there á way to accomplish a similar thing in clojure?

— Reply to this email directly or view it on GitHub https://github.com/rbrush/clara-storm/issues/1#issuecomment-50637374.

[kahunamoore]

I got past that problem by making that function public. The other types of changes I've made so far are:

1. Namepace changes - primarily load-rules moved from engine to compiler
2. Changing code to account for new listener argument to various functions in clara

There are some warnings that I ignored but now I think I need to do something about them because I'm getting this error at runtime:

3611 [Thread-6] ERROR backtype.storm.event - Error when processing event java.lang.IllegalStateException: Attempting to call unbound fn: #'backtype.storm.util/some?

The previous warnings were:

WARNING: some? already refers to: #'clojure.core/some? in namespace: backtype.storm.util, being replaced by: #'backtype.storm.util/some? WARNING: some? already refers to: #'clojure.core/some? in namespace: backtype.storm.clojure, being replaced by: #'backtype.storm.util/some? WARNING: some? already refers to: #'clojure.core/some? in namespace: backtype.storm.thrift, being replaced by: #'backtype.storm.util/some? WARNING: some? already refers to: #'clojure.core/some? in namespace: backtype.storm.config, being replaced by: #'backtype.storm.util/some? WARNING: some? already refers to: #'clojure.core/some? in namespace: backtype.storm.messaging.loader, being replaced by: #'backtype.storm.util/some?

I left some out because the list was long and were basically the same issue.

Unfortunately, I think this problem is with the storm library itself. I don't know if they have updated to clojure 0.6.0 but there was an added some? API in the latest clojure release that appears to conflict with the backtype version... TBD.

Here is the discussion on the clojure mailing list:

https://groups.google.com/d/msg/clojure/tseJgAi3HC4/vAozkkE2zOAJ

[rbrush]

Upgrading storm might be worth a try. It looks like 0.9.2-incubating made the jump to Clojure 1.6, and addressed the issue in the below JIRA:

https://issues.apache.org/jira/browse/STORM-265

On Thu, Jul 31, 2014 at 1:04 AM, Alan Moore [email protected] wrote:

I got past that problem by making that function public. The other types of changes I've made so far are:

1. Namepace changes - primarily load-rules moved from engine to compiler
2. Changing code to account for new listener argument to various functions in clara

There are some warnings that I ignored but now I think I need to do something about them because I'm getting this error at runtime:

3611 [Thread-6] ERROR backtype.storm.event - Error when processing event java.lang.IllegalStateException: Attempting to call unbound fn: #'backtype.storm.util/some?

The previous warnings were:

WARNING: some? already refers to: #'clojure.core/some? in namespace: backtype.storm.util, being replaced by: #'backtype.storm.util/some? WARNING: some? already refers to: #'clojure.core/some? in namespace: backtype.storm.clojure, being replaced by: #'backtype.storm.util/some? WARNING: some? already refers to: #'clojure.core/some? in namespace: backtype.storm.thrift, being replaced by: #'backtype.storm.util/some? WARNING: some? already refers to: #'clojure.core/some? in namespace: backtype.storm.config, being replaced by: #'backtype.storm.util/some? WARNING: some? already refers to: #'clojure.core/some? in namespace: backtype.storm.messaging.loader, being replaced by: #'backtype.storm.util/some?

I left some out because the list was long and were basically the same issue.

Unfortunately, I think this problem is with the storm library itself. I don't know if they have updated to clojure 0.6.0 but there was an added some? API in the latest clojure release that appears to conflict with the backtype version... TBD.

Here is the discussion on the clojure mailing list:

https://groups.google.com/d/msg/clojure/tseJgAi3HC4/vAozkkE2zOAJ

— Reply to this email directly or view it on GitHub https://github.com/rbrush/clara-storm/issues/1#issuecomment-50716323.

[kahunamoore]

Looks like that got storm only as far as 1.5.1. I guess we'll have to wait until it gets to 1.6...

[vishaltelangre]

I am able to reproduce the same issue with storm version 0.9.2-incubating and 0.9.3.

[kirankulkarni]

I am able to reproduce the same issue with storm version 0.9.5

[rbrush]

Hi Kiran,

While we're using Clara in production and continuing to invest in it, I don't expect to be spending time on the clara-storm port. It was an experiment to demonstrate ideas but hasn't been made production worthy, and other needs have to take precedence. If there is someone who's interested in working on clara-storm I'd be happy to talk through it.

I don't want this to reflect on clara-rules or other use cases, which I think has a bright future. It's just that our storm use case didn't pan out as expected.

-Ryan

On Fri, Jun 5, 2015 at 5:06 AM, Kiran [email protected] wrote:

I am able to reproduce the same issue with storm version 0.9.5

— Reply to this email directly or view it on GitHub https://github.com/rbrush/clara-storm/issues/1#issuecomment-109247526.

[kahunamoore]

I'm still interested in this but am currently focused on porting Clara to use reader conditionals.

Once that is done I'd be happy to help out on this.

[rbrush]

Just a question to this group...would support for running Clara in Spark Streaming be of interest to anyone? The major challenge to making clara-storm production worthy is going to be persisting the state of the working memory on each node to recover from failures. With Spark I think we could store the entire working memory as a Spark RDD (taking a data-centric approach as opposed to a process-centric approach seen in Storm), and persist or cache the RDD to recover from failures.

This would be a significant project in itself, but I'm actually working on a system now that uses Spark and could benefit from this...so it's something I might work on.

On Fri, Jun 5, 2015 at 9:39 AM, Alan Moore [email protected] wrote:

I'm still interested in this but am currently focused on porting Clara to use reader conditionals.

Once that is done I'd be happy to help out on this.

— Reply to this email directly or view it on GitHub https://github.com/rbrush/clara-storm/issues/1#issuecomment-109313232.

[kahunamoore]

I don't know much about either Storm or Spark but I am interested in being able to scale/distribute Clara-based logic across nodes. My primary use case is for non-batch state/data processing. The system will need to a) scale to a large number of rules/data, b) increase availability and reliability.

However these things are implemented is fine with me. I need a system that continually evaluates rules that may depend on time, data or events of various kinds and then take actions as needed. It also needs to serve/receive data/state to/from external systems of various types such as web apps, mobile apps, twitter, github, linked in, etc.

Let me know how I can help. As you know my primary focus right now is getting rid of cross overs but after that I'm free to help with this.

[Snillocydoc]

I have a very similar need to @kahunamoore and would also be looking to contribute.

[rbrush]

The big challenge is going to be mapping a Clara rules session onto a set of Spark RDDs. I'm pretty sure this is possible but need to make time to look at the specifics. (I've been busy pushing through an unrelated project for the past couple weeks.) I'll try to dig into this in the next week or so, but if anyone else wants to take a closer look I think a lot of the effort will be two things:

1. Taking clara.rules.memory.IReaderMemory and clara.rules.memory.ITransientMemory and having an implementation of those to store the state in Spark RDDs
2. Probably less clear, but we need to express the flow of rules and joins in clara.rules.engine.ITransport as transformations on those RDDs.

I'm getting close to getting my unrelated project to a state where I don't need to be so focused on this so I hope to get back to more Clara work soon.

On Tue, Jul 28, 2015 at 2:50 PM, Snillocydoc [email protected] wrote:

I have a very similar need to @kahunamoore https://github.com/kahunamoore and would also be looking to contribute.

— Reply to this email directly or view it on GitHub https://github.com/rbrush/clara-storm/issues/1#issuecomment-125734371.