External data source

[sami-misa]

Hello all,

I am trying mupd8 application on a cluster and its working fine with the example given.

What I would like to do, is to have the data source of the application streamed from a different host. The stream of packets contains string type variables. Could someone provide me with example because I tried to edit the configuration file and specify a host that is sending stream of data but it didn't work.

Thanks

[sunweik]

Hi Sami,

I want to confirm your question before I try to find the answer. So same data source reader already works in node same as mupd8, and now you want to put this source started on a node other than node running mupd8? If this is what you want, you can start your data source on the node you like, say "node1", and replace "mupd8::sources[host]" field in config file with "node1".

Thanks, Wei

Wei Sun | 650.556.3886

On Mon, Jul 29, 2013 at 1:44 AM, sami-misa [email protected] wrote:

Hello all,

I am trying mupd8 application on a cluster and its working fine with the example given.

What I would like to do, is to have the data source of the application streamed from a different host. The stream of packets contains string type variables. Could someone provide me with example because I tried to edit the configuration file and specify a host that is sending stream of data but it didn't work.

Thanks

— Reply to this email directly or view it on GitHubhttps://github.com/walmartlabs/mupd8/issues/41 .

[sami-misa]

Hi

Thanks for the reply. My problem is that i am not able to change the source of the stream.In the example given in mupd8 , the stream of the application is generated locally from a specified file . But what I wantt to do is to change the source of the stream .I want to have a client application that sends a stream to mupd8 Application. What I have tried to do is to configure the client application to connect using to mupd8 application using sockets and changed the corresponding host and port entries in the application configuration file.Unfortunately it didn't work. Can you please tell me what I am doing wrong ?

[sunweik]

Here is an example of source config of a socket source. Please let me know if you have problem understanding it. Note: "com.walmartlabs.mupd8.JSONSource" only accepts stream each line of which is a JSON object. If you want to use something other than JSON as input you need to implement your own class extending Mupd8Source defined in ./src/main/java/com/walmartlabs/mupd8/application/Mupd8Source.java to replace JSONSource in config.

"sources" : [ { "name" : "source1", "source" : "com.walmartlabs.mupd8.JSONSource", "host" : "mtest0", "performer" : "T10Source", "parameters" : [ "mtest0:1234", "k1" ] } ]

-Wei

On Tue, Jul 30, 2013 at 1:39 AM, sami-misa [email protected] wrote:

Hi

Thanks for the reply. My problem is that i am not able to change the source of the stream.In the example given in mupd8 , the stream of the application is generated locally from a specified file . But what I wantt to do is to change the source of the stream .I want to have a client application that sends a stream to mupd8 Application. What I have tried to do is to configure the client application to connect using to mupd8 application using sockets and changed the corresponding host and port entries in the application configuration file.Unfortunately it didn't work. Can you please tell me what I am doing wrong ?

— Reply to this email directly or view it on GitHubhttps://github.com/walmartlabs/mupd8/issues/41#issuecomment-21776917 .

[sami-misa]

Hello again,

I have tried the configuration that you have provided unfortunately it doesn't work. Mupd8 is not giving any error message and says that it is connected to the source stream but actually their is not connection to the source stream as I am not seeing any connection established when I use netstat -vatn .Furthermore the application itself is not running and there is no resultant output.

[mimi-91]

Hi, I am a newbie to mupd8 and I want to configure mupd8 to get an external source stream from a client machine , I tried to configure the IP and port address as given , but that didn't work out for me.

I really need to do it and I can not afford getting the stream from a local file . The packets that my client wants to send contain JSON objects.

Can any one help ? thanks

[sunweik]

Following is a workable example "sources" array in config file,

{"sources" : [ { "name" : "source1", "source" : "com.walmartlabs.mupd8.JSONSource", "host" : "mtest0", "performer" : "T10Source", "parameters" : [ "mtest0:1234", "k1" ] } ]}

In order to make external socket source work, your data stream first needs to stream of text line and each line must be a json object. And in "parameters" array you can put "external host:port" as first parameter and "key path" to key as second parameter. Also "host" is the hostname or ip which is going to start this source reader.

If content of your external source is NOT json object at each line, then you need to implement "Mupd8Source" interface to support your own data format.

-Wei

On Wed, Aug 7, 2013 at 5:37 AM, mimi-91 [email protected] wrote:

Hi, I am a newbie to mupd8 and I want to configure mupd8 to get an external source stream from a client machine , I tried to configure the IP and port address as given , but that didn't work out for me.

I really need to do it and I can not afford getting the stream from a local file . The packets that my client wants to send contain JSON objects.

Can any one help ? thanks

— Reply to this email directly or view it on GitHubhttps://github.com/walmartlabs/mupd8/issues/41#issuecomment-22247825 .

[syang23]

Hi,

First of all, thanks for offering this wonderful tool/framework.

My question is: what if I want to extract two or more keys from JSON source stream, e.g. "request:requestUrl" and "response:responseHeaderList:Content-Type", do we support this? and how?

My 2nd question is: for the key, can key be "request:requestHeaderList:Cookie:com.wm.visitor", given the following JSON definition:

{ "request" : { "protocol" : "HTTP/1.1", "requestUrl" : "/beacon.gif", "remotePort" : 5949, "requestHeaderList" : [ { "valueList" : [ "SSID=BQA7PxsOAAAAAADIT3ZP13MCHMhPdk8BAAAAAAD03Z1PyE92TwAiAQADrScAAMhPdk8BAA; SSSC=2.G5725851697130140631.1|290.10157; com.wm.visitor=20543076174" ], "name" : "Cookie" }, .................

---

Thanks, Shenghu

[sunweik]

No, there is no existing way to use object inside array as key now. But you can implement your own soruce reader to do this. Every source reader needs to inherit "Mupd8Source" interface.

-Wei

On Wed, Aug 7, 2013 at 10:53 AM, syang0 [email protected] wrote:

Hi,

First of all, thanks for offering this wonderful tool/framework.

My question is: what if I want to extract two or more keys from JSON source stream, e.g. "request:requestUrl" and "response:responseHeaderList:Content-Type", do we support this? and how? My 2nd question is: for the key, can key be "request:requestHeaderList:Cookie:com.wm.visitor", given the following JSON definition:

{ "request" : { "protocol" : "HTTP/1.1", "requestUrl" : "/beacon.gif", "remotePort" : 5949, "requestHeaderList" : [ { "valueList" : [ "SSID=BQA7PxsOAAAAAADIT3ZP13MCHMhPdk8BAAAAAAD03Z1PyE92TwAiAQADrScAAMhPdk8BAA; SSSC=2.G5725851697130140631.1|290.10157; com.wm.visitor=20543076174" ], "name" : "Cookie" },

.................

Thanks, Shenghu

— Reply to this email directly or view it on GitHubhttps://github.com/walmartlabs/mupd8/issues/41#issuecomment-22270171 .

[syang23]

Thanks Wei.

How about my other question: what if I want to extract two or more keys from JSON source stream, e.g. "request:requestUrl" and "response:responseHeaderList:Content-Type", do we support this? and how?

Regards, Shenghu

[sunweik]

oops, I missed this question.

Regarding this question, the answer is same. :-) Existing source reader doesn't support this, but you can add this support into you created source reader.

-Wei

On Wed, Aug 7, 2013 at 2:40 PM, syang0 [email protected] wrote:

Thanks Wei.

How about my other question: what if I want to extract two or more keys from JSON source stream, e.g. "request:requestUrl" and "response:responseHeaderList:Content-Type", do we support this? and how?

Regards, Shenghu

— Reply to this email directly or view it on GitHubhttps://github.com/walmartlabs/mupd8/issues/41#issuecomment-22286438 .

[wlam]

Hi--just to catch up, I'd add one more remark: Besides trying to overload everything into the source reader, it may be simpler and more powerful to write a simple Mapper to perform the extra steps you desire.

For example, a new Mapper can take its input event, parse out the desired (two in your case above) keys from the JSON, and emit a separate event for each key. As a bonus side effect, the Mapper can be written to emit the events to different streams, so that different recipient Mappers and Updaters can subscribe to each kind of event as desired.