Stream data into a nanocube in real time

[Nut]

Hello,

for the last two weeks, I've been dealing with nanocubes. Everything was going very well so far, but I failed to load data into a nanocube in real time.

Here is a short explanation of what I tried to do:

A message broker waits for messages asynchronously. If a message arrives, it should be converted into a binary format and loaded into a nanocube.

I tried to do this with pipes as described on the GitHub page (https://github.com/laurolins/nanocube/wiki#load-data-from-comma-separated-value-csv-files). For this purpose, I have a python script which waits for messages and if a message arrives the script will output a line in csv style on “stdout”:

def callback(ch, method, properties, body):
     sys.stdout.write(body.decode('utf-8'))

channel.basic_consume(callback,
                      queue='test_queue',
                      no_ack=True)

The output of the script then looks something like this (one line per data row):

9418031,HW561348,12/06/2013 06:25:00 PM,040XX W WILCOX ST,2024,NARCOTICS,POSS: HEROIN(WHITE),SIDEWALK,true,false,1115,011,28,26,18,1149444,1899069,2013,12/11/2013 12:40:36 AM,41.8789661034259,-87.72673345412568,"(41.8789661034259, -87.72673345412568)"

I start my script, the binning script and the nanocube server like this:

python3 receive.py | nanocube-binning-csv --header='ID,Case Number,time,Block,IUCR,crime,Description,Location Description,Arrest,Domestic,Beat,District,Ward,Community Area,FBI Code,X Coordinate,Y Coordinate,Year,Updated On,Latitude,Longitude,Location' --sep=',' --latcol='Latitude' --loncol='Longitude' --timecol='time' --catcol='crime' - | nanocube-leaf -q 29512

Unfortunately, nothing happens. My script will print the messages correctly, but the nanocube-binning-csv script won’t be executed. So, my question is whether someone has an idea how to convert data in real time and load it into a nanocube? Many thanks in advance.

Kind regards, Nut

[salivian]

Can you please check the output of your script and nanocube-binning-csv ?

We have done something similar to what you are doing and it should work.