Elastic / Kibana Support

[lealog]

Can you please add the possibility to upload the reports to elastic/kibana (https://www.elastic.co/)? This way, the tool will have option for splunk and also for elastic/kibana.

[wifinigel]

Hi lealog,

Do you have any information about how to get data in to elastic/kibana perhaps via an API? I don't know these products unfortunately.

[lealog]

Hi Nigel,

I will ask some help. Later I will provide you the right information ;)

[wifinigel]

I am looking at how I can add support for elastisearch, as I would like to support multiple data destinations. Would you be able to test if I get the integration module written?

[lealog]

Hi Nigel,

Yes. I can test :) I did some search and ask information to some friends. Using python it's easy to to this. You can use this URL - https://www.logicalfeed.com/posts/1183/upload-bulk-csv-data-to-elasticsearch-using-python as reference.

[wifinigel]

OK, will take a look at the article and put this in the queue to implement.

[lealog]

Thanks Nigel :)

[wifinigel]

Still under consideration, but given the increased support effort incurred adding Influx/Grafana, unlikely in short term TBH.

[wifinigel]

Note for future:

https://stackoverflow.com/questions/43981275/index-json-files-in-elasticsearch-using-python

For this task you should be using elasticsearch-py (pip install elasticsearch):

from elasticsearch import Elasticsearch, helpers
import sys, json

es = Elasticsearch()

def load_json(directory):
    " Use a generator, no need to load all in memory"
    for filename in os.listdir(directory):
        if filename.endswith('.json'):
            with open(filename,'r') as open_file:
                yield json.load(open_file)

helpers.bulk(es, load_json(sys.argv[1]), index='my-index', doc_type='my-type')

https://www.logicalfeed.com/posts/1182/upload-bulk-json-data-to-elasticsearch-using-python

import sys
import json
from pprint import pprint
from elasticsearch import Elasticsearch
es = Elasticsearch(
    ['localhost'],
    port=9200

)

MyFile= open("C:\ElasticSearch\shakespeare_6.0.json",'r').read()
ClearData = MyFile.splitlines(True)
i=0
json_str=""
docs ={}
for line in ClearData:
    line = ''.join(line.split())
    if line != "},":
        json_str = json_str+line
    else:
        docs[i]=json_str+"}"
        json_str=""
        print(docs[i])
        es.index(index='shakespeare', doc_type='Blog', id=i, body=docs[i])
        i=i+1

[wifinigel]

@lealog I'm not sure how easy this is, but now we have local dumping of data in json files, this bulk upload may be possible more easily. Do you have a test server to try this out on if I could generate some test code?

[lealog]

Hi Nigel,

Yes, I have a server to do some tests :) When do you want to start doing some tests?

Nigel Bowden [email protected] escreveu no dia quinta, 28/01/2021 à(s) 07:49:

@lealog https://github.com/lealog I'm not sure how easy this is, but now we have local dumping of data in json files, this bulk upload may be possible more easily. Do you have a test server to try this out on if I could generate some test code?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/wifinigel/wiperf/issues/8#issuecomment-768867483, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ6363XXG3LSFNXGC5KRVITS4EJHLANCNFSM4K2XZR2A .

-- Cumprimentos, Óscar Leal

[wifinigel]

I'll create a test script for you to run from the CLI of a probe and see if it gives you any output.

Will email something across in the next day or two,

Not sure how easy/hard this will be.