Complete integration with Kibana

[sanikolaev]

Manticore can work with Kibana like this:

# download manticore beta version with support for Kibana, check https://repo.manticoresearch.com/repository/kibana_beta/ for different OS versions  
wget https://repo.manticoresearch.com/repository/kibana_beta/ubuntu/jammy.zip  
  
# unarchive it  
unzip jammy.zip  
  
# install the packages  
dpkg -i build/*  
  
# switch Manticore to the mode supporting Kibana  
mysql -P9306 -h0 -e "set global log_management = 0; set global log_management = 1;"  
  
# start Kibana pointing it to Manticore Search instance listening on port 9308  
docker run -d --name kibana --rm -e ELASTICSEARCH_HOSTS=http://127.0.0.1:9308 -p 5601:5601 --network=host docker.elastic.co/kibana/kibana:7.4.2  
  
# install php and composer, download loading script and put into Manticore 1 million docs of fake users  
apt install php composer php8.1-mysql  
wget https://gist.githubusercontent.com/sanikolaev/13bf61bbe6c39350bded7c577216435f/raw/8d8029c0d99998c901973fd9ac66a6fb920deda7/load_manticore_sharded.php  
composer require fakerphp/faker  
php load_manticore_sharded.php 10000 16 1000000 16 1  
  
# don't forget to create an index patter in Kibana (user*)  
  
# run `docker stop kibana` to stop the Kibana server  

If all went well you should see:

But it has some limitations. This task is to complete this integration.

[unixfox]

What are the current limitations?

[tomatolog]

user can not manage indexes \ cluster from Kibana UI as searchd provide only dummy replies.

Search with nested aggregations does not work.

There could be some more requests that daemon failed to parse or returns wrong reply

[nickchomey]

I just discovered manticore and am very curious about it. But I need Kibana. Has any progress been made on this integration/any estimate of when it might be done?

[sanikolaev]

Has any progress been made on this integration/any estimate of when it might be done?

Yes, we've made great progress on this integration, but unfortunately it's still in an active development stage. In particular we've been working actively on adapting Elasticsearch aggregation capabilities to Manticore which is essential for the integration with Kibana. Unfortunately, we won't make it to the upcoming release we are doing soon, but hopefully we'll have something to check out in a month or so.

[unixfox]

thank you for the feedback @sanikolaev. against which version of kibana are you working on? last time I tried manticore search on a recent kibana version 7.17.10 and 8.8.2 it was complaining about manticore search not being a compatible elasticsearch product and thus failing to launch completely.

[sanikolaev]

We are testing against 7.6.0.

[nickchomey]

That's great that you're working on es aggregations!

May I suggest using the latest version of kibana for testing? 7.6 is both 3.5 years old and not even the latest major version (8.x).

The changes to the elastic license probably dont really affect most manticore users - my understanding is it just stops you from starting something that competes with elastic cloud (like aws elasticsearch service did).

[sc0rp10]

Hello! Is there any progress with integration with Kibana? We're looking for a replacement for ES as a log storage, but any other dashboards otheer than Kibana are pretty unusable.

[sanikolaev]

There's progress, but the work is not publicly available yet due to dependencies on some other things that are being implemented. We hope to make it all available for testing soon.

[b17k0]

Any news?

[sanikolaev]

Any news?

Yes. This task has been blocked by the work related with improving date/time functions in Manticore - https://github.com/manticoresoftware/manticoresearch/commit/514d35b497f4bdb20b6473ff963752a978a4bb8d

That's now done and we are getting back to this task soon.

I remember @nickchomey and a few others in the Public Slack were interested in beta-testing the new functionality. We'll ping you guys as soon as we have something to give you. Hope it will happen soon.

[deathalt]

I'd like to join kibana integration beta-testing :slightly_smiling_face:

[tomatolog]

I've built the package for testing. You could grab it from here https://github.com/manticoresoftware/manticoresearch/actions/runs/7957643370 and test integration.

[sanikolaev]

I've built the package for testing. You could grab it from here https://github.com/manticoresoftware/manticoresearch/actions/runs/7957643370 and test integration.

Thanks. Let's prepare a better instruction for beta-testers, so it's clear how to install the beta version.

[tomatolog]

seems @timestamp field handling got changed that breaks the Logstash integration

curl localhost:9308/cli -d 'create table test(@timestamp text)'

curl localhost:9308/_bulk -H "Content-Type: application/x-ndjson" -d '
{ "index" : {"_index":"test"} }
{"@timestamp": "a"}
'

[{"total":0,"warning":"","error":"row 1, column 1: string expected"}]

[tomatolog]

should be fixed at the master head https://github.com/manticoresoftware/manticoresearch/commit/15975cf3e6471d3a3ca4c2f19e4e930359a14622

[sanikolaev]

@Nick-S-2018 As discussed, reassining to you to continue working on the project from the Buddy's standpoint. Here's an updated instruction how to set up Kibana + Manticore + Elasticsearch + HAproxy + Mitmproxy:

Testing Kibana

# Schema

kibana @5601 -> haproxy @9201 -> mitm @19308 (UI @29308) -> manticore @9308
                              -> mitm @19200 (UI @29200) -> elastic @9200

apt -y update
apt -y install docker.io mysql-client php php-mysql jq

# Init .screenrc
cat << EOF > ~/.screenrc
caption always "%{= 45}%{+b w}Screen: %n | %h %=%t"
hardstatus alwayslastline "%-Lw%{= BW}%50>%n%f* %t%{-}%+Lw%<"
shell -/bin/bash
EOF

# Install and start Manticore normally

# Install mitmproxy
wget https://downloads.mitmproxy.org/10.2.2/mitmproxy-10.2.2-linux-x86_64.tar.gz
tar -xvf mitmproxy-10.2.2-linux-x86_64.tar.gz

# Run Elasticsearch
mkdir es_data && chmod 777 es_data
docker run --name elasticsearch --rm -e discovery.type=single-node -v $(pwd)/es_data:/usr/share/elasticsearch/data --network=host docker.elastic.co/elasticsearch/elasticsearch:7.6.0


# Run haproxy

cat << EOF > haproxy_kibana.cfg
global
   log stdout format raw local0
   daemon

defaults
   log global
   mode http

frontend esms
   bind *:9201
   default_backend manticore
   acl idx path_beg /. /_
   use_backend elasticsearch if idx

backend manticore
   server manti 127.0.0.1:19308

backend elasticsearch
   server es 127.0.0.1:19200
EOF

docker run -v $(pwd)/haproxy_kibana.cfg:/usr/local/etc/haproxy/haproxy.cfg --network=host haproxy:2.6.5

# Run mitmproxy for Elasticsearch 
./mitmproxy --mode reverse:http://localhost:9200 -p 19200

  # Alternatively, run mitmweb
  # ./mitmweb --web-host 0.0.0.0 --web-port 29200 --mode reverse:http://localhost:9200 -p 19200
  # Open http://<host>:29200

# Run mitmproxy for Manticore
./mitmproxy --mode reverse:http://localhost:9308 -p 19308

  # Alternatively, run mitmweb
  # ./mitmweb --web-host 0.0.0.0 --web-port 29308 --mode reverse:http://localhost:9308 -p 19308
  # Open http://<host>:29308


# Install and start Manticore normally
wget https://repo.manticoresearch.com/manticore-dev-repo.noarch.deb && \
sudo dpkg -i manticore-dev-repo.noarch.deb && \
sudo apt -y update && \
sudo apt -y install manticore manticore-extra manticore-common manticore-server manticore-server-core manticore-tools manticore-executor manticore-buddy manticore-backup manticore-columnar-lib manticore-server-core-dbgsym manticore-tools-dbgsym manticore-columnar-lib-dbgsym manticore-icudata-65l manticore-galera manticore-galera-dbgsym 

# Run Kibana through the proxy
docker run --name kibana --rm -e SERVER_NAME=kibana -e ELASTICSEARCH_HOSTS=http://127.0.0.1:9201 -e ELASTICSEARCH_USERNAME=elastic -e ELASTICSEARCH_PASSWORD=pass -e SERVER_PORT=5601 -e KIBANA_AUTOCOMPLETETIMEOUT=10000 -e KIBANA_AUTOCOMPLETETERMINATEAFTER=100000000 -e ELASTICSEARCH_REQUESTTIMEOUT=600000 -e ELASTICSEARCH_SHARDTIMEOUT=0 --network=host docker.elastic.co/kibana/kibana:7.6.0

[gaby]

@sanikolaev Why such a complicated setup? Can I just run manticore / kibana in compose without all this?

Can Kibana v8.x be used?

[sanikolaev]

@gaby That's a temporary setup for development. Of course when we are done with the integration you'll be able to just point Kibana to Manticore. But we are not there yet. Still some work left until we can announce that.

[gaby]

@sanikolaev Thanks for the explanation, looking forward to this! 💪

[Nick-S-2018]

Blocked by https://github.com/manticoresoftware/manticoresearch/issues/1993

[sanikolaev]

Blocked by https://github.com/manticoresoftware/manticoresearch/issues/1993

#1933 is fixed and closed.

[Nick-S-2018]

Was blocked by https://github.com/manticoresoftware/manticoresearch/issues/2064, which is fixed now.

[sanikolaev]

Blocked by https://github.com/manticoresoftware/manticoresearch/issues/2064

[tomatolog]

the #2064 is fixed in the master head

[sanikolaev]

Blocked by https://github.com/manticoresoftware/manticoresearch/issues/2110

[sanikolaev]

Blocked by https://github.com/manticoresoftware/manticoresearch/issues/2110

Fixed

[Nick-S-2018]

Blocked by https://github.com/manticoresoftware/manticoresearch/issues/1744

[sanikolaev]

Blocked by https://github.com/manticoresoftware/manticoresearch/issues/1744

Unblocked.