is there any ELK support for v6d?

[lidh15]

Is it possible to introduce metricbeat and other ELK ecosystem beats into v6d so that we can monitor the status in Kibana Web UI?

[sighingnow]

We previously have some integration with prometheus (see https://github.com/v6d-io/v6d/tree/main/prometheus).

Integration of metrics with ELK could be added as well. May I know a bit about your scenarios of leveraging vineyard? Then we can consider the detail integration plan more carefully.

Thanks!

[lidh15]

we use vineyard as a memory data file system, previously we tends to have lots of pd.DataFrame in a single jupyter notebook but with the project developing, we have many engineers and everyone has many ipynbs, pd.to_csv and pd.read_csv were too slow on comparing with putting things in v6d so we take v6d into the project. we heavily depends on the put_name and get_name methods in v6d because every dataframe is provided with such a name as they were treated in csv file. the demands including:

1. sometimes we give a used name to a new dataframe and we lost some dataframes in v6d, so we want to see those objects without names and delete them asap.
2. we want to check the memory usage of every named object as well as how far it is to the memory limit so that we can evaluate if we can put a new dataframe in or we need to delete some objects, I know there are some apis like ipcclient.memory_usage for this purpose, but we want a real time web GUI rather than open an ipynb and execute a cell everytime.
3. of course there are more, but let's starts with the above ones.

[sighingnow]

Hi @lidh15,

Thanks for the detail information. Base the the scenario above, I'm wondering if prometheus works for you? Or your software stack already has the ELK dependency?

If prometheus is acceptable, I think we could prioritize the integration with prometheus. Other a bit longer development time would be required, IMO.

[lidh15]

Hi @lidh15,

Thanks for the detail information. Base the the scenario above, I'm wondering if prometheus works for you? Or your software stack already has the ELK dependency?

If prometheus is acceptable, I think we could prioritize the integration with prometheus. Other a bit longer development time would be required, IMO.

yes, ELK has been embedded in our stack and we have full ELK system deployed. It won't be difficult to enable a new middleware for ELK beats, but Prometheus was not used before. I will try Prometheus after all, thank you.

[sighingnow]

See also #766.