apisix
apisix copied to clipboard
apache
init elasticsearch-logger.lua
Description
I followed the clickhouse-Logger implementation logic and the data store was partially adapted for ELasticsearch;By default, index names are divided by date (yyyy.MM.dd),Dbas can further optimize log storage by combining index templates.
finally,i use elasticsearch Restful Api and Basic Authorization method,it an suitable for 5.x 、6.x and 7.x versions
Hi @fanchangjifen , thanks for your contribution, but please let me know if we ever discussed this on the APISIX mailing list?
没讨论过，我们主要的日志存储、检索沿用ELK技术解决方案，所以看到Clickhouse的实现后，引入Elasticsearch的接入方案。 当然，我们现在的解决方案不仅限于直接直接与Es进行交互
for details：
【金山文档】 Plugin elasticsearch-logger test https://kdocs.cn/l/cgTYj4imEMld
In the production environment, we may not use direct interface interaction with the storage engine. According to experience, there may be some problems in connection pool and performance. Therefore, we now use date slicing of HTTP-Logger and use FileBeat to collect data. Using ingest pipeline preprocessing specification data persistence, this is the full range of solutions to prevent. However, this plug-in might be an easy way to label Clickhouse-Logger
I see. Let's wait for someone who is willing to try this plugin in the product environment.
I see. Let's wait for someone who is willing to try this plugin in the product environment.
I still want to know what do I need to do next for this plug-in to be accepted
@fanchangjifen can you submit a doc about how to use this plugin? I think this plugin is meaningful now.
It can help more users follow this pr.
for details：
【金山文档】 Plugin elasticsearch-logger test https://kdocs.cn/l/cgTYj4imEMld
@membphis hey,there it is.
@fanchangjifen After discussing with @leslie-tsang and @tzssangglass, I am sorry to have to inform you that, given the cost of maintenance, we do not intend to move forward with this feature unless more users are willing to use it.
@fanchangjifen After discussing with @leslie-tsang and @tzssangglass, I am sorry to have to inform you that, given the cost of maintenance, we do not intend to move forward with this feature unless more users are willing to use it.
This is just an idea based on the Clickhosue-Logger plugin, and while we regret the final result, we will continue to follow the Apisix project and contribute our thoughts on SM cryptography and logging
@fanchangjifen I discussed it again with @tzssangglass. We can accept this plugin but with a requirement: we need to test against real ES.
We can run ES like https://github.com/elastic/elastic-github-actions/blob/master/elasticsearch/run-elasticsearch.sh in https://github.com/apache/apisix/blob/master/ci/pod/docker-compose.yml.
The BULK API version has been submitted. In addition to separating data according to newline characters, the id of logs is automatically generated by ES or can be changed to SkyWalking ID late
It might be difficult for me to write test samples and i don’t know how to write
It might be difficult for me to write test samples and i don’t know how to write
You can refer to https://github.com/apache/apisix/blob/master/docs/en/latest/internal/testing-framework.md and run the test locally.
Also, some maintainers think we need to run the test with real elasticsearch. You can run ES like https://github.com/elastic/elastic-github-actions/blob/master/elasticsearch/run-elasticsearch.sh in https://github.com/apache/apisix/blob/master/ci/pod/docker-compose.yml. Then change the mock server to proxy the test requests to the real elasticsearch.
With reference to the official document submit related ci auxiliary https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-file. you may can run the test with real elasticsearch @spacewander What's next?
Let's merge master to make CI run. @spacewander merge conflicts ?
Let's merge master and solve the conflicts.
Hi~ I noticed that you seem to have no time to deal with this pr, and I'm interested in it, would you please let me continue to come up with this issue? : )
Hi~ I noticed that you seem to have no time to deal with this pr, and I'm interested in it, would you please let me continue to come up with this issue? : )
@ccxhwmy I have no way to automate the automated testing,I think you can build on what I've done.. I have read the PR you submitted，and I noticed that you didn't complete the logic of X-PACK certification, but i have finished.
@spacewander @tzssangglass #7643 for this issue,It's up to you
Hi~ I noticed that you seem to have no time to deal with this pr, and I'm interested in it, would you please let me continue to come up with this issue? : )
@ccxhwmy I have no way to automate the automated testing,I think you can build on what I've done.. I have read the PR you submitted，and I noticed that you didn't complete the logic of X-PACK certification, but i have finished.
I will refer to what you have finished, thank you very much!
@spacewander @tzssangglass #7643 for this issue,It's up to you
Thanks for your contribution, let's do it.
@spacewander @tzssangglass #7643 for this issue,It's up to you
Thanks for your contribution, let's do it.
@ccxhwmy As mentioned above, you should use the elasticsearch bulk API (endpoint/index/_bulk)..but I noticed you didn't. @tzssangglass Why not recommend testing on my basis or use my plugin directly.
As mentioned above, you should use the elasticsearch bulk API (endpoint/index/_bulk)..but I noticed you didn't.
Yes, we need to implement this.
Why not recommend testing on my basis or use my plugin directly.
It's not a decision I made. I guess the reason is the read/write access of the branch, he may not have the write access of your PR branch
@spacewander @tzssangglass #7643 for this issue,It's up to you
Thanks for your contribution, let's do it.
@ccxhwmy As mentioned above, you should use the elasticsearch bulk API (endpoint/index/_bulk)..but I noticed you didn't.
Yes, I will implement this suggestion, Thank you.