micro-integrator
micro-integrator copied to clipboard
Published
20 hours ago •
wso2
wso2
Choreo based analytics for MI
Problem
Currenty MI does not support Choreo based analytics for MI
Solution
Choreo based analytics for MI design and implementation
Affected Component
MI
Version
No response
Implementation
No response
Sub Tasks
- [x] wso2/api-manager#1993
- [x] wso2/api-manager#1992
No response
Suggested Labels
No response
Work Log [2023-06-20]
- Got assigned to do the feature “Choreo based analytics for MI”.
- Discussed about the feature and I was asked to try analytics with EI-6.6.
- This is because EI contained all in one single pack.
Work Log [2023-06-21]
- Set the configurations and tested the analytics on EI-6.6.
- Used a simple car with a simple API to test.
- Followed this documentation.
Work Log [2023-06-22]
- Created a choreo account and tested APM 4.2 analytics.
- This is done to get an idea of choreo capabilities on analytics.
- Followed this documentation.
- Further discussed the feature expectations with the lead showing above tested analytics dashboards.
- Started working on a design documentation.
Work Log [2023-06-23]
- Discussed with the developer who worked on “Choreo based analytics for APIM”
- Decided to define a rough scope before going further
- Since MI is a sub part of EI, decided to check the latest MI analytics feature
- Wasn’t able to setup the analytics correctly for Mi 4.2
Work Log [2023-06-26]
- For MI 4.2, we need three packs to correctly test analytics.
- wso2ei-analytics-7.1.0 (Link)[https://github.com/wso2/observability-ei/releases]
- wso2mi-4.2.0
- Wso2mi-dashboard-4.2.0 (To easily check deployed artifacts)
- Identified that issue is because ei-analytics-7.1.0 does not support MI 4.2 analytics and needed to be updated using u2.
- Referred this
- Couldn’t update ei-analytics using u2 (There was no wso2update_darwin file in based pack’s bin directory.
- Checked the U2 Starting Timestamps but couldn’t find ei-analytics 7.1 there.
- Referred (this)[https://sites.google.com/wso2.com/updatemanagertool?pli=1#h.8mvipdfo6jso]
- Configure AWS and checked the S3 bucket and wasn’t able to find ei-analytics-7.1 there.
- Checked the dates for ei-analytics-7.1 (Nov 16, 2020) and u2 starting dates(October, 2020).
- Assumed that EI 7.1 base pack should support U2 but missed the u2 tool in the pack because of the above things.
- Asked from the U2 team to further clarify this issue.
- Later, Got to know that the u2 tool has not been added in the Github release of this.
- Tried using the u2 update tool of another but faced below error.
wso2update: Errors while applying update: Error while modifying files: Error while getting product pack file content: Unable to read file '<wso2ei-analytics-7.1.0 path>/wso2/portal/resources/dashboards/ei-anlaytics.json': open <wso2ei-analytics-7.1.0 path>/wso2/portal/resources/dashboards/ei-anlaytics.json: no such file or directory
Product pack content may have been modified during this failed update. We recommended reverting to the previous state using the '--revert' flag before running the update tool again
- Got a fresh analytics pack and redo the above step and was able to update successfully
- Although MI dashboard is available now in analytics, I wasn’t able to view statistics related to my artifacts.
- This may be because both packs are not connected to the same db so some issue like that.
- Observed that this analytics dashboard (EI 7.1 which has MI dashboard) is the same EI 6.6 dashboard which I earlier tested.
- It contains same equal components (Checked with the documented ones from EI 6.6)
Work Log [2023-06-27]
- Send a mail about the unavailability of the WSO2 EI Analytics 7.1.0
- Subject - “Couldn't Find WSO2 EI Analytics 7.1.0 [1] on Product Page of Our Website”
- MI-Analytics is not working properly and trying to find out the reason behind the issue.
- EI_ANALYTICS.mv.db Database in analytics server side is empty so there must be a data publishing issue.
- Configured ei_analytics.server_url is "tcp://localhost:7612" so connection is checked using
nc -zv localhost 7612and observed that the connection to localhost port 7612 [tcp/*] succeeded.
- Wasn't able to fix the issue
Work Log [2023-06-28]
- Tried to fix the classical MI analytics solutions setup's isssue.
- Had a call and tried to fix the issue in that call.
- Changed the EI analytics setup to mysql and try to diagnose the issue.
- Identified that there is a documentation issue.
- Identified the the problem as
MI is not trying to find the EI analytics server urls even though we followed the documentation. Problem is because there is no mentioned way to enable EI analytics other than adding the configurations - Note that we even tried to enable [analytics] in MI deployment.toml which updates the synapse.properties
- This led the product to print the analytics in the server logs.
- Had a meeting with Lead and feature mentor and decided the future plan.
-
[Discussion] Choreo based analytics for MI @2.00PM - Discussed about the plan going ahead.
- Decided to update the design doc and to have a call with a Choreo expert tomorrow.
- Decided to stop trying to fix the above mentioned issue for now.
-
- Updated the design documentation.
- Started analysing the ELK analytics code.
Work Log [2023-06-29]
- Had a meeting with a Choreo team lead.
-
[Discussion] Feasibility of Choreo based analytics for MI @11.00AM - Decided to find out the real sample payloads for all artifacts.
-
- Updated the design document with founded samples.
- Wasn't able to finish this since IntegrationStudio is new to me so there was some learning curve.
Work Log [2023-06-30]
- Finished finding all the payloads and updated the document with those payloads.
- Started working on finding the required data to generate analytics components.
Work Log [2023-07-04]
- Sorted out the all data needed for the components.
- Updated the above document with those data and a proposed payload.
- Roughly estimated the required data and how we can generate and store them
- Had a meeting to review those findings.
-
[Discussion] [Design Doc] Choreo Analytics for MI @3.30PM - Decided to publish all the data we get as they can later be important.
- Asked to find out about analytics metadata and can we add that too to the payload.
-
- Started analysing the code to find the root of that analytics metadata.
Work Log [2023-07-05]
- Since metadata is a thing in ELK where customers will get to add additional values and keys, we are unable to get this to a flat pre defined json format. (Since keys can be changed)
- Decided to ignore this for now since it's not important for the dashboards.
- Had another meeting with a Choreo team member.
-
[Discussion] Choreo Analytics for MI 4.3 Payload Format @11.00AM - Decided that designed payload is fine.
- Decided to initiate a mail thread in architecture group about the feature.
-
- Formatted the design document to the official design format.
- Started the mail thread in architecture group.
- Started the implementation.
Work Log [2023-07-06]
- Got asked to change the design doc to a design outline.
- Changed the design doc to a design outline.
- Official Design Doc - Click here
- Official Design Outline - Click here
Work Log [2023-07-07]
- Implemented the functionalities to generate the required payload for Choreo analytics.
Work Log [2023-07-10]
- Fixed the implementation to work with all artefacts.
- Tested the payload with all 5 artefacts.
Work Log [2023-07-11]
- Completed the work for generating the payload.
- Had a discussion about the future of the feature.
-
[Discussion] Micro Integrator Analytics Using Choreo @3.00PM - Decided to have a wider discussion.
-
Work Log [2023-07-12]
- Had a discussion about the feature.
-
[Discussion] Micro Integrator Analytics Using Choreo @11.30AM - Decided to create Choreo UIs before going forward.
-
- Started doing the UI Designing
Work Log [2023-07-13]
- Completed designing UIs.
- Had a UI review.
-
[UI Review] Choreo Analytics for Micro Integrator @4.00PM - Decided to share the UIs with architecture group after documenting.
-
Work Log [2023-07-14]
- Decided to only focus on # 1 task for now after some discussions.
- Refer
[Architecture] Micro Integrator Analytics Using Choreofor more info.
- Refer
- Studied the anlytics-publisher repository.
Work Log [2023-07-18]
- Decided to hold the feature.
- Offline discussion
Discussion] Micro Integrator Analytics Using Choreo @10.30AM - Refer
[Architecture] Micro Integrator Analytics Using Choreofor more info.
- Offline discussion
- Finished the started implementations.
- Created a new feature branch in micro integrator.
- https://github.com/wso2/micro-integrator/tree/choreo-analytics
- Sent the PR
- https://github.com/wso2/micro-integrator/pull/2927
Work Log [2023-07-19]
- Had a further discussion about the future of the feature.
- Waited for the PR to get reviewed.