[MD] Add design documents of multiple data source feature

[zhongnansu]

Signed-off-by: Su [email protected]

Description

Add below design documents

• high level design
• user stories
• client management design

Issues Resolved

#2524

Check List

• [ ] All tests pass
  • [ ] yarn test:jest
  • [ ] yarn test:jest_integration
  • [ ] yarn test:ftr
• [ ] New functionality includes testing.
• [ ] New functionality has been documented.
• [ ] Update CHANGELOG.md
• [ ] Commits are signed per the DCO using --signoff

[noCharger]

@zhongnansu My PR https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2510 contains some significant design updates. I'd appreciate it if we could get it finalized first.

[codecov-commenter]

Codecov Report

Merging #2538 (9f78f46) into main (8ac127a) will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #2538   +/-   ##
=======================================
  Coverage   66.73%   66.73%           
=======================================
  Files        3209     3209           
  Lines       61248    61248           
  Branches     9350     9350           
=======================================
  Hits        40875    40875           
  Misses      18137    18137           
  Partials     2236     2236           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[zhongnansu]

I like the idea of sharing design docs, but i dont think this is the right place to add them. Have we considered any other places to share these? Ideally we do that by opening an issue and add it to the contents there.

Developer docs can definitely reside in the code, design docs, i'm not so sure.

cc: @AMoo-Miki @joshuarrrr @kavilla @zengyan-amazon , what are your thought?

@ashwin-pc Glad we on the same page that adding design documents is good practice. I prefer adding those doc to repo, for iteration, and better reachability. Another benefit I can think of is external contributors could just use PR of design docs for design review. An example of reviewing user story of big feature that was contributed from external.

I am open to discussion

[seraphjiang]

@zhongnansu My PR https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2510 contains some significant design updates. I'd appreciate it if we could get it finalized first.

@noCharger Consider convert the ECD/issue to doc and commit with PR

[ashwin-pc]

Are we setting precedence here or has any other opensearch project done something similar?

[zhongnansu]

Are we setting precedence here or has any other opensearch project done something similar?

Reporting OSD plugin has done the similar. https://github.com/opensearch-project/dashboards-reports/tree/main/docs. Also check my previous comments how external contributors are contributing big feature to reporting, by also updating design docs & user stories.

plus notification plugin https://github.com/opensearch-project/notifications/tree/main/docs

[kavilla]

Couple of comments:

These are only adding the images to source control and not the files that were used to create these images. It would be really hard to recreate these diagrams from scratch as a community member if I need to make an update to those files.

There are references to OSD, for functional purposes we got the approval to utilize OSD. But references to the application should utilize OpenSearch Dashboards if it adds too much clutter then maybe a legend that lets the reader know that OSD = OpenSearch Dashboards. But I'm not sure if that's okay because we've been explicitly told to never reference OpenSearch Dashboards in public documentation as OSD.

There are references to OpenSearch Dashboards as Dashboards. Please use OpenSearch Dashboards, dashboards are specific component of OpenSearch Dashboards.

There are references to OpenSearch as opensearch. Please make sure this is OpenSearch.

Have we verified that this does not get built in to the final distribution (the min release)?

I do think in the past we have used blogs as design docs as a way to not commit to source controlling design docs and being held to update those. We also would also have to be responsible for updating this documents if a community member wanted to update multiple data sources. As maintainers we currently have issues with contributors communicating best practices per commit, so if a maintainer can't easily update these files then I can see this becoming outdated. Not that this a good reason to note include design documents but food for thought. More documentation is what what the community is asking for.

Thank you so much!

[zhongnansu]

@kavilla I have fixed the opensearch/osd naming conventions, thx. For img, I'll update with puml files. I also check the min artifacts, it includes the doc folder, same for any other components/plugins, will that be a concern ?

[ashwin-pc]

I like the idea of sharing design docs, but i dont think this is the right place to add them. Have we considered any other places to share these? Ideally we do that by opening an issue and add it to the contents there. Developer docs can definitely reside in the code, design docs, i'm not so sure. cc: @AMoo-Miki @joshuarrrr @kavilla @zengyan-amazon , what are your thought?

@ashwin-pc Glad we on the same page that adding design documents is good practice. I prefer adding those doc to repo, for iteration, and better reachability. Another benefit I can think of is external contributors could just use PR of design docs for design review. An example of reviewing user story of big feature that was contributed from external.

I am open to discussion

Ah good point @zhongnansu. That is a very valid use-case. My question then becomes, do we want to pull the location of these docs up higher in the repository? like a root level /docs folder like the PR you described since these are more architectural docs, they may touch more than a single plugin or component and it may be worth it to colocate them in a single place. E.g. All MDS work is split in 2 plugins, but it also affects the data plugin and how it functions. Changes there cant be harder to track if this doc stays in the datasource plugin folder and for more complex designs that touch more plugins

[zhongnansu]

@ashwin-pc Does a /docs/data-source path from root sound good to you?

[ashwin-pc]

@zhongnansu Yep! ./docs/multi-datasource maybe? Tying the name to a feature rather than the plugin itself. We can use the same folder for all the new projects including Angular migration, Visualization builder and Point in time etc

[seraphjiang]

@zhongnansu would you take a look @ashwin-pc 's comments?

[zhongnansu]

@ashwin-pc Thanks for all the comments! I addressed all of them, and also update the implementation section to reflect the latest change. Could you take another look