[RFC][Build] extra plugins and sourcing functionality in OSD that is not available in non-min artifact OpenSearch Core

[kavilla]

Glossary

• OSD: Abbreviation for OpenSearch Dashboards
• Third-party plugin: Any plugin not included in the core repository, officially referred to as "third-party plugin" (literally within code)

Problem

Historically, the inclusion of features within the OSD repository has been guided by whether the minimal distribution of OpenSearch offers those features without necessitating a third-party plugin. If not, such features should be available through an OSD third-party plugin that isn't an internal one.

This guideline aimed to preserve the integrity of default experiences for projects and developers forking from OSD, ensuring seamless operations in their instances.

This approach has generally been effective, facilitating:

• Clear ownership and maintenance models for contributors
• Assurance that the open-source minimal distribution remains free of non-open-source features
• Maintenance of a 1:1 feature parity with upstream OpenSearch dependencies
• Development stability, offering a consistent default state without additional configurations or setup, and avoiding version mismatches
• Enabling developers to utilize capability APIs or plugin checks for conditional logic
• Allowing extension of existing OSD functionalities for additional registrable functions

However, this guideline has introduced several challenges:

• Necessity for conditional logic or code duplication by plugins for functionalities not available by default
• Potential for circular plugin dependencies
• Extension requirements for unavailable features or components, leading to increased ramp-up times
• Development and build complications due to local path and reference discrepancies
• Lack of guaranteed runtime precedence for third-party plugins over core plugins, leading to non-standard solutions
• Unlike OpenSearch, plugin interactions with each other and the core through APIs are not fully shared

Proposed Solution

The reintroduction of extras plugins (and possibly packages) to OSD is proposed. This concept has never been entirely phased out from the repository [example]

The proposal includes:

Building all plugins by default, including those requiring non-minimal OpenSearch functionalities Serving all plugins by default in distribution processes, leveraging existing OSD core logic Excluding extra plugins by default in development mode Introducing a flag to exclude extra plugins from the build process, catering to partners preferring a minimal distribution Offering a flag to prevent serving extra plugins in distributions, allowing partners to limit access Providing a flag to enable all extra plugins in development mode While the specifics of these flags and default settings are open for discussion, the overarching goal is to facilitate access to foundational logic not present in the minimal distribution but utilized by multiple plugins, benefiting partners and developers alike.

This approach also aims to centralize foundational logic, ensuring application stability through comprehensive linting and automation, thereby preventing potential issues and bad practices from downstream plugins.

Related PR adding some of this solution: https://github.com/opensearch-project/OpenSearch-Dashboards/pull/7080

Other info

This is targeting https://github.com/kavilla/queryEnhancements which registers PPL and SQL languages to the data plugin