[META] Implement Extension Points

[dbwiddis]

This issue is created to track Extension Point implementation.

• Unchecked boxes are not implemented at all.
• Checked boxes have at least a partial implementation complete which addresses the most important use cases; some may need additional work for completeness, which will be identified by a linked issue.
• Extension points highlighted in bold are a priority as they are required for plugin migration

Not all extension points will need to be implemented for Extensions, and will be removed from the list when that determination is made.

The general approach to implementation is:

1. Create a FooExtension interface corresponding to the FooPlugin interface, with the same default implementations.
2. During initialization, communicate which interfaces an Extension implements to OpenSearch
3. Review existing functionality, usually implemented in the Node class that runs on OpenSearch bootstrap, where a module iterates over plugins implementing a particular interface and and "registers" the implemented extensions points, adding them to maps/lists, etc.
4. Determine whether functionality is primarily local to the extension (the easy case) or integral to OpenSearch (which requires serializing and deserializing via Writable interface) and implement appropriately. For example:
   • NamedXContent is local to an extension. The static initializers from OpenSearch are created and no transport communication actually happens.
   • RestHandlers are integrated with the OpenSearch RestController. The handled methods and paths must be communicated over transport to OpenSearch to be registered. This is done using Strings.
   • Settings are integrated with OpenSearch. The settings must be turned into a Writeable object to be sent over transport, and re-created on the other end. This is done using the WriteableSetting class.

Extension Points applicable to all plugins are defined in the Plugin interface:

• [ ] getFeature
• [ ] createGuiceModules
• [ ] getGuiceServiceClasses
• [ ] additionalSettings
• [x] createComponents (see comment)
• [x] getNamedWriteables (partially implemented, needs to be merged with local extension point, see #291)
• [x] getNamedXContent
• [ ] #776
• [x] getSettings (see #148)
• [ ] getSettingsFilter
• [ ] getSettingUpgraders
• [ ] getIndexTemplateMetadataUpgrader
• [ ] getExecutorBuilders
• [ ] getBootstrapChecks
• [ ] getRoles
• [ ] getAdditionalIndexSettingProviders

The following interfaces also implement custom functionality:

• ActionPlugin

  • [x] getActions (Implemented locally on an Extension but not yet registered in ActionModule, see #107)
  • [ ] getClientActions
  • [x] getActionFilters
  • [x] getRestHandlers
  • [ ] getRestHeaders
  • [ ] getTaskHeaders
  • [ ] getRestHandlerWrapper
  • [ ] mappingRequestValidators
  • [ ] indicesAliasesRequestValidators

• AnalysisPlugin

  • [ ] getCharFilters
  • [ ] getTokenFilters
  • [ ] getTokenizers
  • [ ] getAnalyzers
  • [ ] getPreBuiltAnalyzerProviderFactories
  • [ ] getPreConfiguredCharFilters
  • [ ] getPreConfiguredTokenFilters
  • [ ] getPreConfiguredTokenizers
  • [ ] getHunspellDictionaries

• CircuitBreakerPlugin

  • [ ] getCircuitBreaker
  • [ ] setCircuitBreaker

• EnginePlugin

  • [ ] getEngineFactory
  • [ ] getCustomCodecServiceFactory
  • [ ] getCustomTranslogDeletionPolicyFactory

• ExtensiblePlugin

  • [ ] loadExtensions

• IndexStorePlugin

  • [ ] getDirectoryFactories
  • [ ] getRecoveryStateFactories

• IngestPlugin

  • [ ] getProcessors

• MapperPlugin

  • [ ] getMappers
  • [ ] getMetadataMappers
  • [ ] getFieldFilter

• PersistentTaskPlugin

  • [ ] getPersistentTasksExecutor

• RepositoryPlugin

  • [ ] getRepositories
  • [ ] getInternalRepositories

• ReloadablePlugin

  • [ ] reload

• ScriptPlugin

  • [ ] getScriptEngine
  • [ ] getContexts

• SearchPlugin

  • [ ] getScoreFunctions
  • [ ] getSignificanceHeuristics
  • [ ] getMovingAverageModels
  • [ ] getFetchSubPhases
  • [ ] getSearchExts
  • [ ] getHighlighters
  • [ ] getSuggesters
  • [ ] getQueries
  • [ ] getSorts
  • [ ] getAggregations
  • [ ] getAggregationExtentions
  • [ ] getCompositeAggregations
  • [ ] getPipelineAggregations
  • [ ] getRescorers
  • [ ] getQueryPhaseSearcher
  • [ ] getIndexSearcherExecutorProvider

• SystemIndexPlugin

  • [ ] getSystemIndexDescriptors

[dbwiddis]

We don't have a create components method, and instead are directly implementing the required components. Some are associated with other extension points.

The following are part of createComponents and are significant enough to be tracked separately as part of that extension point:

• [x] Client (We have SDKClient which provides this access)
• [x] ClusterService
• [x] ThreadPool (Not needed for now but will be needed for in-process)
• [ ] ResourceWatcherService
• [ ] ScriptService
• [x] NamedXContentRegistry
• [x] Environment
• [ ] NodeEnvironment
• [x] NamedWriteableRegistry
• [ ] IndexNameExpressionResolver
• [ ] Supplier<RepositoriesService>

Extensions which interact with Job Scheduler will need JobSchedulerExtension extension points. These are being implemented in the job-scheduler repository.

• [x] getJobType
• [x] getJobIndex
• [x] getJobRunner
• [x] getJobParser

[dbwiddis]

The following plugins may not be relevant for extensions. They are documented here for completeness:

• ClusterPlugin

  • [ ] createAllocationDeciders
  • [ ] getShardsAllocators
  • [ ] getExistingShardsAllocators
  • [ ] onNodeStarted

• DiscoveryPlugin

  • [ ] getCustomNameResolver
  • [ ] getSeedHostProviders
  • [ ] getJoinValidator
  • [ ] getElectionStrategies

• NetworkPlugin

  • [ ] getTransportInterceptors
  • [ ] getTransports
  • [ ] getHttpTransports

[minalsha]

@dbwiddis why are https://github.com/opensearch-project/opensearch-sdk-java/issues/315#issuecomment-1377742853 not relevant for extensions?

[dbwiddis]

why are #315 (comment) not relevant for extensions?

@minalsha I used the qualifier "may not be relevant". I'd appreciate another set of eyes on this (@saratvemulapalli).

• ClusterPlugin directly deals with shards on the Opensearch cluster and I don't think that's something we're directly handling on the Extension side
• DiscoveryPlugin is also an OpenSearch cluster feature dealing with nodes joining, which to elect manager, etc.
• NetworkPlugin deals with the transport later comms inside OpenSearch, but we are using Rest comms for extensions so they shouldn't be relevant.

[minalsha]

List of prioritized and ranked Extension Points, we want to go after are as follows:

• AnalysisPlugin

  • [ ] getTokenFilters
  • [ ] getTokenizers
  • [ ] getAnalyzers
  • [ ] getCharFilters

• Plugin

  • [ ] createComponents
  • [x] getSettings
  • [x] getNamedXContent
  • [ ] getExecutorBuilders
  • [ ] onIndexModule
  • [ ] close
  • [ ] getGuiceServiceClasses
  • [ ] getNamedWriteables

• ActionPlugin

  • [x] getActionFilters
  • [x] getRestHandlers
  • [x] getActions

• ReloadablePlugin

  • [ ] reload

• RepositoryPlugin

  • [ ] getRepositories

• MapperPlugin

  • [ ] getMappers

• IngestPlugin

  • [ ] getProcessors

• ExtensiblePlugin

  • [ ] loadExtensions

• ScriptPlugin

  • [ ] getScriptEngine

• SystemIndexPlugin

  • [ ] getSystemIndexDescriptors

• Extensions which interact with Job Scheduler will need JobSchedulerExtension extension points. These are being implemented in the job-scheduler repository.

  • [x] getJobType
  • [x] getJobIndex
  • [x] getJobRunner
  • [x] getJobParser