Storage layer plugin based on `go plugin`

[Iceber]

What would you like to be added?

Clusterpedia works with different storage components such as MYSQL/Postgres, memory or ES through a storage layer.

Currently these storage layers need to be compiled into Clusterpedia components, and users who want to implement their own storage layers need to recompile Clusterpedia to build new images.

We want an open mechanism that allows users to freely interface to any storage layer without the performance loss associated with external storage layers.

Usually the RPC is a less invasive and more decoupled plug-in option, but the performance loss due to resource serialization and deserialization can have a significant impact on Clusterpedia components.

So we decided to use the Go Plugin and provide a series of scripts and images to simplify user implementation of the storage layer plug-in.

With the storage layer plugin based on Go Plugins, Users can

• Use the arbirary storage components
• Implement a more appropriate storage layer based on your business
• Implement more search criteria in the custom storage layer

How do users use the storage layer plugin?

Local developing and running

Clusterpedia provides the hack/builder.sh script, so you can use hack/builder.sh to compile Clusterpedia components and storage layer plugins separately, and then specify the plugins directory through the environment variable STORAGE_PLUGINS when running Clusterpedia components.

The storage layer needs to be loaded into the component program before initializing command flags, so you can't specify the storage layer plugin directory via flags, but use the environment variable to set.

Example:

$ ls -al
drwxr-xr-x  45 icebergu  staff   1.4K 11  3 12:53 clusterpedia
drwxr-xr-x  20 icebergu  staff   640B 11  2 13:34 sample-storage-layer

$ cd sample-storage-layer
$ # builder.sh --help
$ CLUSTERPEDIA_REPO=../clusterpedia builder.sh apiserver
$ CLUSTERPEDIA_REPO=../clusterpedia builder.sh plugins sample-storage-layer.so

$ STORAGE_PLUGINS=./plugins ./bin/apiserver --storage-name=sample-storage-layer --storage-config=./config.yaml

Deploying with helm

Clusterpedia provides use cases for building plug-in images in https://github.com/clusterpedia-io/sample-storage-layer, and users can refer to the use case scripts to build plug-in images

Helm will support setting up the storage layer plugin image, Clusterpedia Pod will copy the plugins from the storage layer plugin image to the container's plugin directory, and then set the name of the storage layer plugin

Example:

$ helm install --set storage.image="ghrc.clusterpedia.io/sample-storage-layer:latest" --set storage.name="sample-storage-layer" clusterpedia clusterpedia

Tasks that support storage layer plugins

• [x] Clusterpedia supports storage tier plugins and provides the build flow that supports go plugins https://github.com/clusterpedia-io/clusterpedia/pull/436
• [x] Create a sample-storage-layer repository and provide the build and development flow for the storage layer plugin https://github.com/clusterpedia-io/sample-storage-layer
• [ ] helm support for setting the storage layer plug-in
• [ ] Add storage layer usage and development documentation to the website
• [ ] To keep clusterpedia's main repository dependencies to a minimum, internalstorage is split into a separate module/repo, and clusterpedia loads the internalstorage storage layer via the storage layer plugin

TODO:

• The plugin binary file name should be suffixed with the corresponding clusterpedia version, clusterpedia will load the plugin according to the current version

[clusterpedia-bot]

Hi @Iceber, Thanks for opening an issue! We will look into it as soon as possible.

Details

Instructions for interacting with me using comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the gh-ci-bot repository.

[Iceber]

We need to add an init-plugin script to initialize the plugin directory, with a directory like https://github.com/clusterpedia-io/sample-storage-layer example:

$ init-plugin.sh github.com/iceber/new-storage
$ ls -al new-storage
-rw-r--r--   1 icebergu  staff     28 11  4 14:37 .dockerignore
drwxr-xr-x  13 icebergu  staff    416 11  4 14:59 .git
-rw-r--r--   1 icebergu  staff    101 11  1 15:03 .gitignore
-rw-r--r--   1 icebergu  staff    107 11  3 14:35 .gitmodules
-rw-r--r--   1 icebergu  staff    260 11  4 14:56 Dockerfile
-rw-r--r--   1 icebergu  staff   1183 11  4 14:57 Makefile
drwxr-xr-x  33 icebergu  staff   1056 11  4 14:54 clusterpedia
-rw-r--r--   1 icebergu  staff   2376 11  3 14:37 go.mod
-rw-r--r--   1 icebergu  staff  46109 11  3 14:37 go.sum
-rw-r--r--   1 icebergu  staff    139 11  4 10:42 main.go
drwxr-xr-x   3 icebergu  staff     96 11  4 13:30 plugins
drwxr-xr-x  16 icebergu  staff    512 11  4 13:21 storage
drwxr-xr-x   9 icebergu  staff    288 11  4 14:40 vendor

[Iceber]

We need to add an init-plugin script to initialize the plugin directory, with a directory like https://github.com/clusterpedia-io/sample-storage-layer example:

$ init-plugin.sh github.com/iceber/new-storage
$ ls -al new-storage
-rw-r--r--   1 icebergu  staff     28 11  4 14:37 .dockerignore
drwxr-xr-x  13 icebergu  staff    416 11  4 14:59 .git
-rw-r--r--   1 icebergu  staff    101 11  1 15:03 .gitignore
-rw-r--r--   1 icebergu  staff    107 11  3 14:35 .gitmodules
-rw-r--r--   1 icebergu  staff    260 11  4 14:56 Dockerfile
-rw-r--r--   1 icebergu  staff   1183 11  4 14:57 Makefile
drwxr-xr-x  33 icebergu  staff   1056 11  4 14:54 clusterpedia
-rw-r--r--   1 icebergu  staff   2376 11  3 14:37 go.mod
-rw-r--r--   1 icebergu  staff  46109 11  3 14:37 go.sum
-rw-r--r--   1 icebergu  staff    139 11  4 10:42 main.go
drwxr-xr-x   3 icebergu  staff     96 11  4 13:30 plugins
drwxr-xr-x  16 icebergu  staff    512 11  4 13:21 storage
drwxr-xr-x   9 icebergu  staff    288 11  4 14:40 vendor

• [ ] Using the stoage-template repository is better and does not require the use of scripts

[Iceber]

• [x] Update the sample repository name from sample-storage-layer to sample-storage
• [ ] Update the name of the sample storage plugin to sample-storage

And both the storage layer plugin repository and the storage layer plugin name are recommended to be suffixed with storage, e.g. *-storage

[Iceber]

• [x] Clusterpedia supports storage tier plugins and provides the build flow that supports go plugins https://github.com/clusterpedia-io/clusterpedia/pull/436
• [x] Create a sample-storage-layer repository and provide the build and development flow for the storage layer plugin https://github.com/clusterpedia-io/sample-storage-layer
• [x] Update the sample repository name from sample-storage-layer to sample-storage

/assign

[Iceber]

• [ ] helm support for setting the storage layer plug-in

/assign calvin0327

[hanweisen]

I suggest we use *-storage for module name, repository name and image name, use * as the plugin registration name without "-storage".

[tobehardest]

I'm a newbie, can I get involved too?