repository type [s3] does not exist

[dipenpatel235]

I am working with s3 index backup. I created one s3 bucket, installed the repository-s3, and added the access_key and secret.

it's not working when I am creating the repository using devtool.

Please look at the query.

PUT _snapshot/my-s3-repository { "type": "s3", "settings": { "bucket": "my-s3-bucket", "base_path": "my/snapshot/directory" } }

it's giving the following error

ABC repository type [s3] does not exist

[dreamer-89]

Thanks @dipenpatel235 for sharing the issue.

it's not working when I am creating the repository using devtool.

Can you please share more details around the devtool ? How are you creating the S3 bucket, bucket policy etc. Please ensure the bucket has appropriate access policies.

You can find step by step guide in below documentation https://opensearch.org/docs/latest/opensearch/snapshots/snapshot-restore#register-repository

[dipenpatel235]

Hello @dreamer-89,

I have provided the step-by-step details that I have done.

1. create the s3 bucket
2. install the s3-repository plugin in all the nodes.

sudo ./bin/opensearch-plugin install repository-s3

3. provide access key and secret key to all the nodes.

sudo ./bin/opensearch-keystore add s3.client.default.access_key sudo ./bin/opensearch-keystore add s3.client.default.secret_key

4. reload all node settings using this dev tool query.

POST _nodes/reload_secure_settings

5. I have tried to create a repository using devtool query and get the error that repository type [s3] does not exist.

PUT _snapshot/my-s3-repository { "type": "s3", "settings": { "bucket": "my-s3-bucket", "base_path": "my/snapshot/directory" } }

[saratvemulapalli]

@dipenpatel235 thanks for the steps. Trying to understand, when did you run ./bin/opensearch and when did you install the plugin? Plugins should always be installed first and then run opensearch.

[kotwanikunal]

Thanks @saratvemulapalli. I can indeed confirm the above statement. I was able to reproduce the issue if I start OpenSearch before installing the plugin. I was able to fix it by first installing the plugin, configuring it and then starting OpenSearch.

[dipenpatel235]

@saratvemulapalli we have already installed s3 plugin of that.

[saratvemulapalli]

@dipenpatel235 could you drop your logs when you start the node? Here is mine which clearly loads respository-s3 plugin:

[2022-09-22T17:38:52,796][INFO ][o.o.i.r.ReindexPlugin    ] [ip-172-31-42-176] Unable to find any implementation for RemoteReindexExtension
[2022-09-22T17:38:52,847][INFO ][o.o.j.JobSchedulerPlugin ] [ip-172-31-42-176] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2022-09-22T17:38:52,851][INFO ][o.o.j.JobSchedulerPlugin ] [ip-172-31-42-176] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2022-09-22T17:38:52,852][INFO ][o.o.j.JobSchedulerPlugin ] [ip-172-31-42-176] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2022-09-22T17:38:52,858][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded module [aggs-matrix-stats]
[2022-09-22T17:38:52,858][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded module [analysis-common]
[2022-09-22T17:38:52,858][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded module [geo]
[2022-09-22T17:38:52,859][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded module [ingest-common]
[2022-09-22T17:38:52,859][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded module [ingest-geoip]
[2022-09-22T17:38:52,859][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded module [ingest-user-agent]
[2022-09-22T17:38:52,859][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded module [lang-expression]
[2022-09-22T17:38:52,859][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded module [lang-mustache]
[2022-09-22T17:38:52,859][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded module [lang-painless]
[2022-09-22T17:38:52,860][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded module [mapper-extras]
[2022-09-22T17:38:52,860][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded module [opensearch-dashboards]
[2022-09-22T17:38:52,860][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded module [parent-join]
[2022-09-22T17:38:52,860][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded module [percolator]
[2022-09-22T17:38:52,860][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded module [rank-eval]
[2022-09-22T17:38:52,860][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded module [reindex]
[2022-09-22T17:38:52,860][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded module [repository-url]
[2022-09-22T17:38:52,861][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded module [systemd]
[2022-09-22T17:38:52,861][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded module [transport-netty4]
[2022-09-22T17:38:52,861][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded plugin [opensearch-alerting]
[2022-09-22T17:38:52,861][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded plugin [opensearch-anomaly-detection]
[2022-09-22T17:38:52,862][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded plugin [opensearch-asynchronous-search]
[2022-09-22T17:38:52,862][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded plugin [opensearch-cross-cluster-replication]
[2022-09-22T17:38:52,862][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded plugin [opensearch-geospatial]
[2022-09-22T17:38:52,862][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded plugin [opensearch-index-management]
[2022-09-22T17:38:52,862][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded plugin [opensearch-job-scheduler]
[2022-09-22T17:38:52,862][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded plugin [opensearch-knn]
[2022-09-22T17:38:52,862][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded plugin [opensearch-ml]
[2022-09-22T17:38:52,863][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded plugin [opensearch-notifications]
[2022-09-22T17:38:52,863][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded plugin [opensearch-notifications-core]
[2022-09-22T17:38:52,863][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded plugin [opensearch-observability]
[2022-09-22T17:38:52,863][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded plugin [opensearch-performance-analyzer]
[2022-09-22T17:38:52,863][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded plugin [opensearch-reports-scheduler]
[2022-09-22T17:38:52,863][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded plugin [opensearch-sql]
[2022-09-22T17:38:52,864][INFO ][o.o.p.PluginsService     ] [ip-172-31-42-176] loaded plugin [repository-s3]

[kotwanikunal]

@dipenpatel235 Can you please help with the logs to move this further?

[dipenpatel235]

@kotwanikunal Here you can check In my cluster repository S3 plugin loaded successfully please check it.

[2022-09-27T09:21:04,108][INFO ][o.o.j.JobSchedulerPlugin ] [cordinator] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2022-09-27T09:21:04,133][INFO ][o.o.j.JobSchedulerPlugin ] [cordinator] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2022-09-27T09:21:04,134][INFO ][o.o.j.JobSchedulerPlugin ] [cordinator] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2022-09-27T09:21:04,136][INFO ][o.o.p.PluginsService     ] [cordinator] loaded module [aggs-matrix-stats]
[2022-09-27T09:21:04,136][INFO ][o.o.p.PluginsService     ] [cordinator] loaded module [analysis-common]
[2022-09-27T09:21:04,136][INFO ][o.o.p.PluginsService     ] [cordinator] loaded module [geo]
[2022-09-27T09:21:04,136][INFO ][o.o.p.PluginsService     ] [cordinator] loaded module [ingest-common]
[2022-09-27T09:21:04,137][INFO ][o.o.p.PluginsService     ] [cordinator] loaded module [ingest-geoip]
[2022-09-27T09:21:04,137][INFO ][o.o.p.PluginsService     ] [cordinator] loaded module [ingest-user-agent]
[2022-09-27T09:21:04,137][INFO ][o.o.p.PluginsService     ] [cordinator] loaded module [lang-expression]
[2022-09-27T09:21:04,137][INFO ][o.o.p.PluginsService     ] [cordinator] loaded module [lang-mustache]
[2022-09-27T09:21:04,137][INFO ][o.o.p.PluginsService     ] [cordinator] loaded module [lang-painless]
[2022-09-27T09:21:04,138][INFO ][o.o.p.PluginsService     ] [cordinator] loaded module [mapper-extras]
[2022-09-27T09:21:04,138][INFO ][o.o.p.PluginsService     ] [cordinator] loaded module [opensearch-dashboards]
[2022-09-27T09:21:04,138][INFO ][o.o.p.PluginsService     ] [cordinator] loaded module [parent-join]
[2022-09-27T09:21:04,138][INFO ][o.o.p.PluginsService     ] [cordinator] loaded module [percolator]
[2022-09-27T09:21:04,138][INFO ][o.o.p.PluginsService     ] [cordinator] loaded module [rank-eval]
[2022-09-27T09:21:04,139][INFO ][o.o.p.PluginsService     ] [cordinator] loaded module [reindex]
[2022-09-27T09:21:04,139][INFO ][o.o.p.PluginsService     ] [cordinator] loaded module [repository-url][2022-09-27T09:21:04,139][INFO ][o.o.p.PluginsService     ] [cordinator] loaded module [transport-netty4]
[2022-09-27T09:21:04,140][INFO ][o.o.p.PluginsService     ] [cordinator] loaded plugin [opensearch-alerting]
[2022-09-27T09:21:04,140][INFO ][o.o.p.PluginsService     ] [cordinator] loaded plugin [opensearch-anomaly-detection]
[2022-09-27T09:21:04,140][INFO ][o.o.p.PluginsService     ] [cordinator] loaded plugin [opensearch-asynchronous-search]
[2022-09-27T09:21:04,140][INFO ][o.o.p.PluginsService     ] [cordinator] loaded plugin [opensearch-cross-cluster-replication]
[2022-09-27T09:21:04,140][INFO ][o.o.p.PluginsService     ] [cordinator] loaded plugin [opensearch-index-management]
[2022-09-27T09:21:04,141][INFO ][o.o.p.PluginsService     ] [cordinator] loaded plugin [opensearch-job-scheduler]
[2022-09-27T09:21:04,141][INFO ][o.o.p.PluginsService     ] [cordinator] loaded plugin [opensearch-knn][2022-09-27T09:21:04,141][INFO ][o.o.p.PluginsService     ] [cordinator] loaded plugin [opensearch-observability]
[2022-09-27T09:21:04,141][INFO ][o.o.p.PluginsService     ] [cordinator] loaded plugin [opensearch-performance-analyzer]
[2022-09-27T09:21:04,141][INFO ][o.o.p.PluginsService     ] [cordinator] loaded plugin [opensearch-reports-scheduler]
[2022-09-27T09:21:04,142][INFO ][o.o.p.PluginsService     ] [cordinator] loaded plugin [opensearch-security]
[2022-09-27T09:21:04,142][INFO ][o.o.p.PluginsService     ] [cordinator] loaded plugin [opensearch-sql][2022-09-27T09:21:04,142][INFO ][o.o.p.PluginsService     ] [cordinator] loaded plugin [repository-s3]

Thanks, Dipen.

[saratvemulapalli]

@dipenpatel235 thanks for the logs. I do see: [cordinator] loaded plugin [repository-s3]. I am really surprised why type s3 error is showing up.

Do you mind turning on debug logs and try registering the snapshot repo?