[multikv] application is shut down when secondary store(memberlist) failed to start

[vlad-diachenko]

In our case, we use primary store (consul) and secondary store (memberlist). So, when the app established a connection with consul, the service becomes Ready, and after some time if memberlist module failed to start, the whole application is shut down with messages:

level=error ts=2022-07-05T09:25:39.8466124Z caller=memberlist_client.go:466 msg="failed to join memberlist cluster" err="1 error occurred:\n\t │
│ level=info ts=2022-07-05T09:25:39.8468695Z caller=memberlist_client.go:591 msg="leaving memberlist cluster"                                    │
│ level=warn ts=2022-07-05T09:25:39.847216Z caller=module_service.go:112 msg="module failed with error" module=memberlist-kv err="service &{0xc0 │
│ level=error ts=2022-07-05T09:25:39.847582Z caller=loki.go:382 msg="module failed" module=memberlist-kv error="service &{0xc00097e8c0 { true 10 │
│ level=info ts=2022-07-05T09:25:39.8477152Z caller=basic_lifecycler.go:202 msg="ring lifecycler is shutting down" ring=admin-api                │
│ level=info ts=2022-07-05T09:25:39.8484468Z caller=compactor.go:369 msg="compactor exiting"                                                     │
│ level=info ts=2022-07-05T09:25:39.8485662Z caller=basic_lifecycler.go:202 msg="ring lifecycler is shutting down" ring=compactor                │
│ level=info ts=2022-07-05T09:25:39.8477701Z caller=manager.go:238 msg="stopping user managers"                                                  │
│ level=info ts=2022-07-05T09:25:39.8490085Z caller=manager.go:252 msg="all user managers stopped"

Actual Result: service is stopped if the secondary store(memberlist) does not work. Expected Result: service continues working if the primary store works.