containers icon indicating copy to clipboard operation
containers copied to clipboard

Invalid notify_amqp configuration prevents container from running

Open JD-Robertson opened this issue 3 years ago • 0 comments

Name and Version

bitnami/minio:2022.8.5-debian-11-r0

What steps will reproduce the bug?

  1. Deploy the minio container in standalone mode.
  2. Configure the notify_amqp notification target to point at a valid AMQP bus. We were using RabbitMQ but I don't think that's relevant.
  3. Restart the minio container. Verify things are working.
  4. Disable the AQMP bus (shut it down, disable network access, etc.). The minio container should no longer be able to access the bus.
  5. Restart the minio container.

What is the expected behavior?

An invalid notify target should not prevent the minio server from launching.

What do you see instead?

The container will fail to launch. The log will look like:

image

As best as I can tell, the setup.sh script is failing to properly shutdown the minio instance it uses to apply the configuration. So, when the production instance tries to start minio is already running and startup fails due to a port conflict. If I run the container with BITNAMI_DEBUG set to "true" I see some additional logging between configuring the "mc" instance and creating default buckets:

API: SYSTEM()
Time: 17:35:24 UTC 08/08/2022
DeploymentID: a9c6759d-5bec-4167-af96-10a33778e2dd
Error: dial tcp 10.42.6.89:5672: i/o timeout (*net.OpError)
15: internal/logger/logger.go:259:logger.LogIf()
14: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
13: internal/logger/logonce.go:135:logger.LogOnceIf()
12: internal/event/target/amqp.go:331:target.NewAMQPTarget()
11: internal/config/notify/parse.go:134:notify.fetchSubSysTargets()
10: internal/config/notify/parse.go:392:notify.FetchRegisteredTargets()
9: internal/config/notify/parse.go:95:notify.RegisterNotificationTargets()
8: internal/config/notify/parse.go:86:notify.GetNotificationTargets()
7: cmd/config-current.go:607:cmd.lookupConfigs()
6: cmd/config-current.go:871:cmd.loadConfig()
5: cmd/config.go:235:cmd.initConfig()
4: cmd/config.go:195:cmd.(*ConfigSys).Init()
3: cmd/server-main.go:406:cmd.initConfigSubsystem()
2: cmd/server-main.go:373:cmd.initServer()
1: cmd/server-main.go:554:cmd.serverMain()

API: SYSTEM()
Time: 17:35:24 UTC 08/08/2022
DeploymentID: a9c6759d-5bec-4167-af96-10a33778e2dd
Error: Unable to initialize notification target(s): one or more targets are offline. Please use `mc admin info --json` to check the offline targets (*fmt.wrapError)
8: internal/logger/logger.go:259:logger.LogIf()
7: cmd/config-current.go:609:cmd.lookupConfigs()
6: cmd/config-current.go:871:cmd.loadConfig()
5: cmd/config.go:235:cmd.initConfig()
4: cmd/config.go:195:cmd.(*ConfigSys).Init()
3: cmd/server-main.go:406:cmd.initConfigSubsystem()
2: cmd/server-main.go:373:cmd.initServer()
1: cmd/server-main.go:554:cmd.serverMain()

These are the errors you get for an invalid AMQP configuration. These errors are not fatal. If you run the minio server manually without going through the setup process, the same callstacks are printed to output but the server instance will start successfully. The bug here seems to be in the setup script itself, but I have not debugged into the guts of the script to work out what is going wrong.

Additional information

I was able to work around this issue but deploying the container without running the server, and then running the server manually and using mc to remove the broken notification target. We ran into this because a developer was experimenting in a test environment and didn't properly cleanup after themselves. I don't expect to run into the specific scenario again. But I thought it was worth filing the issue in case there are other cases where non-fatal errors in the minio configuration file can result in a broken container.

JD-Robertson avatar Aug 08 '22 19:08 JD-Robertson