mongodb-container

mongodb-container copied to clipboard

petset-replicas, after removing all db pods, new pod cannot be running again, the expect result is all db pods are running again

# oc logs -f mongodb-0
=> sourcing 10-check-env-vars.sh ...
=> sourcing 30-set-config-file.sh ...
=> sourcing 35-setup-default-datadir.sh ...
=> sourcing 40-setup-keyfile.sh ...
=> [Tue Jul 25 09:29:37] Waiting for local MongoDB to accept connections ...
note: noprealloc may hurt performance in many applications
2017-07-25T09:29:38.317+0000 I CONTROL [initandlisten] MongoDB starting : pid=26 port=27017 dbpath=/var/lib/mongodb/data 64-bit host=mongodb-0
2017-07-25T09:29:38.317+0000 I CONTROL [initandlisten] db version v3.0.11
2017-07-25T09:29:38.317+0000 I CONTROL [initandlisten] git version: 48f8b49dc30cc2485c6c1f3db31b723258fcbf39
2017-07-25T09:29:38.317+0000 I CONTROL [initandlisten] build info: Linux c1bg.rdu2.centos.org 2.6.32-573.22.1.el6.x86_64 #1 SMP Wed Mar 23 03:35:39 UTC 2016 x86_64 BOOST_LIB_VERSION=1_53
2017-07-25T09:29:38.317+0000 I CONTROL [initandlisten] allocator: tcmalloc
2017-07-25T09:29:38.317+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017 }, replication: { oplogSizeMB: 64, replSet: "rs0" }, security: { keyFile: "/var/lib/mongodb/keyfile" }, storage: { dbPath: "/var/lib/mongodb/data", mmapv1: { preallocDataFiles: false, smallFiles: true } }, systemLog: { quiet: true } }
2017-07-25T09:29:38.322+0000 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/data/mongod.lock is not empty.
2017-07-25T09:29:38.332+0000 I STORAGE [initandlisten] **************
old lock file: /var/lib/mongodb/data/mongod.lock. probably means unclean shutdown,
but there are no journal files to recover.
this is likely human error or filesystem corruption.
please make sure that your journal directory is mounted.
found 2 dbs.
see: http://dochub.mongodb.org/core/repair for more information

2017-07-25T09:29:38.334+0000 I STORAGE [initandlisten] exception in initAndListen: 12596 old lock file, terminating
2017-07-25T09:29:38.334+0000 I CONTROL [initandlisten] dbexit: rc: 100

Originally reported by @dongboyan77 in https://github.com/sclorg/mongodb-container/pull/239#issuecomment-317691746

Deployment of application was done using this template https://github.com/sclorg/mongodb-container/blob/master/examples/petset/mongodb-petset-persistent.yaml

@dongboyan77 Could you provide oc commands how to reproduce this please?

create template oc create -f https://github.com/sclorg/mongodb-container/blob/master/examples/petset/mongodb-petset-persistent.yaml Then modify template instead of your expected image oc edit template create app oc new-app mongodb-petset-replication

@dongboyan77 Thanks. And then?

oc scale --replicas=0 statefulsets/mongodb
oc scale --replicas=3 statefulsets/mongodb

is working for me. So I am not possible to reproduce this issue with current docker.io/mongodb-XY-centos7 :-( Could you provide more information please?

delete all pods oc delete pod --all

Thanks.

After this I can also see "Detected unclean shutdown - /var/lib/mongodb/data/mongod.lock is not empty." error message in logs. But mongod recovers automatically so deployment does not fail. (so after oc delete pod --all 3 pods starts again) @dongboyan77 Does oc delete pod --all fail every time?

It seems that even if all cleanup() is run, mongod.lock isn't empty. Also upstream suggests to resync member when recover is needed. So we should think about "more intelligent" way of starting run-mongod-replication when data directory isn't empty.

mongodb container is not maintained any more in this org. closing.