Rocket.Chat
Rocket.Chat copied to clipboard
RocketChat
Failed Upgrade From 4.7.2 to 4.8.1
Description:
I run RocketChat on CentOS 7 and currently running version 4.7.2. When upgrading to 4.8.1 following the upgrade steps, my RocketChat instance fails to start up throwing the following error:
Jun 28 14:51:50 core rocketchat: Error creating indexes for integration_history MongoError: Index with name: _updatedAt_1 already exists with different options
Jun 28 14:51:50 core rocketchat: at MessageStream.messageHandler (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:272:20)
Jun 28 14:51:50 core rocketchat: at MessageStream.emit (events.js:400:28)
Jun 28 14:51:50 core rocketchat: at MessageStream.emit (domain.js:475:12)
Jun 28 14:51:50 core rocketchat: at processIncomingData (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:144:12)
Jun 28 14:51:50 core rocketchat: at MessageStream._write (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:42:5)
Jun 28 14:51:50 core rocketchat: at writeOrBuffer (internal/streams/writable.js:358:12)
Jun 28 14:51:50 core rocketchat: at MessageStream.Writable.write (internal/streams/writable.js:303:10)
Jun 28 14:51:50 core rocketchat: at Socket.ondata (internal/streams/readable.js:731:22)
Jun 28 14:51:50 core rocketchat: at Socket.emit (events.js:400:28)
Jun 28 14:51:50 core rocketchat: at Socket.emit (domain.js:475:12)
Jun 28 14:51:50 core rocketchat: at addChunk (internal/streams/readable.js:293:12)
Jun 28 14:51:50 core rocketchat: at readableAddChunk (internal/streams/readable.js:267:9)
Jun 28 14:51:50 core rocketchat: at Socket.Readable.push (internal/streams/readable.js:206:10)
Jun 28 14:51:50 core rocketchat: at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
Jun 28 14:51:50 core rocketchat: at TCP.callbackTrampoline (internal/async_hooks.js:130:17) {
Jun 28 14:51:50 core rocketchat: operationTime: Timestamp { _bsontype: 'Timestamp', low_: 1, high_: 1656420702 },
Jun 28 14:51:50 core rocketchat: ok: 0,
Jun 28 14:51:50 core rocketchat: code: 85,
Jun 28 14:51:50 core rocketchat: codeName: 'IndexOptionsConflict',
Jun 28 14:51:50 core rocketchat: '$clusterTime': {
Jun 28 14:51:50 core rocketchat: clusterTime: Timestamp { _bsontype: 'Timestamp', low_: 1, high_: 1656420702 },
Jun 28 14:51:50 core rocketchat: signature: { hash: [Binary], keyId: 0 }
Jun 28 14:51:50 core rocketchat: }
Jun 28 14:51:50 core rocketchat: }
- Version of Rocket.Chat Server: 4.7.2
- Operating System: CentOS 7
- NodeJS Version: v14.18.3
- MongoDB Version: 4.2.21 / wiredTiger (oplog Enabled)
- NPM: 6.14.15
This and one other have happened couple of times with the latest versions of RC.
integration_history
nps_vote (not seen as often as integration_history)
I did a comparison of the existing index options using
db.rocketchat_integration_history.getIndexes()
db.rocketchat_nps_vote.getIndexes()
and then dropped the indexes using
db.rocketchat_integration_history.dropIndexes()
db.rocketchat_nps_vote.dropIndexes()
After the next start of RC node the indexes were just created again, with exactly same options as before.
I guess the devs are just ensuring that these indexes get created during the start if they weren't already.
(I actually dropped the indexes in between the updates too to see whether they'd change during the schema upgrade, but they remained the same.)
Unfortunately that dropping wasn't the solution for my problem. Still the server is in a restart loop.
My RC is running in Snapd. I stopped the RC-Process for that and restarted it. Do I have to do something else?
What I have tried so far:
mongo
use rocketchat
db.rocketchat_nps_vote.dropIndexes()
db.users.dropIndexes()
db.rocketchat_room.dropIndexes()
db.rocketchat_message.dropIndexes()
db.rocketchat_integration_history.dropIndexes()
db.rocketchat_apps_logs.dropIndexes()
Nothing, still end up with the same index error:
Jun 28 14:51:50 core rocketchat: Error creating indexes for integration_history MongoError: Index with name: _updatedAt_1 already exists with different options
I tried upgrading now to 4.8.2, and it seems to throw no index errors. However, now there's a segmentation fault:
Jul 22 16:43:14 core systemd: rocketchat.service: main process exited, code=killed, status=11/SEGV
Jul 22 16:43:14 core systemd: Unit rocketchat.service entered failed state.
Jul 22 16:43:14 core systemd: rocketchat.service failed.
[root@core ~]#
[root@core server]# node /opt/Rocket.Chat/main.js
/opt/Rocket.Chat/programs/server/node_modules/fibers/fibers.js:92
return fn.apply(this, arguments);
^
Error: Must pass options.rootUrl or set ROOT_URL in the server environment
at Object.Meteor.absoluteUrl (packages/meteor.js:1412:11)
at runWebAppServer (packages/webapp/webapp_server.js:996:48)
at packages/webapp/webapp_server.js:1494:1
at module (packages/webapp/webapp_server.js:1494:16)
at fileEvaluate (packages/modules-runtime.js:336:7)
at Module.require (packages/modules-runtime.js:238:14)
at require (packages/modules-runtime.js:258:21)
at /opt/Rocket.Chat/programs/server/packages/webapp.js:1963:15
at /opt/Rocket.Chat/programs/server/packages/webapp.js:1972:3
at /opt/Rocket.Chat/programs/server/boot.js:401:38
at Array.forEach (<anonymous>)
at /opt/Rocket.Chat/programs/server/boot.js:226:21
at /opt/Rocket.Chat/programs/server/boot.js:464:7
at Function.run (/opt/Rocket.Chat/programs/server/profile.js:280:14)
at /opt/Rocket.Chat/programs/server/boot.js:463:13
[root@core server]#
[root@core ~]# /usr/local/bin/node /opt/Rocket.Chat/main.js
Segmentation fault
[root@core ~]#
Same issue here on Centos 7 , this is preventing me to update (still on v.4.7.2) :
Aug 06 10:52:37 chat systemd[1]: rocketchat.service: main process exited, code=killed, status=11/SEGV
Aug 06 10:52:37 chat systemd[1]: Unit rocketchat.service entered failed state.
Aug 06 10:52:37 chat systemd[1]: rocketchat.service failed.
Aug 06 10:52:38 chat systemd[1]: rocketchat.service holdoff time over, scheduling restart.
Aug 06 10:52:38 chat systemd[1]: Stopped The Rocket.Chat server.
If I try to run the application manually after setting the ROOT_URL env variable, I get :
# node /opt/Rocket.Chat/main.js
/opt/Rocket.Chat/programs/server/node_modules/fibers/future.js:280
throw(ex);
^
TypeError: Cannot read property 'includes' of undefined
at module (packages/rocketchat:mongo-config/server/index.js:20:29)
at fileEvaluate (packages/modules-runtime.js:336:7)
at Module.require (packages/modules-runtime.js:238:14)
at require (packages/modules-runtime.js:258:21)
at /opt/Rocket.Chat/programs/server/packages/rocketchat_mongo-config.js:116:15
at /opt/Rocket.Chat/programs/server/packages/rocketchat_mongo-config.js:121:3
at /opt/Rocket.Chat/programs/server/boot.js:401:38
at Array.forEach (<anonymous>)
at /opt/Rocket.Chat/programs/server/boot.js:226:21
at /opt/Rocket.Chat/programs/server/boot.js:464:7
at Function.run (/opt/Rocket.Chat/programs/server/profile.js:280:14)
at /opt/Rocket.Chat/programs/server/boot.js:463:13
Tried Upgrading from 4.7.4 to 5.02 and still getting errors on CentOS 7:
Aug 10 20:40:14 core rocketchat: Some indexes for collection 'rocketchat_nps_vote' could not be created:
Aug 10 20:40:14 core rocketchat: Index with name: npsId_1_identifier_1 already exists with different options
Aug 10 20:40:17 core systemd: rocketchat.service: main process exited, code=killed, status=11/SEGV
Aug 10 20:40:17 core systemd: Unit rocketchat.service entered failed state.
Aug 10 20:40:17 core systemd: rocketchat.service failed.
Had a the same issue trying to upgrade from 4.7.2 to 5.02 last week on CentOS 7. Had to restore a snapshot. Any updates on this?
Also the issue I am facing, looking forward to a solution. I am seeing:
# node /opt/Rocket.Chat/main.js /opt/Rocket.Chat/programs/server/node_modules/fibers/future.js:280 throw(ex); ^
TypeError: Cannot read property 'includes' of undefined
at module (packages/rocketchat:mongo-config/server/index.js:21:29)
at fileEvaluate (packages/modules-runtime.js:336:7)
at Module.require (packages/modules-runtime.js:238:14)
at require (packages/modules-runtime.js:258:21)
at /opt/Rocket.Chat/programs/server/packages/rocketchat_mongo-config.js:117:15
at /opt/Rocket.Chat/programs/server/packages/rocketchat_mongo-config.js:122:3
at /opt/Rocket.Chat/programs/server/boot.js:401:38
at Array.forEach (
I put some logging in that file and it returned:
# node /opt/Rocket.Chat/main.js Callback Length: 1 Value: <meta charset="utf-8" /> <meta http-equiv="content-type" content="text/html; charset=utf-8" /> <meta http-equiv="expires" content="-1" /> <meta http-equiv="X-UA-Compatible" content="IE=edge" /> <meta name="fragment" content="!" /> <meta name="distribution" content="global" /> <meta name="rating" content="general" /> <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no" /> <meta name="mobile-web-app-capable" content="yes" /> <meta name="apple-mobile-web-app-capable" content="yes" /> <meta name="msapplication-TileImage" content="assets/tile_144.png" /> <meta name="msapplication-config" content="images/browserconfig.xml" /> <meta property="og:image" content="assets/favicon_512.png" /> <meta property="twitter:image" content="assets/favicon_512.png" /> <link rel="manifest" href="images/manifest.json" /> <link rel="chrome-webstore-item" href="https://chrome.google.com/webstore/detail/nocfbnnmjnndkbipkabodnheejiegccf" /> <link rel="mask-icon" href="assets/safari_pinned.svg" color="#04436a"> <link rel="apple-touch-icon" sizes="180x180" href="assets/touchicon_180.png" /> <link rel="apple-touch-icon-precomposed" href="assets/touchicon_180_pre.png" /> Ref[0]: [Function: cb] Ref[1]: undefined Callback Length: 1 Value: <meta charset="utf-8" /> <meta http-equiv="content-type" content="text/html; charset=utf-8" /> <meta http-equiv="expires" content="-1" /> <meta http-equiv="X-UA-Compatible" content="IE=edge" /> <meta name="fragment" content="!" /> <meta name="distribution" content="global" /> <meta name="rating" content="general" /> <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no" /> <meta name="mobile-web-app-capable" content="yes" /> <meta name="apple-mobile-web-app-capable" content="yes" /> <meta name="msapplication-TileImage" content="assets/tile_144.png" /> <meta name="msapplication-config" content="images/browserconfig.xml" /> <meta property="og:image" content="assets/favicon_512.png" /> <meta property="twitter:image" content="assets/favicon_512.png" /> <link rel="manifest" href="images/manifest.json" /> <link rel="chrome-webstore-item" href="https://chrome.google.com/webstore/detail/nocfbnnmjnndkbipkabodnheejiegccf" /> <link rel="mask-icon" href="assets/safari_pinned.svg" color="#04436a"> <link rel="apple-touch-icon" sizes="180x180" href="assets/touchicon_180.png" /> <link rel="apple-touch-icon-precomposed" href="assets/touchicon_180_pre.png" /> Ref[0]: [Function: cb] Ref[1]: undefined Callback Length: 1 Value: undefined Ref[0]: [Function: cb] Ref[1]: undefined /opt/Rocket.Chat/programs/server/node_modules/fibers/future.js:284 throw(ex); ^
TypeError: Cannot read property 'includes' of undefined
at module (packages/rocketchat:mongo-config/server/index.js:21:29)
at fileEvaluate (packages/modules-runtime.js:336:7)
at Module.require (packages/modules-runtime.js:238:14)
at require (packages/modules-runtime.js:258:21)
at /opt/Rocket.Chat/programs/server/packages/rocketchat_mongo-config.js:117:15
at /opt/Rocket.Chat/programs/server/packages/rocketchat_mongo-config.js:122:3
at /opt/Rocket.Chat/programs/server/boot.js:401:38
at Array.forEach (
What I have tried so far:
mongo use rocketchat db.rocketchat_nps_vote.dropIndexes() db.users.dropIndexes() db.rocketchat_room.dropIndexes() db.rocketchat_message.dropIndexes() db.rocketchat_integration_history.dropIndexes() db.rocketchat_apps_logs.dropIndexes()Nothing, still end up with the same index error:
Jun 28 14:51:50 core rocketchat: Error creating indexes for integration_history MongoError: Index with name: _updatedAt_1 already exists with different options
I had those issues trying to upgrade from 4.7x lately. After the latest security fix wasn't released to 4.7.x I had a look at it again. After dropping the above mentioned indexes the upgrade 4.7.5 -> 4.8.5 worked like a charm. I'll stick on the 4.8.x for a little longer and then migrate to 5.0.x at some point. Running docker container (the none alpine one)
Just for future reference: The index issue is not impactful on later versions of Rocket.Chat. The instance starts successfully and everything will work normally. To remove the index warnings you have cleared the index errors manually for now.
