Upgrade 5.8.0 to 5.9.0 - got error while upgrading accounts: RangeError: Maximum call stack size exceeded

[snorwin]

I am aware that the latest build is not a stable version, however I just want to give you a heads up.

Environment info

• NooBaa 5.9.0-f0850fe
• Platform: Kubernetes v1.21.1+6438632 | OpenShift 4.8.17

Actual behavior

noobaa-core pod is crash looping due to a failed upgrade from 5.8.0-e060925 to 5.9.0-f0850fe.

Dec-10 12:31:35.251 [Upgrade/21]    [L0] UPGRADE:: system store loaded
Dec-10 12:31:35.252 [Upgrade/21]    [L0] UPGRADE:: container version is 5.9.0-f0850fe and server version is 5.8.0-e060925. will upgrade
Dec-10 12:31:35.253 [Upgrade/21]    [L0] UPGRADE:: found the following versions with upgrade scripts which are newer than server version (5.8.0-e060925): [ '5.9.0' ]
Dec-10 12:31:35.253 [Upgrade/21]    [L0] UPGRADE:: running upgrade script /root/node_modules/noobaa-core/src/upgrade/upgrade_scripts/5.9.0/upgrade_accounts_ns_resources.js: Update accounts default_pool to default_resource and namespace bucket structure
Dec-10 12:31:35.253 [Upgrade/21]    [L0] UPGRADE:: starting upgrade accounts...
Dec-10 12:31:35.254 [Upgrade/21]    [L0] UPGRADE:: replacing account default_pool by default_resource: 61adf41c1a3dd70029940ebf, 61adf41d1a3dd70029940ecf, 61ae31101a3dd70029940ef2, 61b1bdd51a3dd70029940f18
Dec-10 12:31:35.254 [Upgrade/21]    [L0] core.server.system_services.system_store:: SystemStore.make_changes: {
  update: {
    accounts: [ { _id: 61adf41c1a3dd70029940ebf, '$set': { default_resource: { _id: 61adf4761a3dd70029940ed6, system: [Object], name: 'noobaa-default-backing-store', resource_type: 'HOSTS', pool_node_type: 'BLOCK_STORE_FS', storage_stats: [Object], hosts_pool_info: [Object] } }, '$unset': { default_pool: true } }, { _id: 61adf41d1a3dd70029940ecf, '$set': { default_resource: { _id: 61adf41c1a3dd70029940ec2, system: [Object], name: 'system-internal-storage-pool-61adf41c1a3dd70029940ec0', resource_type: 'INTERNAL', pool_node_type: 'BLOCK_STORE_MONGO', storage_stats: [Object], mongo_pool_info: [Object] } }, '$unset': { default_pool: true } }, { _id: 61ae31101a3dd70029940ef2, '$set': { default_resource: { _id: 61ae30c11a3dd70029940ee8, system: [Object], name: 's3-nas740', resource_type: 'CLOUD', pool_node_type: 'BLOCK_STORE_S3', storage_stats: [Object], cloud_pool_info: [Object] } }, '$unset': { default_pool: true } }, { _id: 61b1bdd51a3dd70029940f18, '$set': { default_resource: { _id: 61ae30c11a3dd70029940ee8, system: [Object], name: 's3-nas740', resource_type: 'CLOUD', pool_node_type: 'BLOCK_STORE_S3', storage_stats: [Object], cloud_pool_info: [Object] } }, '$unset': { default_pool: true } } ]
  }
}
Dec-10 12:31:35.298 [Upgrade/21] [ERROR] UPGRADE:: got error while upgrading accounts: RangeError: Maximum call stack size exceeded 
    at calculateElement (/root/node_modules/noobaa-core/node_modules/bson/lib/bson/parser/calculate_size.js:99:51)
    at calculateObjectSize (/root/node_modules/noobaa-core/node_modules/bson/lib/bson/parser/calculate_size.js:48:22)
    at calculateElement (/root/node_modules/noobaa-core/node_modules/bson/lib/bson/parser/calculate_size.js:198:11)
    at calculateObjectSize (/root/node_modules/noobaa-core/node_modules/bson/lib/bson/parser/calculate_size.js:48:22)
    at calculateElement (/root/node_modules/noobaa-core/node_modules/bson/lib/bson/parser/calculate_size.js:198:11)
    at calculateObjectSize (/root/node_modules/noobaa-core/node_modules/bson/lib/bson/parser/calculate_size.js:48:22)
    at calculateElement (/root/node_modules/noobaa-core/node_modules/bson/lib/bson/parser/calculate_size.js:198:11)
    at calculateObjectSize (/root/node_modules/noobaa-core/node_modules/bson/lib/bson/parser/calculate_size.js:48:22)
    at calculateElement (/root/node_modules/noobaa-core/node_modules/bson/lib/bson/parser/calculate_size.js:198:11)
    at calculateObjectSize (/root/node_modules/noobaa-core/node_modules/bson/lib/bson/parser/calculate_size.js:48:22)

Expected behavior

NooBaa should be successfully upgraded.

Steps to reproduce

1. deploy a NooBaa using the noobaa-operator and set version to 5.8.0
2. upgrade NooBaa to 5.9.0 (e.g. using the image version master-20211209)

[guymguym]

@jackyalbo Any idea?

[liranmauda]

@snorwin I haven't looked at it but I just want to point 2 things:

1. master-20211209 is master which should point to 4.10 now. if you want something stable you can use the 4.9 branches.
2. Maximum call stack size exceeded is impaling that we have a recursive call for a function without promoting something, which leads to an endless recurse.

@jackyalbo or me Will have a look

[snorwin]

Hi @liranmauda, thanks for pointing out the release branches. I also tired to migrate NooBaa from 5.8.0 to 5.9-20211115, however the error still occurs. It is not urgent, all the features we rely on are available in 5.8.0.

[liranmauda]

@snorwin Can you attach the logs? and is it possible for you to change the debug level? (I am assuming that this is not a production, but a test system).

[nimrod-becker]

insufficient data