Database upgrade script now breaks upgrading a container

[adrielldagasuan]

We have a build process where as part of our application upgrade, we also upgrade the database (same version), using an updated oracle image. We build the image as part of the build process.

We noticed that after the upgrade, our applications cannot connect to the presumably upgraded database and that's because the data has been wiped out. Possibly due to this commit https://github.com/oracle/docker-images/commit/67bff664d5b495d34d471dec81af92daab051b25

The previous behavior when we tried connecting a newer image to old data was that it would result in an ORA39700 error, which we capture and thus we then proceed to upgrade it. But now it just proceeds to proceed with a new deployment.

Is this something that was supposed to break? Or are we doing something non-standard?

[mahipalm0911]

We now have a checkpoint file , which if exists denotes database has been created successfully . We didn't had this earlier . So before upgrading from older image to a newer image, manually please create the checkpoint file using $ touch $ORACLE_BASE/oradata/.${ORACLE_SID}.created

[adrielldagasuan]

Thanks for the reply. I'm trying out the proposed workaround.

[adrielldagasuan]

Hi,

Manually creating the checkpoint flag now proceeds to run the upgrade. However, we are hitting an issue we first discovered after this commit https://github.com/oracle/docker-images/commit/881332135c6026b3d7bbf8f198c0e629ac6b2498.

We are running the following:

+ cd /opt/oracle/product/19c/dbhome_1/rdbms/admin
+ /opt/oracle/product/19c/dbhome_1/perl/bin/perl catctl.pl catupgrd.sql

which appears to complete without any errors. But as as soon as we start the database, we get a DATABASE SETUP WAS NOT SUCCESSFUL! message. The only thing I can find in the logs is this:

Oracle Database 19c Clusterware Release 19.0.0.0.0 - Production
Version 19.3.0.0.0 Copyright 1996, 2019 Oracle. All rights reserved.
[ default][12094528]u_set_comp_error: comptype '103' : error '29'
2021-07-28 04:48:46.589: [GIPCXCPT][12094528] gipcInternalConnectSync: failed sync request, addr 0xa2ec60 [000000000000001d] { gipcAddress : name 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=CRSD_UI_SOCKET)(GIPCID=00000000-00000000-0))', objFlags 0x0, addrFlags 0x4 }, ret gipcretConnectionRefused (29)
2021-07-28 04:48:46.589: [GIPCXCPT][12094528] gipcConnectSyncF [clscrsconGipcConnect : clscrscon.c : 686]: EXCEPTION[ ret gipcretConnectionRefused (29) ]  failed sync connect endp 0xa8a100 [000000000000001a] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=)(GIPCID=00000000-00000000-0))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=CRSD_UI_SOCKET)(GIPCID=00000000-00000000-0))', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj 0xa29e00, sendp 0xa2f6e0 status 13flags 0xa108071a, flags-2 0x0, usrFlags 0x0 }, addr 0xa2ec60 [000000000000001d] { gipcAddress : name 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=CRSD_UI_SOCKET)(GIPCID=00000000-00000000-0))', objFlags 0x0, addrFlags 0x4 }, flags 0x0
[ default][12094528]u_set_comp_error: comptype '103' : error '29'
2021-07-28 04:48:46.591: [GIPCXCPT][12094528] gipcInternalConnectSync: failed sync request, addr 0xa2ff00 [000000000000002b] { gipcAddress : name 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=CRSD_UI_SOCKET)(GIPCID=00000000-00000000-0))', objFlags 0x0, addrFlags 0x4 }, ret gipcretConnectionRefused (29)
2021-07-28 04:48:46.591: [GIPCXCPT][12094528] gipcConnectSyncF [clscrsconGipcConnect : clscrscon.c : 686]: EXCEPTION[ ret gipcretConnectionRefused (29) ]  failed sync connect endp 0xa2f140 [0000000000000028] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=)(GIPCID=00000000-00000000-0))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=CRSD_UI_SOCKET)(GIPCID=00000000-00000000-0))', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj 0xa929a0, sendp 0xa898b0 status 13flags 0xa108071a, flags-2 0x0, usrFlags 0x0 }, addr 0xa2ff00 [000000000000002b] { gipcAddress : name 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=CRSD_UI_SOCKET)(GIPCID=00000000-00000000-0))', objFlags 0x0, addrFlags 0x4 }, flags 0x0

The log file is /opt/oracle/product/19c/dbhome_1/log/aac691c98dbb/client/tnslsnr_32.log). Is there anything from that commit that might result in the error above?

[mahipalm0911]

This issue is not because of commit 67bff66 Its just a flag to either remove datafiles , create new database or Start database with existing datafiles

[adrielldagasuan]

The issue with commit https://github.com/oracle/docker-images/commit/67bff664d5b495d34d471dec81af92daab051b25 is fixed, with that .created flag. Apologies for the confusion. I am now talking about an older commit: https://github.com/oracle/docker-images/commit/881332135c6026b3d7bbf8f198c0e629ac6b2498. So now we can proceed with the upgrade, then fails starting up after the upgrade.

[mahipalm0911]

It doesn't looks like 8813321 commit is causing the above error

[adrielldagasuan]

Okay. Can you point us to which log then would contain the relevant error? Or if there's something in commit https://github.com/oracle/docker-images/commit/881332135c6026b3d7bbf8f198c0e629ac6b2498 that would potentially cause any issues?

[abhisbyk]

Hi @adrielldagasuan, is this still an issue ?