docker-images
docker-images copied to clipboard
oracle
database enterprise:21.3.0.0 has issue with Kubernetes StorageClass NFS
Dear Oracle dev team,
I'm new to oracle docker and I'm trying to deploy an oracle enterprise database 21.3 on our kubernetes and ran into the following issue StorageClass NFS. I successfully ran with local storage, then I change to StorageClass NFS as below StatefulSet:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: haiph-oracle-db
namespace: abcd
spec:
replicas: 1
serviceName: haiph-oracle-db-svc
selector:
matchLabels:
app: haiph-oracle-db
template:
metadata:
labels:
app: haiph-oracle-db
spec:
initContainers:
- name: fix-permissions
image: busybox
# when NFS volume mounted on container, I saw user root is owner of /opt/oracle/oradata so I change owner to user oracle:oinstall with UID 54321 to give all permissions
command: ["sh", "-c", "chown -R 54321:54321 /opt/oracle/oradata && chmod -R 777 /opt/oracle/oradata"]
volumeMounts:
- mountPath: /opt/oracle/oradata
name: nfs-oracle-data
containers:
- name: haiph-oracle-db
image: container-registry.oracle.com/database/enterprise:21.3.0.0
ports:
- containerPort: 1521
env:
- name: ORACLE_SID
value: "testdb"
- name: ORACLE_PDB
value: "testdata"
- name: ORACLE_PWD
value: "123456"
volumeMounts:
- mountPath: /opt/oracle/oradata
name: nfs-oracle-data
imagePullSecrets:
- name: oracle-registry-secret
volumeClaimTemplates:
- metadata:
name: nfs-oracle-data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "nfs-nas01"
resources:
requests:
storage: 20Gi
This is permissions of /opt/oracle/oradata/
bash-4.2$ cd /opt/oracle/oradata/
bash-4.2$ ls -alh
total 35K
drwxrwxrwx 4 oracle oinstall 6 Dec 23 08:25 .
drwxr-xr-x 1 oracle dba 4.0K Dec 23 08:25 ..
-rwxrwxrwx 1 oracle oinstall 0 Dec 23 08:25 .TESTDB.create_lck
-rwxrwxrwx 1 oracle oinstall 0 Dec 23 08:41 .TESTDB.exist_lck
drwxrwxrwx 4 oracle oinstall 4 Dec 23 08:25 TESTDB
drwxrwxrwx 3 oracle oinstall 3 Dec 23 08:25 dbconfig
This is output logs from oracle container:
[2024:12:23 08:25:13]: Acquiring lock .TESTDB.create_lck with heartbeat 30 secs
[2024:12:23 08:25:13]: Lock acquired
[2024:12:23 08:25:13]: Starting heartbeat
[2024:12:23 08:25:13]: Lock held .TESTDB.create_lck
ORACLE EDITION: ENTERPRISE
LSNRCTL for Linux: Version 21.0.0.0.0 - Production on 23-DEC-2024 08:25:13
Copyright (c) 1991, 2021, Oracle. All rights reserved.
Starting /opt/oracle/product/21c/dbhome_1/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 21.0.0.0.0 - Production
System parameter file is /opt/oracle/homes/OraDB21Home1/network/admin/listener.ora
Log messages written to /opt/oracle/diag/tnslsnr/haiph-oracle-db-0/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 21.0.0.0.0 - Production
Start Date 23-DEC-2024 08:25:13
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /opt/oracle/homes/OraDB21Home1/network/admin/listener.ora
Listener Log File /opt/oracle/diag/tnslsnr/haiph-oracle-db-0/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
The listener supports no services
The command completed successfully
[WARNING] [DBT-06208] The 'SYS' password entered does not conform to the Oracle recommended standards.
CAUSE:
a. Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9].
b.The password entered is a keyword that Oracle does not recommend to be used as password
ACTION: Specify a strong password. If required refer Oracle documentation for guidelines.
[WARNING] [DBT-06208] The 'SYSTEM' password entered does not conform to the Oracle recommended standards.
CAUSE:
a. Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9].
b.The password entered is a keyword that Oracle does not recommend to be used as password
ACTION: Specify a strong password. If required refer Oracle documentation for guidelines.
[WARNING] [DBT-06208] The 'PDBADMIN' password entered does not conform to the Oracle recommended standards.
CAUSE:
a. Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9].
b.The password entered is a keyword that Oracle does not recommend to be used as password
ACTION: Specify a strong password. If required refer Oracle documentation for guidelines.
Prepare for db operation
8% complete
Copying database files
31% complete
100% complete
[FATAL] Recovery Manager failed to restore datafiles. Refer logs for details.
8% complete
0% complete
Look at the log file "/opt/oracle/cfgtoollogs/dbca/TESTDB/TESTDB.log" for further details.
[ 2024-12-23 08:25:20.281 UTC ] [WARNING] [DBT-06208] The 'SYS' password entered does not conform to the Oracle recommended standards.
[ 2024-12-23 08:25:20.282 UTC ] [WARNING] [DBT-06208] The 'SYSTEM' password entered does not conform to the Oracle recommended standards.
[ 2024-12-23 08:25:20.282 UTC ] [WARNING] [DBT-06208] The 'PDBADMIN' password entered does not conform to the Oracle recommended standards.
[ 2024-12-23 08:25:24.247 UTC ] Prepare for db operation
DBCA_PROGRESS : 8%
[ 2024-12-23 08:25:24.344 UTC ] Copying database files
DBCA_PROGRESS : 31%
DBCA_PROGRESS : 100%
[ 2024-12-23 08:25:37.082 UTC ] [FATAL] Recovery Manager failed to restore datafiles. Refer logs for details.
DBCA_PROGRESS : 8%
DBCA_PROGRESS : 0%
SQL*Plus: Release 21.0.0.0.0 - Production on Mon Dec 23 08:25:39 2024
Version 21.3.0.0.0
Copyright (c) 1982, 2021, Oracle. All rights reserved.
Connected to an idle instance.
SQL> ALTER SYSTEM SET control_files='/opt/oracle/oradata/TESTDB/control01.ctl' scope=spfile
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> ALTER SYSTEM SET local_listener=''
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> ALTER PLUGGABLE DATABASE TESTDATA SAVE STATE
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> BEGIN DBMS_XDB_CONFIG.SETGLOBALPORTENABLED (TRUE); END;
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> SQL> ALTER SESSION SET "_oracle_script" = true
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> CREATE USER OPS$oracle IDENTIFIED EXTERNALLY
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> GRANT CREATE SESSION TO OPS$oracle
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> GRANT SELECT ON sys.v_$pdbs TO OPS$oracle
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> GRANT SELECT ON sys.v_$database TO OPS$oracle
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> ALTER USER OPS$oracle SET container_data=all for sys.v_$pdbs container = current
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> SQL> Disconnected
ORACLE_HOME = [/home/oracle] ? ORACLE_BASE environment variable is not being set since this
information is not available for the current user ID .
You can set ORACLE_BASE manually if it is required.
Resetting ORACLE_BASE to its previous value or ORACLE_HOME
The Oracle base remains unchanged with value /opt/oracle
/opt/oracle/checkDBStatus.sh: line 26: sqlplus: command not found
mv: cannot stat '/opt/oracle/dbs/spfileTESTDB.ora': No such file or directory
mv: cannot stat '/opt/oracle/dbs/orapwTESTDB': No such file or directory
Executing user defined scripts
/opt/oracle/runUserScripts.sh: running /opt/oracle/scripts/extensions/setup/swapLocks.sh
[2024:12:23 08:25:39]: Releasing lock .TESTDB.create_lck
[2024:12:23 08:25:39]: Lock released .TESTDB.create_lck
[2024:12:23 08:25:39]: Acquiring lock .TESTDB.exist_lck with heartbeat 30 secs
[2024:12:23 08:25:39]: Lock acquired
[2024:12:23 08:25:39]: Starting heartbeat
[2024:12:23 08:25:39]: Lock held .TESTDB.exist_lck
DONE: Executing user defined scripts
ORACLE_HOME = [/home/oracle] ? ORACLE_BASE environment variable is not being set since this
information is not available for the current user ID .
You can set ORACLE_BASE manually if it is required.
Resetting ORACLE_BASE to its previous value or ORACLE_HOME
The Oracle base remains unchanged with value /opt/oracle
/opt/oracle/checkDBStatus.sh: line 26: sqlplus: command not found
#####################################
########### E R R O R ###############
DATABASE SETUP WAS NOT SUCCESSFUL!
Please check output for further info!
########### E R R O R ###############
#####################################
The following output is now a tail of the alert.log:
2024-12-23T08:25:37.213010+00:00
Shutting down ORACLE instance (abort) (OS id: 767)
Shutdown is initiated by sqlplus@haiph-oracle-db-0 (TNS V1-V3).
License high water mark = 4
2024-12-23T08:25:37.213806+00:00
USER (ospid: 767): terminating the instance
2024-12-23T08:25:38.219579+00:00
Instance terminated by USER, pid = 767
2024-12-23T08:25:38.851440+00:00
Instance shutdown complete (OS id: 767)
This folder oradata from running oracle container with local storage. I saw that missing file control01.ctl as above log
bash-4.2$ cd /opt/oracle/oradata/
bash-4.2$ ls -alh
total 2.7G
drwxrwxr-x 4 oracle oinstall 4.0K Dec 23 04:05 .
drwxrwxr-x 4 oracle oinstall 4.0K Dec 23 04:46 ..
drwxrwxr-x 2 oracle oinstall 4.0K Dec 23 04:14 TESTDB
-rwxrwxr-x 1 oracle oinstall 18M Dec 23 08:52 control01.ctl
-rwxrwxr-x 1 oracle oinstall 18M Dec 23 04:30 control02.ctl
drwxrwxr-x 2 oracle oinstall 4.0K Dec 23 04:06 pdbseed
-rwxrwxr-x 1 oracle oinstall 201M Dec 23 08:51 redo01.log
-rwxrwxr-x 1 oracle oinstall 201M Dec 23 04:46 redo02.log
-rwxrwxr-x 1 oracle oinstall 201M Dec 23 04:46 redo03.log
-rwxrwxr-x 1 oracle oinstall 591M Dec 23 08:51 sysaux01.dbf
-rwxrwxr-x 1 oracle oinstall 1.3G Dec 23 08:38 system01.dbf
-rwxrwxr-x 1 oracle oinstall 238M Dec 23 04:09 temp01.dbf
-rwxrwxr-x 1 oracle oinstall 121M Dec 23 08:51 undotbs01.dbf
-rwxrwxr-x 1 oracle oinstall 5.1M Dec 23 04:46 users01.dbf
Please help to advise my mistake, thank you for your support.
The failure happens before the controlfile error message during the datafile restore:
[ 2024-12-23 08:25:24.247 UTC ] Prepare for db operation
DBCA_PROGRESS : 8%
[ 2024-12-23 08:25:24.344 UTC ] Copying database files
DBCA_PROGRESS : 31%
DBCA_PROGRESS : 100%
[ 2024-12-23 08:25:37.082 UTC ] [FATAL] Recovery Manager failed to restore datafiles. Refer logs for details. <<<<<<<<<
DBCA_PROGRESS : 8%
DBCA_PROGRESS : 0%
This error is usually a matter of storage permissions. It looks like you're trying to manipulate the permissions here:
# when NFS volume mounted on container, I saw user root is owner of /opt/oracle/oradata so I change owner to user oracle:oinstall with UID 54321 to give all permissions
command: ["sh", "-c", "chown -R 54321:54321 /opt/oracle/oradata && chmod -R 777 /opt/oracle/oradata"]
Can you try adding a security context instead? Something like this? https://github.com/PureStorage-OpenConnect/Oracle-on-Kubernetes-with-Portworx/blob/c5ffad04e62754acc43804c5f0745a9af9b4bdc5/21c_statefulset_PX.yaml#L16-L28
Dear @oraclesean,
I updated security context as your suggestion and return different error, please help to advise:
StatefulSet:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: haiph-oracle-db
namespace: test
spec:
replicas: 1
serviceName: haiph-oracle-db-svc
selector:
matchLabels:
app: haiph-oracle-db
template:
metadata:
labels:
app: haiph-oracle-db
spec:
securityContext:
fsGroup: 54321
containers:
- name: haiph-oracle-db
image: container-registry.oracle.com/database/enterprise:21.3.0.0
ports:
- containerPort: 1521
env:
- name: ORACLE_SID
value: "TESTDB"
- name: ORACLE_PDB
value: "testpdb"
- name: ORACLE_PWD
value: "Abcd1234"
volumeMounts:
- mountPath: /opt/oracle/oradata
name: nfs-oracle-data
imagePullSecrets:
- name: oracle-registry-secret
volumeClaimTemplates:
- metadata:
name: nfs-oracle-data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "nfs-nas01"
resources:
requests:
storage: 20Gi
Container logs:
[2024:12:25 08:15:03]: Acquiring lock .TESTDB.create_lck with heartbeat 30 secs
[2024:12:25 08:15:03]: Lock acquired
[2024:12:25 08:15:03]: Starting heartbeat
[2024:12:25 08:15:03]: Lock held .TESTDB.create_lck
ORACLE EDITION: ENTERPRISE
LSNRCTL for Linux: Version 21.0.0.0.0 - Production on 25-DEC-2024 08:15:03
Copyright (c) 1991, 2021, Oracle. All rights reserved.
Starting /opt/oracle/product/21c/dbhome_1/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 21.0.0.0.0 - Production
System parameter file is /opt/oracle/homes/OraDB21Home1/network/admin/listener.ora
Log messages written to /opt/oracle/diag/tnslsnr/haiph-oracle-db-0/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 21.0.0.0.0 - Production
Start Date 25-DEC-2024 08:15:03
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /opt/oracle/homes/OraDB21Home1/network/admin/listener.ora
Listener Log File /opt/oracle/diag/tnslsnr/haiph-oracle-db-0/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
The listener supports no services
The command completed successfully
Prepare for db operation
8% complete
Copying database files
31% complete
100% complete
[FATAL] Recovery Manager failed to restore datafiles. Refer logs for details.
8% complete
0% complete
Look at the log file "/opt/oracle/cfgtoollogs/dbca/TESTDB/TESTDB.log" for further details.
[ 2024-12-25 08:15:14.303 UTC ] Prepare for db operation
DBCA_PROGRESS : 8%
[ 2024-12-25 08:15:14.420 UTC ] Copying database files
DBCA_PROGRESS : 31%
DBCA_PROGRESS : 100%
[ 2024-12-25 08:15:28.252 UTC ] [FATAL] Recovery Manager failed to restore datafiles. Refer logs for details.
DBCA_PROGRESS : 8%
DBCA_PROGRESS : 0%
SQL*Plus: Release 21.0.0.0.0 - Production on Wed Dec 25 08:15:30 2024
Version 21.3.0.0.0
Copyright (c) 1982, 2021, Oracle. All rights reserved.
Connected to an idle instance.
SQL> ALTER SYSTEM SET control_files='/opt/oracle/oradata/TESTDB/control01.ctl' scope=spfile
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> ALTER SYSTEM SET local_listener=''
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> ALTER PLUGGABLE DATABASE TESTPDB SAVE STATE
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> BEGIN DBMS_XDB_CONFIG.SETGLOBALPORTENABLED (TRUE); END;
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> SQL> ALTER SESSION SET "_oracle_script" = true
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> CREATE USER OPS$oracle IDENTIFIED EXTERNALLY
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> GRANT CREATE SESSION TO OPS$oracle
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> GRANT SELECT ON sys.v_$pdbs TO OPS$oracle
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> GRANT SELECT ON sys.v_$database TO OPS$oracle
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> ALTER USER OPS$oracle SET container_data=all for sys.v_$pdbs container = current
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> SQL> Disconnected
ORACLE_HOME = [/home/oracle] ? ORACLE_BASE environment variable is not being set since this
information is not available for the current user ID .
You can set ORACLE_BASE manually if it is required.
Resetting ORACLE_BASE to its previous value or ORACLE_HOME
The Oracle base remains unchanged with value /opt/oracle
/opt/oracle/checkDBStatus.sh: line 26: sqlplus: command not found
mv: cannot stat '/opt/oracle/dbs/spfileTESTDB.ora': No such file or directory
mv: cannot stat '/opt/oracle/dbs/orapwTESTDB': No such file or directory
Executing user defined scripts
/opt/oracle/runUserScripts.sh: running /opt/oracle/scripts/extensions/setup/swapLocks.sh
[2024:12:25 08:15:30]: Releasing lock .TESTDB.create_lck
[2024:12:25 08:15:30]: Lock released .TESTDB.create_lck
[2024:12:25 08:15:30]: Acquiring lock .TESTDB.exist_lck with heartbeat 30 secs
[2024:12:25 08:15:30]: Lock acquired
[2024:12:25 08:15:30]: Starting heartbeat
[2024:12:25 08:15:30]: Lock held .TESTDB.exist_lck
DONE: Executing user defined scripts
ORACLE_HOME = [/home/oracle] ? ORACLE_BASE environment variable is not being set since this
information is not available for the current user ID .
You can set ORACLE_BASE manually if it is required.
Resetting ORACLE_BASE to its previous value or ORACLE_HOME
The Oracle base remains unchanged with value /opt/oracle
/opt/oracle/checkDBStatus.sh: line 26: sqlplus: command not found
#####################################
########### E R R O R ###############
DATABASE SETUP WAS NOT SUCCESSFUL!
Please check output for further info!
########### E R R O R ###############
#####################################
The following output is now a tail of the alert.log:
2024-12-25T08:15:28.387430+00:00
Shutting down ORACLE instance (abort) (OS id: 767)
Shutdown is initiated by sqlplus@haiph-oracle-db-0 (TNS V1-V3).
License high water mark = 4
2024-12-25T08:15:28.388301+00:00
USER (ospid: 767): terminating the instance
2024-12-25T08:15:29.394449+00:00
Instance terminated by USER, pid = 767
2024-12-25T08:15:30.075385+00:00
Instance shutdown complete (OS id: 767)
Terminal to container:
bash-4.2$ cd /opt/oracle/dbs
bash-4.2$ ls -alh
total 20K
drwxr-x--- 1 oracle dba 4.0K Dec 25 08:15 .
drwxr-xr-x 1 oracle dba 4.0K Dec 25 08:15 ..
-rw-rw---- 1 oracle dba 1.6K Dec 25 08:15 hc_TESTDB.dat
-rw-r----- 1 oracle dba 24 Dec 25 08:15 lkTESTDB
lrwxrwxrwx 1 oracle oinstall 47 Dec 25 08:15 orapwTESTDB -> /opt/oracle/oradata/dbconfig/TESTDB/orapwTESTDB
lrwxrwxrwx 1 oracle oinstall 52 Dec 25 08:15 spfileTESTDB.ora -> /opt/oracle/oradata/dbconfig/TESTDB/spfileTESTDB.ora