oracle-database-operator
oracle-database-operator copied to clipboard
Published
20 hours ago •
oracle
oracle
ORDS errors are not reported back to the PDB resource
When creating a PDB with e.g. an invalid name, it now gets the following status:
status:
msg: Waiting for PDB to be created
phase: Creating
which I believe is caused by https://github.com/oracle/oracle-database-operator/pull/78.
But even if that is fixed, only the stats code is reported back, not an error message one can use to figure out what is wrong:
https://github.com/oracle/oracle-database-operator/blob/bb344e4dee6ae72e5132f55e15ca4deb55557e71/controllers/database/pdb_controller.go#L546
In a multitenant cluster, users won't have access to the operator logs, and therefore rely on the user-facing resources reporting back the relevant information.
This issue should already be fixed and should be available in the next release. Can you please share your yaml file so that I can verify against the latest version? What invalid name pdb or cdb? Please upload also the backtrace.
Testing wrong cdb name: problem does not reproduce - no crash. Please share your testcase
16:45:10 oracle@mitk01:# diff pdb_create.yaml pdb_issue86.yaml
9c9
< cdbResName: "cdb-dev"
---
> cdbResName: "cdb-dv"
2024-02-15T16:45:10Z INFO controllers.PDB Validating PDB phase for: pdb1 {"validatePhase": {"name":"pdb1","namespace":"pdbnamespace"}, "Action": "CREATE"}
2024-02-15T16:45:10Z INFO controllers.PDB Validation complete {"validatePhase": {"name":"pdb1","namespace":"pdbnamespace"}}
2024-02-15T16:45:10Z INFO controllers.PDB PDB: {"multitenantoperator": {"name":"pdb1","namespace":"pdbnamespace"}, "Name": "pdb1", "Phase": "Creating", "Status": "false"}
2024-02-15T16:45:10Z INFO controllers.PDB Failed to get CRD for CDB {"getCDBResource": {"name":"pdb1","namespace":"pdbnamespace"}, "Name": "cdb-dv", "Namespace": "cdbnamespace", "Error": "CDB.database.oracle.com \"cdb-dv\" not found"}
2024-02-15T16:45:10Z INFO controllers.PDB Reconcile completed {"multitenantoperator": {"name":"pdb1","namespace":"pdbnamespace"}}
Thanks for looking into this. I mean an invalid PDB name, so that the DB server wont create the PDB inside the existing (and working) CDB.
---
apiVersion: database.oracle.com/v1alpha1
kind: PDB
metadata:
name: pdb10
namespace: oracle-database-operator-system
labels:
cdb: cdb-dev
spec:
cdbResName: cdb-dev
cdbName: cdb1
pdbName: _pdb-10
adminName:
secret:
secretName: pdb1-secret
key: sysadmin_user
adminPwd:
secret:
secretName: pdb1-secret
key: sysadmin_pwd
fileNameConversions: "NONE"
totalSize: 1G
tempSize: 100M
action: Create
pdbTlsKey:
secret:
secretName: oracle-cert
key: tls.key
pdbTlsCrt:
secret:
secretName: oracle-cert
key: tls.crt
pdbTlsCat:
secret:
secretName: oracle-cert
key: ca.crt
2024-02-16T07:39:49Z INFO controllers.PDB Found CR for CDB {"getCDBResource": "oracle-database-operator-system/pdb10", "Name": "cdb-dev", "CR Name": "cdb-dev"}
2024-02-16T07:39:49Z INFO Starting workers {"controller": "singleinstancedatabase", "controllerGroup": "database.oracle.com", "controllerKind": "SingleInstanceDatabase", "worker count": 100}
2024-02-16T07:39:49Z INFO controllers.PDB Issuing REST call {"callAPI": "oracle-database-operator-system/pdb10", "URL": "https://cdb-dev-ords:8888/ords/_/db-api/latest/database/pdbs/_pdb-10/status", "Action": "GET"}
2024-02-16T07:39:49Z INFO controllers.PDB Found CR for CDB {"getCDBResource": "oracle-database-operator-system/pdb10", "Name": "cdb-dev", "CR Name": "cdb-dev"}
2024-02-16T07:39:49Z INFO controllers.PDB ORDS Error - HTTP Status Code :404 {"callAPI": "oracle-database-operator-system/pdb10", "Err": "{\n \"code\": \"NotFound\",\n \"message\": \"Not Found\",\n \"type\": \"tag:oracle.com,2020:error/NotFound\",\n \"instance\": \"tag:oracle.com,2020:ecid/MKp4jbywUmPdI85yu0utJQ\"\n}"}
2024-02-16T07:39:49Z DEBUG events cdb-dev {"type": "Warning", "object": {"kind":"PDB","namespace":"oracle-database-operator-system","name":"pdb10","uid":"4c88e5c8-cc5d-4ce9-ab0a-452cb1261e54","apiVersion":"database.oracle.com/v1alpha1","resourceVersion":"935019059"}, "reason": "Done"}
2024-02-16T07:39:49Z DEBUG events Failed: Not Found {"type": "Warning", "object": {"kind":"PDB","namespace":"oracle-database-operator-system","name":"pdb10","uid":"4c88e5c8-cc5d-4ce9-ab0a-452cb1261e54","apiVersion":"database.oracle.com/v1alpha1","resourceVersion":"935019059"}, "reason": "ORDSError"}
2024-02-16T07:39:49Z INFO controllers.PDB Issuing REST call {"callAPI": "oracle-database-operator-system/pdb10", "URL": "https://cdb-dev-ords:8888/ords/_/db-api/latest/database/pdbs/", "Action": "POST"}
2024-02-16T07:39:49Z INFO controllers.PDB Found CR for CDB {"getCDBResource": "oracle-database-operator-system/pdb10", "Name": "cdb-dev", "CR Name": "cdb-dev"}
2024-02-16T07:39:49Z DEBUG events cdb-dev {"type": "Warning", "object": {"kind":"PDB","namespace":"oracle-database-operator-system","name":"pdb10","uid":"4c88e5c8-cc5d-4ce9-ab0a-452cb1261e54","apiVersion":"database.oracle.com/v1alpha1","resourceVersion":"935019065"}, "reason": "Done"}
2024-02-16T07:39:49Z DPANIC controllers.PDB odd number of arguments passed as key-value pairs for logging {"createPDB": "oracle-database-operator-system/pdb10", "ignored key": "ORDS Error"}
github.com/oracle/oracle-database-operator/controllers/database.(*PDBReconciler).createPDB
/workspace/controllers/database/pdb_controller.go:689
github.com/oracle/oracle-database-operator/controllers/database.(*PDBReconciler).Reconcile
/workspace/controllers/database/pdb_controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234
2024-02-16T07:39:49Z INFO Observed a panic in reconciler: odd number of arguments passed as key-value pairs for logging {"controller": "pdb", "controllerGroup": "database.oracle.com", "controllerKind": "PDB", "PDB": {"name":"pdb10","namespace":"oracle-database-operator-system"}, "namespace": "oracle-database-operator-system", "name": "pdb10", "reconcileID": "ead0ae80-07e7-4058-a910-c96b16da7d27"}
panic: odd number of arguments passed as key-value pairs for logging [recovered]
panic: odd number of arguments passed as key-value pairs for logging
goroutine 784 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118 +0x1f4
panic({0x1c6a640, 0xc003336930})
/usr/local/go/src/runtime/panic.go:884 +0x212
go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x1?, 0x1?, {0x0?, 0x0?, 0xc003330140?})
/go/pkg/mod/go.uber.org/[email protected]/zapcore/entry.go:198 +0x65
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0009dc5b0, {0xc00305b280, 0x1, 0x1})
/go/pkg/mod/go.uber.org/[email protected]/zapcore/entry.go:264 +0x3ec
go.uber.org/zap.(*Logger).DPanic(0x2091b7c?, {0x20f4181?, 0x1c6a640?}, {0xc00305b280, 0x1, 0x1})
/go/pkg/mod/go.uber.org/[email protected]/logger.go:241 +0x59
github.com/go-logr/zapr.(*zapLogger).handleFields(0xc0007ef5c0, 0xffffffffffffffff, {0xc0033368e0, 0x1, 0x22b8?}, {0xc00305b180?, 0x1, 0x0?})
/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:147 +0xd3f
github.com/go-logr/zapr.(*zapLogger).Error(0xc0007ef5c0, {0x238b5c0?, 0xc0033368d0}, {0x2094469?, 0x0?}, {0xc0033368e0, 0x1, 0x1})
/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:216 +0x1ac
github.com/go-logr/logr.Logger.Error({{0x23b06d0?, 0xc0007ef5c0?}, 0x23adeb8?}, {0x238b5c0, 0xc0033368d0}, {0x2094469, 0xd}, {0xc0033368e0, 0x1, 0x1})
/go/pkg/mod/github.com/go-logr/[email protected]/logr.go:279 +0xba
github.com/oracle/oracle-database-operator/controllers/database.(*PDBReconciler).createPDB(0xc0003da870, {0x23adeb8, 0xc0007eee70}, {{{0xc0007d83e0, 0x1f}, {0xc00081e8e0, 0x5}}}, 0xc00007b000)
/workspace/controllers/database/pdb_controller.go:689 +0xf2f
github.com/oracle/oracle-database-operator/controllers/database.(*PDBReconciler).Reconcile(0xc0003da870, {0x23adeb8, 0xc0007eee70}, {{{0xc0007d83e0, 0x1f}, {0xc00081e8e0, 0x5}}})
/workspace/controllers/database/pdb_controller.go:218 +0x1087
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x23ade10?, {0x23adeb8?, 0xc0007eee70?}, {{{0xc0007d83e0?, 0x1f8b6e0?}, {0xc00081e8e0?, 0xc00015fdd0?}}})
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00059d220, {0x23ade10, 0xc000735840}, {0x1e1d180?, 0xc000434c60?})
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320 +0x33c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00059d220, {0x23ade10, 0xc000735840})
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:230 +0x333
There are two problems here 1) panic due to odd number of args passed to log function: this is fixed in the latest version and 2) the invalid database name; Ords throws an HTTP error instead of ORA-911 , discussing internally
2024-02-19T10:48:23Z INFO controllers.PDB Issuing REST call {"callAPI": {"name":"pdb10","namespace":"pdbnamespace"}, "URL": "https://cdb-dev-ords.cdbnamespace:8888/ords/_/db-api/latest/database/pdbs/_pdbdev/status", "Action": "GET"}
2024-02-19T10:48:23Z INFO controllers.PDB ORDS Error - HTTP Status Code :404 {"callAPI": {"name":"pdb10","namespace":"pdbnamespace"}, "Err": "{\n \"code\": \"NotFound\",\n \"message\": \"Not Found\",\n \"type\": \"tag:oracle.com,2020:error/NotFound\",\n \"instance\": \"tag:oracle.com,2020:ecid/kifsXCri1gUWE1tKerUlpA\"\n}"}
2024-02-19T10:48:23Z DEBUG events cdb-dev {"type": "Warning", "object": {"kind":"PDB","namespace":"pdbna