dbptk-developer icon indicating copy to clipboard operation
dbptk-developer copied to clipboard

Published 20 hours ago • verified publisher icon keeps

dbptk stops unexpectedly "Found an unsupported datatype value, and an attempt was made to save it as text."

Open Laurira opened this issue 3 years ago • 5 comments

How to turn on more detailed log? At the moment dbptk stops with no errors and in the log the last line is "Found an unsupported datatype value, and an attempt was made to save it as text."

What to do?

Laurira avatar Nov 30 '21 14:11 Laurira

When DPBTK Developer detects a datatype that is not supported by the JDBC driver it logs the information about the datatype. One time during the process of converting the JDBC datatype into the internal DBPTK datatype.

https://github.com/keeps/dbptk-developer/blob/6605b4b1ed70e4af8b61a8277e1eb1007e7aadbe/dbptk-modules/dbptk-module-jdbc/src/main/java/com/databasepreservation/modules/jdbc/in/JDBCDatatypeImporter.java#L468-L473

And another during the process of writting the metadata.xml file.

https://github.com/keeps/dbptk-developer/blob/6605b4b1ed70e4af8b61a8277e1eb1007e7aadbe/dbptk-modules/dbptk-module-siard/src/main/java/com/databasepreservation/modules/siard/out/content/Sql2008toXSDType.java#L111-L115

hmiguim avatar Feb 15 '22 13:02 hmiguim

the dba sent me the report not log and in the end of log there is something more:

2021-11-30 15:52:35,272 [main] INFO (c.d.m.m.f.ProgressLoggerObserver) Completed table 'vastus' (17 of 17 tables completed) 2021-11-30 15:52:35,272 [main] INFO (c.d.m.m.f.ProgressLoggerObserver) Completed schema 'egt' (1 of 1 schemas completed) 2021-11-30 15:52:35,272 [main] DEBUG (c.d.m.j.i.JDBCImportModule) Freeing resources 2021-11-30 15:52:35,820 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10} 2021-11-30 15:52:35,820 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10} 2021-11-30 15:52:35,822 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10} 2021-11-30 15:52:35,826 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10} 2021-11-30 15:52:35,826 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10} 2021-11-30 15:52:35,829 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10} 2021-11-30 15:52:35,829 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10} 2021-11-30 15:52:35,830 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10} 2021-11-30 15:52:35,831 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10} 2021-11-30 15:52:35,832 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10} 2021-11-30 15:52:35,833 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10} 2021-11-30 15:52:35,833 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10} 2021-11-30 15:52:35,833 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10} 2021-11-30 15:52:35,834 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10} 2021-11-30 15:52:35,834 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10} 2021-11-30 15:52:35,835 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10} 2021-11-30 15:52:35,835 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10} 2021-11-30 15:52:35,835 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10} 2021-11-30 15:52:36,059 [main] INFO (c.d.m.m.f.ProgressLoggerObserver) Completed database migration 2021-11-30 15:52:36,060 [main] DEBUG (c.d.m.j.i.JDBCImportModule) Closing connection to source database

Why does it print out this message 18 times?

Laurira avatar Feb 15 '22 14:02 Laurira

And it is weird that siard file were not created without any other error. We will try to re-run it on thursday and then give additional feedback.

Laurira avatar Feb 15 '22 14:02 Laurira

Any news on this?

hmiguim avatar Mar 07 '22 09:03 hmiguim

Yes, they made a new attampt on Friday. There were slight changes in the yml and it seems that dbptk did not crash any more. But I have log as well and there are 66 weird SQLExceptions.

2022-02-27 19:16:34,720 [main] DEBUG (c.d.m.r.Reporter) dataTypeChangedOnImport, invoker: com.databasepreservation.modules.oracle.in.Oracle12cJDBCDatatypeImporter; message: - Type conversion in import module: in KOLA.HA_ISIKUD.ROA_TAGASI_NOUTUD (format: schema.table.column) has original type NUMBER and was converted to the standard type NUMERIC(10,2); and type: Type{description='null', originalTypeName='NUMBER', sql99TypeName='NUMERIC(10,2)', sql2008TypeName='NUMERIC(10,2)'}-->SimpleTypeNumericExact{precision=10, scale=2} 2022-02-27 19:16:34,720 [main] DEBUG (c.d.m.j.i.JDBCImportModule) Data type: 2 Type name: NUMBER Column size: 10 Decimal digits: 2 Radix: 10 Remarks: null Default value: null Index: 28 Is Nullable: true Is auto increment: false Calculated type: SimpleTypeNumericExact ColumnType hash: 1455568852

2022-02-27 19:16:34,798 [main] DEBUG (c.d.m.j.i.JDBCImportModule) handled SQLException java.sql.SQLException: Invalid column name at oracle.jdbc.driver.OracleStatement.getColumnIndexPrimitive(OracleStatement.java:3899) at oracle.jdbc.driver.OracleStatement.getColumnIndex(OracleStatement.java:3857) at oracle.jdbc.driver.InsensitiveScrollableResultSet.findColumn(InsensitiveScrollableResultSet.java:270) at oracle.jdbc.driver.GeneratedResultSet.getString(GeneratedResultSet.java:596) at com.databasepreservation.modules.jdbc.in.JDBCImportModule.getTriggers(JDBCImportModule.java:1582) at com.databasepreservation.modules.jdbc.in.JDBCImportModule.getTableStructure(JDBCImportModule.java:886) at com.databasepreservation.modules.oracle.in.Oracle12cJDBCImportModule.getTableStructure(Oracle12cJDBCImportModule.java:186) at com.databasepreservation.modules.jdbc.in.JDBCImportModule.getTables(JDBCImportModule.java:674) at com.databasepreservation.modules.jdbc.in.JDBCImportModule.getSchemaStructure(JDBCImportModule.java:511) at com.databasepreservation.modules.oracle.in.Oracle12cJDBCImportModule.getSchemas(Oracle12cJDBCImportModule.java:156) at com.databasepreservation.modules.jdbc.in.JDBCImportModule.getDatabaseStructure(JDBCImportModule.java:409) at com.databasepreservation.modules.jdbc.in.JDBCImportModule.migrateDatabaseTo(JDBCImportModule.java:2075) at com.databasepreservation.DatabaseMigration.migrate(DatabaseMigration.java:123) at com.databasepreservation.Main.runMigration(Main.java:294) at com.databasepreservation.Main.internalMain(Main.java:125) at com.databasepreservation.Main.main(Main.java:81) 2022-02-27 19:16:35,126 [main] INFO (c.d.m.j.i.JDBCImportModule) Ignoring table KOLA.HA_ISIKUD_ASENDAMINE 2022-02-27 19:16:35,126 [main] INFO (c.d.m.j.i.JDBCImportModule) Obtaining table structure for KOLA.HA_LAHENDITE_SISUD 2022-02-27 19:16:35,141 [main] DEBUG (c.d.m.r.Reporter) dataTypeChangedOnImport, invoker: com.databasepreservation.modules.oracle.in.Oracle12cJDBCDatatypeImporter; message: - Type conversion in import module: in KOLA.HA_LAHENDITE_SISUD.KOOD (format: schema.table.column) has original type NUMBER and was converted to the standard type NUMERIC(10); and type: Type{description='null', originalTypeName='NUMBER', sql99TypeName='NUMERIC(10)', sql2008TypeName='NUMERIC(10)'}-->SimpleTypeNumericExact{precision=10, scale=0} 2022-02-27 19:16:35,141 [main] DEBUG (c.d.m.j.i.JDBCImportModule) Data type: 2 Type name: NUMBER Column size: 10 Decimal digits: 0 Radix: 10 Remarks: null Default value: null Index: 1 Is Nullable: false Is auto increment: false Calculated type: SimpleTypeNumericExact ColumnType hash: -1642140362

This column has letter "õ" missing from its description. It should be "Riigi õigusabi (RÕA)...": ROA_TAGASI_NOUTUD NUMERIC(10,2) <typeOriginal>NUMBER</typeOriginal> true Riigi ?igusabi (R?A) tagasi n?utud summa, alates jaanuar 2005

Laurira avatar Mar 07 '22 10:03 Laurira