linq2db.EntityFrameworkCore
linq2db.EntityFrameworkCore copied to clipboard
Getting error "ORA-00928" when trying to context.BulkCopy
Hey!
I just discovered linq2db and linq2db.EntityFramwrokCore while searching for a Bulk Insert option that would work with EF Core 3.1 and Oracle. At first I was a bit skeptical of it being so simple to work with. Now I think I may be missing something, because it's not working for me.
Ok, so I have a collection of <FlatConsult> entity that I need to upsert into my Oracle (19c) database. This is the code:
LinqToDBForEFTools.Initialize();
List<FlatConsult> data = GetData();
dwDb.BulkCopy(new BulkCopyOptions
{
BulkCopyType = OracleTools.DefaultBulkCopyType,
TableName = "ST.\"FlatConsults\""
}, data);
I had to specify the table name otherwise I would get a “ORA-00942: table or view does not exist” error. But now I keep getting a "ORA-00928: missing SELECT keyword" error:
Oracle.ManagedDataAccess.Client.OracleException (0x80004005): ORA-00928: palavra-chave SELECT não encontrada
em OracleInternal.ServiceObjects.OracleConnectionImpl.VerifyExecution(Int32& cursorId, Boolean bThrowArrayBindRelatedErrors, SqlStatementType sqlStatementType, Int32 arrayBindCount, OracleException& exceptionForArrayBindDML, Boolean& hasMoreRowsInDB, Boolean bFirstIterationDone)
em OracleInternal.ServiceObjects.OracleCommandImpl.ExecuteNonQuery(String commandText, OracleParameterCollection paramColl, CommandType commandType, OracleConnectionImpl connectionImpl, Int32 longFetchSize, Int64 clientInitialLOBFS, OracleDependencyImpl orclDependencyImpl, Int64[]& scnFromExecution, OracleParameterCollection& bindByPositionParamColl, Boolean& bBindParamPresent, OracleException& exceptionForArrayBindDML, OracleConnection connection, OracleLogicalTransaction& oracleLogicalTransaction, Boolean isFromEF)
em Oracle.ManagedDataAccess.Client.OracleCommand.ExecuteNonQuery()
em LinqToDB.Data.DataConnection.ExecuteNonQuery(IDbCommand command) na D:\a\1\s\Source\LinqToDB\Data\DataConnection.cs:linha 1256
em LinqToDB.Data.DataConnection.ExecuteNonQuery() na D:\a\1\s\Source\LinqToDB\Data\DataConnection.cs:linha 1261
em LinqToDB.Data.CommandInfo.Execute() na D:\a\1\s\Source\LinqToDB\Data\CommandInfo.cs:linha 863
em LinqToDB.DataProvider.MultipleRowsHelper.Execute() na D:\a\1\s\Source\LinqToDB\DataProvider\MultipleRowsHelper.cs:linha 100
em LinqToDB.DataProvider.BasicBulkCopy.MultipleRowsCopyHelper(MultipleRowsHelper helper, IEnumerable source, String from, Action`1 prepFunction, Action`3 addFunction, Action`1 finishFunction, Int32 maxParameters, Int32 maxSqlLength) na D:\a\1\s\Source\LinqToDB\DataProvider\BasicBulkCopy.cs:linha 279
em LinqToDB.DataProvider.Oracle.OracleBulkCopy.OracleMultipleRowsCopy1(MultipleRowsHelper helper, IEnumerable source) na D:\a\1\s\Source\LinqToDB\DataProvider\Oracle\OracleBulkCopy.cs:linha 186
em LinqToDB.DataProvider.Oracle.OracleBulkCopy.MultipleRowsCopy[T](ITable`1 table, BulkCopyOptions options, IEnumerable`1 source) na D:\a\1\s\Source\LinqToDB\DataProvider\Oracle\OracleBulkCopy.cs:linha 126
em LinqToDB.DataProvider.BasicBulkCopy.BulkCopy[T](BulkCopyType bulkCopyType, ITable`1 table, BulkCopyOptions options, IEnumerable`1 source) na D:\a\1\s\Source\LinqToDB\DataProvider\BasicBulkCopy.cs:linha 21
em LinqToDB.DataProvider.Oracle.OracleDataProvider.BulkCopy[T](ITable`1 table, BulkCopyOptions options, IEnumerable`1 source) na D:\a\1\s\Source\LinqToDB\DataProvider\Oracle\OracleDataProvider.cs:linha 300
em LinqToDB.Data.DataConnectionExtensions.BulkCopy[T](DataConnection dataConnection, BulkCopyOptions options, IEnumerable`1 source) na D:\a\1\s\Source\LinqToDB\Data\DataConnectionExtensions.cs:linha 2194
em LinqToDB.EntityFrameworkCore.LinqToDBForEFTools.BulkCopy[T](DbContext context, BulkCopyOptions options, IEnumerable`1 source) na D:\a\1\s\Source\LinqToDB.EntityFrameworkCore\LinqToDBForEFTools.ContextExtensions.cs:linha 32
em Mantic.BlackOps.Jobs.Maintenance.Utils.FlatConsultsProcessor.<UpdateConsultsAsync>d__5.MoveNext() na C:\Repos\Tekap.Mantic.BlackOps\Mantic.BlackOps.Jobs.Maintenance\Utils\FlatConsultsProcessor.cs:linha 116
--- Fim do rastreamento de pilha do local anterior onde a exceção foi gerada ---
em System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
em System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
em System.Runtime.CompilerServices.TaskAwaiter.GetResult()
em Mantic.BlackOps.Jobs.Maintenance.Jobs.FlatConsultsJob.<OrchestrateAsync>d__6.MoveNext() na C:\Repos\Tekap.Mantic.BlackOps\Mantic.BlackOps.Jobs.Maintenance\Jobs\FlatConsultsJob.cs:linha 79
[12:28:02 WRN T09] Failed to process the job '1': an exception occurred. Retry attempt 1 of 3 will be performed in 00:00:33. (Hangfire.AutomaticRetryAttribute)
Oracle.ManagedDataAccess.Client.OracleException (0x80004005): ORA-00928: palavra-chave SELECT não encontrada
em OracleInternal.ServiceObjects.OracleConnectionImpl.VerifyExecution(Int32& cursorId, Boolean bThrowArrayBindRelatedErrors, SqlStatementType sqlStatementType, Int32 arrayBindCount, OracleException& exceptionForArrayBindDML, Boolean& hasMoreRowsInDB, Boolean bFirstIterationDone)
em OracleInternal.ServiceObjects.OracleCommandImpl.ExecuteNonQuery(String commandText, OracleParameterCollection paramColl, CommandType commandType, OracleConnectionImpl connectionImpl, Int32 longFetchSize, Int64 clientInitialLOBFS, OracleDependencyImpl orclDependencyImpl, Int64[]& scnFromExecution, OracleParameterCollection& bindByPositionParamColl, Boolean& bBindParamPresent, OracleException& exceptionForArrayBindDML, OracleConnection connection, OracleLogicalTransaction& oracleLogicalTransaction, Boolean isFromEF)
em Oracle.ManagedDataAccess.Client.OracleCommand.ExecuteNonQuery()
em LinqToDB.Data.DataConnection.ExecuteNonQuery(IDbCommand command) na D:\a\1\s\Source\LinqToDB\Data\DataConnection.cs:linha 1256
em LinqToDB.Data.DataConnection.ExecuteNonQuery() na D:\a\1\s\Source\LinqToDB\Data\DataConnection.cs:linha 1261
em LinqToDB.Data.CommandInfo.Execute() na D:\a\1\s\Source\LinqToDB\Data\CommandInfo.cs:linha 863
em LinqToDB.DataProvider.MultipleRowsHelper.Execute() na D:\a\1\s\Source\LinqToDB\DataProvider\MultipleRowsHelper.cs:linha 100
em LinqToDB.DataProvider.BasicBulkCopy.MultipleRowsCopyHelper(MultipleRowsHelper helper, IEnumerable source, String from, Action`1 prepFunction, Action`3 addFunction, Action`1 finishFunction, Int32 maxParameters, Int32 maxSqlLength) na D:\a\1\s\Source\LinqToDB\DataProvider\BasicBulkCopy.cs:linha 279
em LinqToDB.DataProvider.Oracle.OracleBulkCopy.OracleMultipleRowsCopy1(MultipleRowsHelper helper, IEnumerable source) na D:\a\1\s\Source\LinqToDB\DataProvider\Oracle\OracleBulkCopy.cs:linha 186
em LinqToDB.DataProvider.Oracle.OracleBulkCopy.MultipleRowsCopy[T](ITable`1 table, BulkCopyOptions options, IEnumerable`1 source) na D:\a\1\s\Source\LinqToDB\DataProvider\Oracle\OracleBulkCopy.cs:linha 126
em LinqToDB.DataProvider.BasicBulkCopy.BulkCopy[T](BulkCopyType bulkCopyType, ITable`1 table, BulkCopyOptions options, IEnumerable`1 source) na D:\a\1\s\Source\LinqToDB\DataProvider\BasicBulkCopy.cs:linha 21
em LinqToDB.DataProvider.Oracle.OracleDataProvider.BulkCopy[T](ITable`1 table, BulkCopyOptions options, IEnumerable`1 source) na D:\a\1\s\Source\LinqToDB\DataProvider\Oracle\OracleDataProvider.cs:linha 300
em LinqToDB.Data.DataConnectionExtensions.BulkCopy[T](DataConnection dataConnection, BulkCopyOptions options, IEnumerable`1 source) na D:\a\1\s\Source\LinqToDB\Data\DataConnectionExtensions.cs:linha 2194
em LinqToDB.EntityFrameworkCore.LinqToDBForEFTools.BulkCopy[T](DbContext context, BulkCopyOptions options, IEnumerable`1 source) na D:\a\1\s\Source\LinqToDB.EntityFrameworkCore\LinqToDBForEFTools.ContextExtensions.cs:linha 32
em Mantic.BlackOps.Jobs.Maintenance.Utils.FlatConsultsProcessor.<UpdateConsultsAsync>d__5.MoveNext() na C:\Repos\Tekap.Mantic.BlackOps\Mantic.BlackOps.Jobs.Maintenance\Utils\FlatConsultsProcessor.cs:linha 116
--- Fim do rastreamento de pilha do local anterior onde a exceção foi gerada ---
em System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
em System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
em System.Runtime.CompilerServices.TaskAwaiter.GetResult()
em Mantic.BlackOps.Jobs.Maintenance.Jobs.FlatConsultsJob.<OrchestrateAsync>d__6.MoveNext() na C:\Repos\Tekap.Mantic.BlackOps\Mantic.BlackOps.Jobs.Maintenance\Jobs\FlatConsultsJob.cs:linha 88
--- Fim do rastreamento de pilha do local anterior onde a exceção foi gerada ---
em System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
em System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
em System.Runtime.CompilerServices.TaskAwaiter.GetResult()
em Mantic.BlackOps.Jobs.Maintenance.Jobs.FlatConsultsJob.<>c__DisplayClass5_0.<<Start>b__0>d.MoveNext() na C:\Repos\Tekap.Mantic.BlackOps\Mantic.BlackOps.Jobs.Maintenance\Jobs\FlatConsultsJob.cs:linha 41
--- Fim do rastreamento de pilha do local anterior onde a exceção foi gerada ---
em System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
em System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
em System.Runtime.CompilerServices.TaskAwaiter.GetResult()
em Mantic.BlackOps.Jobs.Maintenance.Jobs.FlatConsultsJob.<Start>d__5.MoveNext() na C:\Repos\Tekap.Mantic.BlackOps\Mantic.BlackOps.Jobs.Maintenance\Jobs\FlatConsultsJob.cs:linha 40
I even added linq2db.Oracle.Managed to see if it would help, but no success. I'm probably missing some more configuration, but I couldn't find any help on it.
And thank you for your work!
I think problem here that you are trying to escape table name. Linq2db will do that automatically.
dwDb.BulkCopy(new BulkCopyOptions
{
BulkCopyType = OracleTools.DefaultBulkCopyType,
TableName = "FlatConsults",
SchemaName = "ST"
}, data);
You do not have to specify these names if it is regular EF Model.
I tried doing the way you recommended but it returns the same error that I get if I don't specify the table name: Oracle - ORA-00942: table or view does not exist.
If in your EF Core application configured logging, you should see SQL sent to Oracle. Could you please post it?
Anyway, check permissions. It is usual Oracle error.
I forgot to mention that no SQL commands are logged when I get this errors. I get SQL statements in logs from LINQ queries but not from context.BulkCopy. I only get the exception log.
Logging should work if you specify optionsBuilder.UseLoggerFactory
Logs are working, I just don't get any SQL logs when I run ctx.BulkCopy().
I had to use DataConnection.TurnTraceSwitchOn();
but now SQL logs are shown:
info: LinqToDB[1]
BeforeExecute
-- Oracle Oracle11
INSERT ALL
INTO "ST."FlatConsults"" (Day, RetailerId, RetailerTradingName, RetailerDocument, SaleCountCash, SaleCountCard, ConsultCountCash, ConsultCountCard) VALUES (TO_TIMESTAMP('2020-09-13 00:00:00.000000', 'YYYY-MM-DD HH24:MI:SS.FF6'),58934,'Retailer 339','325057315',402,297,537,675)
INTO "ST."FlatConsults"" (Day, RetailerId, RetailerTradingName, RetailerDocument, SaleCountCash, SaleCountCard, ConsultCountCash, ConsultCountCard) VALUES (TO_TIMESTAMP('2020-09-16 00:00:00.000000', 'YYYY-MM-DD HH24:MI:SS.FF6'),94179,'Retailer 38','382891318',441,385,535,480)
INTO "ST."FlatConsults"" (Day, RetailerId, RetailerTradingName, RetailerDocument, SaleCountCash, SaleCountCard, ConsultCountCash, ConsultCountCard) VALUES (TO_TIMESTAMP('2020-09-05 00:00:00.000000', 'YYYY-MM-DD HH24:MI:SS.FF6'),15545,'Retailer 593','464420620',959,165,232,791)
INTO "ST."FlatConsults"" (Day, RetailerId, RetailerTradingName, RetailerDocument, SaleCountCash, SaleCountCard, ConsultCountCash, ConsultCountCard) VALUES (TO_TIMESTAMP('2020-08-18 00:00:00.000000', 'YYYY-MM-DD HH24:MI:SS.FF6'),67795,'Retailer 429','618368830',433,197,851,766)
INTO "ST."FlatConsults"" (Day, RetailerId, RetailerTradingName, RetailerDocument, SaleCountCash, SaleCountCard, ConsultCountCash, ConsultCountCard) VALUES (TO_TIMESTAMP('2020-08-27 00:00:00.000000', 'YYYY-MM-DD HH24:MI:SS.FF6'),21638,'Retailer 653','194960673',955,363,817,141)
INTO "ST."FlatConsults"" (Day, RetailerId, RetailerTradingName, RetailerDocument, SaleCountCash, SaleCountCard, ConsultCountCash, ConsultCountCard) VALUES (TO_TIMESTAMP('2020-08-23 00:00:00.000000', 'YYYY-MM-DD HH24:MI:SS.FF6'),71157,'Retailer 543','431510080',69,440,772,78)
SELECT * FROM dual
fail: LinqToDB[2]
Error
Exception: Oracle.ManagedDataAccess.Client.OracleException
Message : ORA-00928: palavra-chave SELECT não encontrada
at OracleInternal.ServiceObjects.OracleConnectionImpl.VerifyExecution(Int32& cursorId, Boolean bThrowArrayBindRelatedErrors, SqlStatementType sqlStatementType, Int32 arrayBindCount, OracleException& exceptionForArrayBindDML, Boolean& hasMoreRowsInDB, Boolean bFirstIterationDone)
at OracleInternal.ServiceObjects.OracleCommandImpl.ExecuteNonQuery(String commandText, OracleParameterCollection paramColl, CommandType commandType, OracleConnectionImpl connectionImpl, Int32 longFetchSize, Int64 clientInitialLOBFS, OracleDependencyImpl orclDependencyImpl, Int64[]& scnFromExecution, OracleParameterCollection& bindByPositionParamColl, Boolean& bBindParamPresent, OracleException& exceptionForArrayBindDML, OracleConnection connection, OracleLogicalTransaction& oracleLogicalTransaction, Boolean isFromEF)
at Oracle.ManagedDataAccess.Client.OracleCommand.ExecuteNonQuery()
at LinqToDB.Data.DataConnection.ExecuteNonQuery(IDbCommand command)
at LinqToDB.Data.DataConnection.ExecuteNonQuery()
Looks like Oracle have broken backward compatibility and we have to correct BulkCopy SQL for 19-th dialect.
"ST."FlatConsults""
- something strange in this name.
@lhmiranda Were these logs taken with your original code (TableName = "ST.\"FlatConsults\""
) or with the suggested revision (TableName = "FlatConsults", SchemaName = "ST"
) ?
@Shane32
-
TableName = "ST.\"FlatConsults\""
yields:...INTO "ST."FlatConsults"" (...
-
TableName = "FlatConsults"
yields:...INTO FlatConsults (...
-
TableName = "FlatConsults", SchemaName = "ST"
yields:...INTO ST.FlatConsults (...
- Not specifying
TableName
andSchemaName
yields...INTO FlatConsults (...
(same as 2) -
SchemaName = "ST"
yields:...INTO ST.FlatConsults (...
(same as 3)
None of which is accepted. I should be: ...INTO ST."FlatConsults" (...
.
@lhmiranda, maybe you have specification link about this escaping and we will update library for next release which is planned on this Thursday.
I don't have Oracle, but I found this link:
- https://docs.oracle.com/database/121/SQLRF/sql_elements008.htm#SQLRF51129
Of particular interest: (added emphasis)
A quoted identifier begins and ends with double quotation marks ("). If you name a schema object using a quoted identifier, then you must use the double quotation marks whenever you refer to that object.
So it would seem that we would need to support table names in the format of "name"
in addition to name
, and that it should not add additional escaping. Also noteworthy:
Nonquoted identifiers can contain only alphanumeric characters from your database character set and the underscore (_), dollar sign ($), and pound sign (#). Database links can also contain periods (.) and "at" signs (@).
Quoted identifiers can contain any characters and punctuations marks as well as spaces. However, neither quoted nor nonquoted identifiers can contain double quotation marks or the null character (\0).
Perhaps it should follow this pseudocode:
- If matches regex
^\"[^\"]*\"$
then use as-is (quoted identifier) - If contains
\"
then throw exception - If matches regex
[^a-zA-Z_$#]
anywhere in the string then assume quoted identifier and add quotes - Otherwise assume unquoted identifier
As for the linq2db.EntityFrameworkCore
library, we may need to investigate how EF Core normally escapes table names, and match that.
Sorry, I don't have Oracle to do testing with.
I really don't have a link for it. I learned it while using OCI's (Oracle Cloud Infrastructure) SQL Developer Web.
I tried the standard "SELECT * FROM FlatConsults" and it didn't worked. Then I dragged the table from the tables list into the query area it let me pick a "select data" option and then filled the query text with this:
Edit: also, I didn't metioned it earlier because I though it wouldn't matter, but now I think it might: This is Oracle's Autonomous Database version 19c.
Perhaps there should also be a global or connection-specific Oracle flag to escape all object names, replacing steps 3 & 4 of the above with the following:
- Assume quoted identifier and add quotes
Looking at EF Core queries in the log, it does uses quotes with table names when running queries, but it does not uses the schema name in front of it:
var consults = await db.FlatConsults.OrderByDescending(o => o.Day).Take(10).ToListAsync();
info: Microsoft.EntityFrameworkCore.Database.Command[20101]
Executed DbCommand (214ms) [Parameters=[:p_0='?' (DbType = Int32)], CommandType='Text', CommandTimeout='0']
SELECT "f"."Day", "f"."RetailerId", "f"."ConsultCountCard", "f"."ConsultCountCash", "f"."RetailerDocument", "f"."RetailerTradingName", "f"."SaleCountCard", "f"."SaleCountCash"
FROM "FlatConsults" "f"
ORDER BY "f"."Day" DESC
FETCH FIRST :p_0 ROWS ONLY
I assume this should also fail?
var consults = await db.FlatConsults.OrderByDescending(o => o.Day).Take(10).ToLinqToDB().ToListAsyncLinqToDB();
Yep, same problem:
info: LinqToDB[1]
BeforeExecute
-- Oracle Oracle11 (asynchronously)
DECLARE @take Int32
SET @take = 10
SELECT
t2.Day_1,
t2.RetailerId,
t2.RetailerTradingName,
t2.RetailerDocument,
t2.SaleCountCash,
t2.SaleCountCard,
t2.ConsultCountCash,
t2.ConsultCountCard
FROM
(
SELECT
t1.Day as Day_1,
t1.RetailerId,
t1.RetailerTradingName,
t1.RetailerDocument,
t1.SaleCountCash,
t1.SaleCountCard,
t1.ConsultCountCash,
t1.ConsultCountCard
FROM
FlatConsults t1
ORDER BY
t1.Day DESC
) t2
WHERE
ROWNUM <= :take
fail: LinqToDB[2]
Error
Exception: Oracle.ManagedDataAccess.Client.OracleException
Message : ORA-00942: table or view does not exist
at OracleInternal.ServiceObjects.OracleConnectionImpl.VerifyExecution(Int32& cursorId, Boolean bThrowArrayBindRelatedErrors, SqlStatementType sqlStatementType, Int32 arrayBindCount, OracleException& exceptionForArrayBindDML, Boolean& hasMoreRowsInDB, Boolean bFirstIterationDone)
at OracleInternal.ServiceObjects.OracleCommandImpl.VerifyExecution(OracleConnectionImpl connectionImpl, Int32& cursorId, Boolean bThrowArrayBindRelatedErrors, OracleException& exceptionForArrayBindDML, Boolean& hasMoreRowsInDB, Boolean bFirstIterationDone)
at OracleInternal.ServiceObjects.OracleCommandImpl.ExecuteReader(String commandText, OracleParameterCollection paramColl, CommandType commandType, OracleConnectionImpl connectionImpl, OracleDataReaderImpl& rdrImpl, Int32 longFetchSize, Int64 clientInitialLOBFS, OracleDependencyImpl orclDependencyImpl, Int64[] scnForExecution, Int64[]& scnFromExecution, OracleParameterCollection& bindByPositionParamColl, Boolean& bBindParamPresent, Int64& internalInitialLOBFS, OracleException& exceptionForArrayBindDML, OracleConnection connection, OracleLogicalTransaction& oracleLogicalTransaction, IEnumerable`1 adrianParsedStmt, Boolean isDescribeOnly, Boolean isFromEF)
at Oracle.ManagedDataAccess.Client.OracleCommand.ExecuteReader(Boolean requery, Boolean fillRequest, CommandBehavior behavior)
at Oracle.ManagedDataAccess.Client.OracleCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
--- End of stack trace from previous location where exception was thrown ---
at LinqToDB.Data.DataConnection.ExecuteReaderAsync(CommandBehavior commandBehavior, CancellationToken cancellationToken)
Could you try and set
OracleTools.DontEscapeLowercaseIdentifiers = false;
option? I wonder why we have it set to true by default.
It might solve this. It actually states in it's description that it should be set to false. I'll test it and report back shortly.
Edit: yep, that did it. Everything seems to be working as intended now. I don't even have to define table or schema names in BulkCopyOptions
anymore.
Thank you all for your time and support!
Closing as resolved