FHIR
FHIR copied to clipboard
Published
20 hours ago •
LinuxForHealth
LinuxForHealth
Truncate String parameter value before persisting to the database
Describe the bug The string parameter value should be truncated before persisting to the database. The size of string columns in the search tables(_str_values) is limited to 1024 bytes.
Environment Which version of LinuxForHealth FHIR Server? 5.0
To Reproduce
- Go to R4ExampleServerTest.main() method
- Pass "json/spec/medicationstatement.profile.json" as the JSON example in driver.processExample(...) .
- Run the main method.
- See error
[13/10/22, 14:10:42:079 UTC] 00000034 ParameterTran E Failed persisting parameter transaction data. Marking transaction for rollback
org.linuxforhealth.fhir.persistence.exception.FHIRPersistenceException: pushBatch failed for 'StructureDefinition' [probeId=c0-a8-1-f-c4963745-837e-4ee4-aba7-5a8e9c31ad09]
at org.linuxforhealth.fhir.persistence.params.database.PlainBatchParameterProcessor.pushBatch(PlainBatchParameterProcessor.java:117)
at org.linuxforhealth.fhir.persistence.params.database.PlainParamValueProcessor.pushBatch(PlainParamValueProcessor.java:110)
at org.linuxforhealth.fhir.persistence.jdbc.impl.FHIRPersistenceJDBCImpl.flush(FHIRPersistenceJDBCImpl.java:3124)
at org.linuxforhealth.fhir.persistence.jdbc.impl.FHIRPersistenceJDBCImpl.onCommit(FHIRPersistenceJDBCImpl.java:3155)
at org.linuxforhealth.fhir.persistence.jdbc.impl.ParameterTransactionDataImpl.persist(ParameterTransactionDataImpl.java:70)
at org.linuxforhealth.fhir.persistence.jdbc.dao.impl.TransactionDataImpl.lambda$persist$0(TransactionDataImpl.java:46)
at java.base/java.util.HashMap$Values.forEach(HashMap.java:977)
at org.linuxforhealth.fhir.persistence.jdbc.dao.impl.TransactionDataImpl.persist(TransactionDataImpl.java:46)
at org.linuxforhealth.fhir.persistence.jdbc.impl.CacheTransactionSync.beforeCompletion(CacheTransactionSync.java:74)
at com.ibm.tx.jta.impl.RegisteredSyncs.coreDistributeBefore(RegisteredSyncs.java:238)
at com.ibm.tx.jta.impl.RegisteredSyncs.distributeBefore(RegisteredSyncs.java:146)
at com.ibm.tx.jta.impl.TransactionImpl.prePrepare(TransactionImpl.java:1404)
at com.ibm.tx.jta.impl.TransactionImpl.stage1CommitProcessing(TransactionImpl.java:814)
at com.ibm.tx.jta.impl.TransactionImpl.processCommit(TransactionImpl.java:770)
at com.ibm.tx.jta.impl.TransactionImpl.commit(TransactionImpl.java:713)
at com.ibm.tx.jta.impl.TranManagerImpl.commit(TranManagerImpl.java:153)
at com.ibm.tx.jta.impl.TranManagerSet.commit(TranManagerSet.java:110)
at com.ibm.tx.jta.impl.UserTransactionImpl.commit(UserTransactionImpl.java:145)
at com.ibm.tx.jta.embeddable.impl.EmbeddableUserTransactionImpl.commit(EmbeddableUserTransactionImpl.java:101)
at com.ibm.ws.transaction.services.UserTransactionService.commit(UserTransactionService.java:72)
at org.linuxforhealth.fhir.persistence.jdbc.connection.FHIRUserTransactionAdapter.end(FHIRUserTransactionAdapter.java:140)
at org.linuxforhealth.fhir.persistence.helper.FHIRTransactionHelper.commit(FHIRTransactionHelper.java:43)
at org.linuxforhealth.fhir.server.util.FHIRRestHelper.doCreate(FHIRRestHelper.java:254)
at org.linuxforhealth.fhir.server.spi.operation.FHIRResourceHelpers.doCreate(FHIRResourceHelpers.java:117)
at org.linuxforhealth.fhir.server.resources.Create.create(Create.java:76)
at org.linuxforhealth.fhir.server.resources.Create$Proxy$_$$_WeldClientProxy.create(Unknown Source)
at jdk.internal.reflect.GeneratedMethodAccessor874.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.ibm.ws.jaxrs20.cdi.component.JaxRsFactoryImplicitBeanCDICustomizer.serviceInvoke(JaxRsFactoryImplicitBeanCDICustomizer.java:350)
at com.ibm.ws.jaxrs20.server.LibertyJaxRsServerFactoryBean.performInvocation(LibertyJaxRsServerFactoryBean.java:641)
at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.performInvocation(LibertyJaxRsInvoker.java:160)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:101)
at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.invoke(LibertyJaxRsInvoker.java:273)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:213)
at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.invoke(LibertyJaxRsInvoker.java:444)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:112)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:123)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:277)
at com.ibm.ws.jaxrs20.endpoint.AbstractJaxRsWebEndpoint.invoke(AbstractJaxRsWebEndpoint.java:137)
at com.ibm.websphere.jaxrs.server.IBMRestServlet.handleRequest(IBMRestServlet.java:146)
at com.ibm.websphere.jaxrs.server.IBMRestServlet.doPost(IBMRestServlet.java:104)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:706)
at com.ibm.websphere.jaxrs.server.IBMRestServlet.service(IBMRestServlet.java:96)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1258)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:746)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:443)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:193)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:98)
at org.linuxforhealth.fhir.server.filter.rest.FHIRRestServletFilter.doFilter(FHIRRestServletFilter.java:177)
at javax.servlet.http.HttpFilter.doFilter(HttpFilter.java:127)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com.ibm.ws.security.jaspi.JaspiServletFilter.doFilter(JaspiServletFilter.java:56)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:1002)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1140)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:5058)
at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.handleRequest(DynamicVirtualHost.java:316)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1007)
at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.run(DynamicVirtualHost.java:281)
at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:1239)
at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.wrapHandlerAndExecute(HttpDispatcherLink.java:468)
at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:427)
at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:566)
at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:500)
at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:360)
at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.ready(HttpInboundLink.java:327)
at com.ibm.ws.channel.ssl.internal.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1100)
at com.ibm.ws.channel.ssl.internal.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:675)
at com.ibm.ws.channel.ssl.internal.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1824)
at com.ibm.ws.tcpchannel.internal.WorkQueueManager.requestComplete(WorkQueueManager.java:514)
at com.ibm.ws.tcpchannel.internal.WorkQueueManager.attemptIO(WorkQueueManager.java:584)
at com.ibm.ws.tcpchannel.internal.WorkQueueManager.workerRun(WorkQueueManager.java:968)
at com.ibm.ws.tcpchannel.internal.WorkQueueManager$Worker.run(WorkQueueManager.java:1057)
at com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:245)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.sql.BatchUpdateException: A truncation error was encountered trying to shri[13/10/22, 14:10:42:102 UTC] 00000034 TransactionDa 1 persisted transaction data [took 8.944 s]
nk VARCHAR 'A record of a medication that is being consumed by a patient&' to length 1024.
at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeBatch(EmbedStatement.java:1113)
at org.apache.derby.impl.jdbc.EmbedStatement.executeBatch(EmbedStatement.java:1045)
at org.apache.derby.iapi.jdbc.BrokeredStatement.executeBatch(BrokeredStatement.java:84)
at com.ibm.ws.rsadapter.jdbc.WSJdbcStatement.executeBatch(WSJdbcStatement.java:495)
at org.linuxforhealth.fhir.persistence.params.database.PlainPostgresParameterBatch.pushBatch(PlainPostgresParameterBatch.java:80)
at org.linuxforhealth.fhir.persistence.params.database.PlainBatchParameterProcessor.pushBatch(PlainBatchParameterProcessor.java:115)
... 81 more
Caused by: java.sql.SQLDataException: A truncation error was encountered trying to shrink VARCHAR 'A record of a medication that is being consumed by a patient&' to length 1024.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:84)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Util.java:230)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:431)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:360)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2405)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:88)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1436)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeBatchElement(EmbedPreparedStatement.java:1065)
at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeBatch(EmbedStatement.java:1087)
... 86 more
Caused by: ERROR 22001: A truncation error was encountered trying to shrink VARCHAR 'A record of a medication that is being consumed by a patient&' to length 1024.
at org.apache.derby.shared.common.error.StandardException.newException(StandardException.java:300)
at org.apache.derby.shared.common.error.StandardException.newException(StandardException.java:295)
at org.apache.derby.iapi.types.SQLChar.hasNonBlankChars(SQLChar.java:1846)
at org.apache.derby.iapi.types.SQLVarchar.normalize(SQLVarchar.java:181)
at org.apache.derby.iapi.types.SQLVarchar.normalize(SQLVarchar.java:158)
at org.apache.derby.iapi.types.DataTypeDescriptor.normalize(DataTypeDescriptor.java:656)
at org.apache.derby.impl.sql.execute.NormalizeResultSet.normalizeColumn(NormalizeResultSet.java:332)
at org.apache.derby.impl.sql.execute.NormalizeResultSet.normalizeRow(NormalizeResultSet.java:376)
at org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(NormalizeResultSet.java:191)
at org.apache.derby.impl.sql.execute.DMLWriteResultSet.getNextRowCore(DMLWriteResultSet.java:148)
at org.apache.derby.impl.sql.execute.InsertResultSet.getNextRowCore(InsertResultSet.java:1082)
at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:451)
at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:472)
at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:351)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1344)
... 88 more
Expected behavior The string parameter value should be truncated before persisting to the database.
Additional context This can probably be done in SearchParametersTransportAdapter.stringValue() method.
This is a regression/change in behavior from 4.11.1. Previously, string search parameter values were truncated in ParameterVisitorBatchDAO:
public void visit(StringParmVal param) throws FHIRPersistenceException {
This class is no longer present in 5.0.0 after the parameter persistence was refactored.
https://github.com/LinuxForHealth/FHIR/pull/4024 fixed our tests so that we actually exercise the insert parameters logic as part of the persistence tests. After that change, the CI pipeline was properly failing due to this issue.
After merging Prasanna's fix, that pull request is now passing which helps to confirm that this is working as expected now.
I created a sample test patient with a name like the following:
"name": [
{
"family": "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz WAS IT TRUNCATED !?",
"given": [
"Lee"
]
}
],
I then confirmed that I am able to search this patient by
- Patient?name=Lee
- Patient?name=abcdefg
- Patient?name:contains=hijklmnop
Further, I verified that I cannot search it by
- Patient?name:contains=TRUNCATED
I think it would be nice to provide a warning back to the user if they submit a searchable string value that has been truncated on the server, but that is separate from this issue so lets call this one done.
linking https://github.com/LinuxForHealth/FHIR/issues/1638 which is somewhat related to that comment about providing a warning in the case of truncation