ml-gradle
ml-gradle copied to clipboard
marklogic
TDE templates no longer load
Hi. I have upgraded to version 3.16. Upon doing that all my templates have stopped loading. They worked up until 3.15.
Using --stacktrace I got this error. I know the template is valid because It works in versions up to 3.15 and I can load it manually with qconsole
Caused by: com.marklogic.client.FailedRequestException: Local message: failed to apply resource at eval: Internal Server Error. Server Message: Server (not a REST instance?) did not respond with an expected REST Error message.
at com.marklogic.client.impl.OkHttpServices.checkStatus(OkHttpServices.java:4395)
at com.marklogic.client.impl.OkHttpServices.postIteratedResourceImpl(OkHttpServices.java:3917)
at com.marklogic.client.impl.OkHttpServices.postEvalInvoke(OkHttpServices.java:3854)
at com.marklogic.client.impl.ServerEvaluationCallImpl.eval(ServerEvaluationCallImpl.java:164)
at com.marklogic.client.impl.ServerEvaluationCallImpl.eval(ServerEvaluationCallImpl.java:153)
at com.marklogic.client.ext.schemasloader.impl.TdeDocumentFileProcessor.validateTdeTemplate(TdeDocumentFileProcessor.java:88)
at com.marklogic.client.ext.schemasloader.impl.TdeDocumentFileProcessor.processDocumentFile(TdeDocumentFileProcessor.java:58)
at com.marklogic.client.ext.file.AbstractDocumentFileReader.processDocumentFile(AbstractDocumentFileReader.java:58)
... 81 more
Hi Seems I cant't attach a JSON file. so here it is pasted.
{
"template": {
"context": "/edocsns:energy_document",
"enabled": true,
"pathNamespace": [
{
"prefix": "edocsns",
"namespaceUri": "http://ihsenergy.com/energy_document"
},
{
"prefix": "cmnns",
"namespaceUri": "http://ihsenergy.com/common"
},
{
"prefix": "avgeo",
"namespaceUri": "http://ihsenergy.com/common/advanced-geo"
}
],
"collections": [
"energy_document"
],
"rows": [
{
"schemaName": "edocs",
"viewName": "energy_document",
"columns": [
{
"name": "edoc_id",
"scalarType": "string",
"val": "edocsns:attributes_group/edocsns:edoc_id",
"nullable": false
},
{
"name": "uwi",
"scalarType": "string",
"val": "edocsns:attributes_group/edocsns:uwi",
"nullable": true
},
{
"name": "text_content",
"scalarType": "string",
"val": "edocsns:text_content",
"nullable": true
},
{
"name": "file_size",
"scalarType": "long",
"val": "edocsns:attributes_group/edocsns:file_size",
"nullable": true
},
{
"name": "pdf_page_count",
"scalarType": "int",
"val": "edocsns:attributes_group/edocsns:pdf_page_count",
"nullable": true
},
{
"name": "confidential_type",
"scalarType": "string",
"val": "edocsns:attributes_group/edocsns:confidential_type",
"nullable": true
},
{
"name": "profile_type",
"scalarType": "string",
"val": "edocsns:attributes_group/edocsns:profile_type",
"nullable": true
},
{
"name": "status_type",
"scalarType": "string",
"val": "edocsns:attributes_group/edocsns:status_type",
"nullable": true
},
{
"name": "current_status",
"scalarType": "string",
"val": "edocsns:attributes_group/edocsns:current_status",
"nullable": true
},
{
"name": "rig_release_date",
"scalarType": "date",
"val": "edocsns:attributes_group/edocsns:rig_release_date",
"nullable": true
},
{
"name": "final_drill_date",
"scalarType": "date",
"val": "edocsns:attributes_group/edocsns:final_drill_date",
"nullable": true
},
{
"name": "completion_date",
"scalarType": "date",
"val": "edocsns:attributes_group/edocsns:completion_date",
"nullable": true
},
{
"name": "drill_td",
"scalarType": "decimal",
"val": "edocsns:attributes_group/edocsns:drill_td",
"nullable": true
},
{
"name": "well_name",
"scalarType": "string",
"val": "edocsns:attributes_group/edocsns:well_name",
"nullable": true
},
{
"name": "well_number",
"scalarType": "string",
"val": "edocsns:attributes_group/edocsns:well_number",
"nullable": true
},
{
"name": "current_licensee",
"scalarType": "string",
"val": "edocsns:attributes_group/edocsns:current_licensee",
"nullable": true
},
{
"name": "operator",
"scalarType": "string",
"val": "edocsns:attributes_group/edocsns:operator",
"nullable": true
},
{
"name": "license_number",
"scalarType": "string",
"val": "edocsns:attributes_group/edocsns:license_number",
"nullable": true
},
{
"name": "ancillary_name",
"scalarType": "string",
"val": "edocsns:attributes_group/edocsns:ancillary_name",
"nullable": true
},
{
"name": "region_name",
"scalarType": "string",
"val": "cmnns:location_group/cmnns:region/cmnns:region_name",
"nullable": true
},
{
"name": "country_name",
"scalarType": "string",
"val": "cmnns:location_group/cmnns:region/cmnns:country/cmnns:country_name",
"nullable": true
},
{
"name": "province_state_name",
"scalarType": "string",
"val": "cmnns:location_group/cmnns:region/cmnns:country/cmnns:provice_state/cmnns:province_state_name",
"nullable": true
},
{
"name": "document_type",
"scalarType": "string",
"val": "edocsns:attributes_group/edocsns:document_type",
"nullable": true
},
{
"name": "sub_document_type",
"scalarType": "string",
"val": "edocsns:attributes_group/edocsns:sub_document_type",
"nullable": true
},
{
"name": "pas_unit",
"scalarType": "string",
"val": "edocsns:attributes_group/edocsns:pas_unit",
"nullable": true
},
{
"name": "top_interval_log",
"scalarType": "decimal",
"val": "edocsns:attributes_group/edocsns:top_interval_log",
"nullable": true
},
{
"name": "base_interval_log",
"scalarType": "decimal",
"val": "edocsns:attributes_group/edocsns:base_interval_log",
"nullable": true
},
{
"name": "top_interval_tvd",
"scalarType": "decimal",
"val": "edocsns:attributes_group/edocsns:top_interval_tvd",
"nullable": true
},
{
"name": "base_interval_tvd",
"scalarType": "decimal",
"val": "edocsns:attributes_group/edocsns:base_interval_tvd",
"nullable": true
},
{
"name": "test_start_date",
"scalarType": "date",
"val": "edocsns:attributes_group/edocsns:test_start_date",
"nullable": true
},
{
"name": "test_end_date",
"scalarType": "date",
"val": "edocsns:attributes_group/edocsns:test_end_date",
"nullable": true
},
{
"name": "surface_latitude",
"scalarType": "decimal",
"val": "avgeo:spatial_group/avgeo:well_surface/@avgeo:lat",
"nullable": true
},
{
"name": "surface_longitude",
"scalarType": "decimal",
"val": "avgeo:spatial_group/avgeo:well_surface/@avgeo:lon",
"nullable": true
},
{
"name": "bottom_hole_latitude",
"scalarType": "decimal",
"val": "avgeo:spatial_group/avgeo:well_bottom/@avgeo:lat",
"nullable": true
},
{
"name": "bottom_hole_longitude",
"scalarType": "decimal",
"val": "avgeo:spatial_group/avgeo:well_bottom/@avgeo:lon",
"nullable": true
},
{
"name": "surface_location_datum",
"scalarType": "string",
"val": "edocsns:attributes_group/edocsns:surface_location_datum",
"nullable": true
},
{
"name": "bottom_location_datum",
"scalarType": "string",
"val": "edocsns:attributes_group/edocsns:bottom_location_datum",
"nullable": true
}
]
}
]
}
}
I haven't been able to reproduce this yet. I created a vanilla app with a REST server, a content database, and a schema database. Then I tossed the above TDE into src/main/ml-schemas/tde/my-tde.json.
When I deploy the app, and/or run mlLoadModules, I see the following logging:
Executing command [com.marklogic.appdeployer.command.schemas.LoadSchemasCommand] with sort order [350]
Loading schemas into database myApp-schemas from: /Users/rrudin/dev/workspace/ml-gradle/build/test/src/main/ml-schemas
TDE templates loaded into myApp-schemas will be validated against content database myApp-content
Initializing ExecutorService
TDE template passed validation: /Users/rrudin/dev/workspace/ml-gradle/build/test/src/main/ml-schemas/tde/my-tde.json
Writing 1 files
Writing: /tde/my-tde.json
Shutting down ExecutorService
Finished loading schemas from: /Users/rrudin/dev/workspace/ml-gradle/build/test/src/main/ml-schemas
There was a change to TDE validation in 3.16.0 - it's in the library ml-javaclient-util - https://github.com/marklogic-community/ml-javaclient-util/issues/118 . But I don't think that would have caused a regression - it involved fixing how the TDE string is passed to ML to be evaluated.
Try adding the following to gradle.properties:
mlTdeValidationEnabled=false
And see if you still get the same error. If you don't, then that isolates the error to the TDE validation feature. But I'd still need a way to reproduce that. If you get to that point, ping me and I'll give you an email address to try to send a copy of your project that is sufficient for producing the error.
One other question - if you deploy the app from scratch, do you get the error too?
HI. Thanks for looking into this.
So the good news is that when I turn off the validation it works fine.
Is there anything I can do to help to try and debug it. Is there a way to step through it in eclipse?
Thanks Trevor
Do you get the error when you deploy the app to a fresh ML instance, or only when you redeploy the app/schemas?
Some other questions:
- What version of MarkLogic?
- Is this a Data Hub project?
- Are you using a custom REST rewriter?
Are you able to share your entire project? I can give you an email address to send it to. I'm thinking there is some other resource/artifact in your project that may result in the error. I'd like to get to the bottom of this because otherwise, I'm leaning towards disabling TDE validation by default. But I'd like to keep it enabed by default as it's more annoying to find out about an invalid TDE template after it's been loaded.
I am using 9.0.10-3
This is not a data hub project .
I am not using a custom Rest Rewriter
@dunntrevor I apologize for the delay on this. If you're still able to send the project, you can email it to my gmail address - rjrudin.
Hi @dunntrevor - is this still an issue? Was hoping to get the project from you to reproduce. A bare minimum project that reproduces it would be preferable.