elasticsearch-java icon indicating copy to clipboard operation
elasticsearch-java copied to clipboard

Published 20 hours ago • verified publisher icon elastic

MissingRequiredPropertyException: Missing required property 'PathHierarchyTokenizer.bufferSize'

Open fs-chris opened this issue 2 years ago • 1 comments

Java API client version

8.4.2

Java version

17

Elasticsearch Version

8.4.2

Problem description

Have an index with a mapping using a path_hierarchy tokenizer like

...	"settings": {
		"analysis": {
			"tokenizer": {
				"path-tokenizer": {
					"type": "path_hierarchy",
					"delimiter": ">"
				}
			}
		}
	}

Trying to get settings for that index (client.indices().getSettings(...)) results in the following exception. The same happens if any of the other optional tokenizer arguments are missing.

co.elastic.clients.json.JsonpMappingException: Error deserializing co.elastic.clients.elasticsearch._types.analysis.TokenizerDefinition: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'PathHierarchyTokenizer.bufferSize' (JSON path: ['njams-data_2022-09-29'].settings.index.analysis.tokenizer['path-tokenizer']) (line no=1, column no=632, offset=-1)
	at co.elastic.clients.json.JsonpMappingException.from0(JsonpMappingException.java:134)
	at co.elastic.clients.json.JsonpMappingException.from(JsonpMappingException.java:121)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:206)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
	at co.elastic.clients.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:53)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:48)
	at co.elastic.clients.json.UnionDeserializer$SingleMemberHandler.deserialize(UnionDeserializer.java:74)
	at co.elastic.clients.json.UnionDeserializer.deserialize(UnionDeserializer.java:291)
	at co.elastic.clients.json.UnionDeserializer.deserialize(UnionDeserializer.java:258)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:347)
	at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:331)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:180)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:180)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:180)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:180)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:347)
	at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:331)
	at co.elastic.clients.elasticsearch.indices.GetIndicesSettingsResponse.lambda$createGetIndicesSettingsResponseDeserializer$0(GetIndicesSettingsResponse.java:175)
	at co.elastic.clients.json.JsonpDeserializer$3.deserialize(JsonpDeserializer.java:126)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.transport.rest_client.RestClientTransport.decodeResponse(RestClientTransport.java:328)
	at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:294)
	at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147)
	at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getSettings(ElasticsearchIndicesClient.java:1106)
	at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getSettings(ElasticsearchIndicesClient.java:1123)
        ....

Caused by: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'PathHierarchyTokenizer.bufferSize'
	at co.elastic.clients.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:76)
	at co.elastic.clients.elasticsearch._types.analysis.PathHierarchyTokenizer.<init>(PathHierarchyTokenizer.java:66)
	at co.elastic.clients.elasticsearch._types.analysis.PathHierarchyTokenizer.<init>(PathHierarchyTokenizer.java:50)
	at co.elastic.clients.elasticsearch._types.analysis.PathHierarchyTokenizer$Builder.build(PathHierarchyTokenizer.java:215)
	at co.elastic.clients.elasticsearch._types.analysis.PathHierarchyTokenizer$Builder.build(PathHierarchyTokenizer.java:148)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:80)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:197)
	... 78 common frames omitted

fs-chris avatar Sep 27 '22 06:09 fs-chris

The workaround with DANGEROUS_disableRequiredPropertiesCheck(true) does not work here because some properties are defined as primitives which leads to NullPointerException in the following.

Caused by: java.lang.NullPointerException: Cannot invoke "java.lang.Integer.intValue()" because the return value of "co.elastic.clients.util.ApiTypeHelper.requireNonNull(Object, Object, String)" is null
        at deployment.App.war//co.elastic.clients.elasticsearch._types.analysis.PathHierarchyTokenizer.<init>(PathHierarchyTokenizer.java:66)
        at deployment.App.war//co.elastic.clients.elasticsearch._types.analysis.PathHierarchyTokenizer.<init>(PathHierarchyTokenizer.java:50)
        at deployment.App.war//co.elastic.clients.elasticsearch._types.analysis.PathHierarchyTokenizer$Builder.build(PathHierarchyTokenizer.java:215)
        at deployment.App.war//co.elastic.clients.elasticsearch._types.analysis.PathHierarchyTokenizer$Builder.build(PathHierarchyTokenizer.java:148)
        at deployment.App.war//co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:80)
        at deployment.App.war//co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
        at deployment.App.war//co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
        at deployment.App.war//co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:197)
        ... 197 more

fs-chris avatar Oct 20 '22 10:10 fs-chris

Hi @fs-chris, did you find a workaround ? I have the same issue.

mvanzalu avatar Oct 20 '23 20:10 mvanzalu

The workaround I use here is to set all properties of the path-tokenizer element using default values. The get request then also includes all properties, avoiding the missing-property error.

fs-chris avatar Oct 23 '23 06:10 fs-chris

It works, thank you !

mvanzalu avatar Oct 23 '23 10:10 mvanzalu

We hit this issue as well.

Frankly, it seems pretty backwards to me that the Java client library requires you to explicitly set all values, when the documentation of the API clearly indicates them to be optional with defaulting behaviour on the server.

The old Elasticsearch Java library did not require this superfluous setting of default values.

oliverlockwood avatar Dec 18 '23 08:12 oliverlockwood

Hello, thank you for the report! I tested this using the most recent version of the client and I couldn't reproduce this issue, so either this was already fixed or there's something I'm missing. Let me know if updating the java client to 8.12.X fixes this for you as well :)

l-trotta avatar Feb 23 '24 11:02 l-trotta

@l-trotta I don't know how you were trying to reproduce this, but I just tried with the very latest release of the Java client, and the bug remains. This is not fixed.

From mvn dependency:tree:

co.elastic.clients:elasticsearch-java:jar:8.12.2:compile

From stacktrace:

...
Caused by: co.elastic.clients.json.JsonpMappingException: Error deserializing co.elastic.clients.elasticsearch._types.analysis.TokenizerDefinition: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'PathHierarchyTokenizer.bufferSize' (JSON path: template.settings.analysis.tokenizer.path_tokenizer) (line no=1, column no=348, offset=347)
	at co.elastic.clients.json.JsonpMappingException.from0(JsonpMappingException.java:134)
	at co.elastic.clients.json.JsonpMappingException.from(JsonpMappingException.java:121)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:218)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:148)
	at co.elastic.clients.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:53)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:48)
	at co.elastic.clients.json.UnionDeserializer$SingleMemberHandler.deserialize(UnionDeserializer.java:75)
	at co.elastic.clients.json.UnionDeserializer.deserialize(UnionDeserializer.java:304)
	at co.elastic.clients.json.UnionDeserializer.deserialize(UnionDeserializer.java:259)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:349)
	at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:333)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:77)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:78)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:192)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:148)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:77)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:78)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:192)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:148)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:77)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:78)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:192)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:148)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:77)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:78)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:192)
	at co.elastic.clients.util.WithJsonObjectBuilderBase.withJson(WithJsonObjectBuilderBase.java:54)
	at co.elastic.clients.json.WithJson.withJson(WithJson.java:57)
	at com.oliverlockwood.myapp.EsIndexClient.lambda$setIndexTemplate$25(EsIndexClient.java:529)
	at co.elastic.clients.elasticsearch.indices.PutIndexTemplateRequest.of(PutIndexTemplateRequest.java:115)
	at com.oliverlockwood.myapp.EsIndexClient.setIndexTemplate(EsIndexClient.java:526)
	at com.oliverlockwood.myapp.EsIndexManager.setIndexTemplate(EsIndexManager.java:109)
	at com.oliverlockwood.myapp.EsIndexManager.initTemplateIndexAndAlias(EsIndexManager.java:75)
	at com.oliverlockwood.myapp.ObjectIndexingService.createIndices(ObjectIndexingService.java:109)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMethod.invoke(InitDestroyAnnotationBeanPostProcessor.java:457)
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:401)
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:219)
	... 57 common frames omitted
Caused by: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'PathHierarchyTokenizer.bufferSize'
	at co.elastic.clients.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:76)
	at co.elastic.clients.elasticsearch._types.analysis.PathHierarchyTokenizer.<init>(PathHierarchyTokenizer.java:78)
	at co.elastic.clients.elasticsearch._types.analysis.PathHierarchyTokenizer.<init>(PathHierarchyTokenizer.java:61)
	at co.elastic.clients.elasticsearch._types.analysis.PathHierarchyTokenizer$Builder.build(PathHierarchyTokenizer.java:231)
	at co.elastic.clients.elasticsearch._types.analysis.PathHierarchyTokenizer$Builder.build(PathHierarchyTokenizer.java:163)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:80)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:78)
	at co.elastic.clients.json.ObjectDeserialize

As my stacktrace shows, I'm trying to create an index template. Perhaps you were trying to create an index directly, and the behaviour there is different?

oliverlockwood avatar Feb 24 '24 17:02 oliverlockwood

Thanks for the quick response! Using the PutIndexTemplateRequest I'm not getting the same error as you are, but I'm getting another one, so there's definitely something wrong with this request. Could you please give me more information on how you're calling the API? It would really help, thanks again.

l-trotta avatar Feb 24 '24 21:02 l-trotta

@l-trotta since the issue occurs even before sending a request to the Elasticsearch server, it is trivial to reproduce it in unit tests.

Therefore, I have created a small repository https://github.com/oliverlockwood/elasticsearch-java-410 which you can clone and run mvn clean install on, and voilà, you'll have the bug reproducing locally!

(The example index template mapping defined in the UT is as minimal as possible for simplicity, while still being valid; obviously in itself it is meaningless as the tokenizer named path_tokenizer being defined is not used anywhere; in our real code, we make use of this in a custom analyzer which is then used for certain dynamic_templates and thereby in our actual index mappings.)

Let me know if you need anything further. Thanks.

oliverlockwood avatar Feb 25 '24 08:02 oliverlockwood

Thank you @oliverlockwood for providing the repo, turns out I wasn't using the correct parameter when writing the request using the java client lambdas ^^" here's how to write the request correctly without using json, for those who are curious:

        PutIndexTemplateRequest.of(pitr -> pitr
            .name("template-things")
            .indexPatterns("things-*")
            .template(t -> t
                .settings(s -> s
                    .analysis(a -> a
                        .tokenizer("path-tokenizer", tk -> tk
                            .definition(def -> def
                                .pathHierarchy(ph -> ph)))))));

As for the issue, it was recently solved and will be fixed in the next release. Thanks you for your patience!

l-trotta avatar Mar 05 '24 13:03 l-trotta

You're welcome, @l-trotta.

If you believe the issue to have been fixed, please would you let me know in which client version the fix will be released? It would be a good thing to have recorded in the issue comments / milestones for posterity, anyway... 😃

oliverlockwood avatar Mar 05 '24 21:03 oliverlockwood

The fix will be part of the new 8.13 version, which will be released around the end of March. Unfortunately it's not that easy for us to keep track of api-spec updates affecting the java client, since the api-spec is used by many other teams other than us.

l-trotta avatar Mar 07 '24 09:03 l-trotta