spring-ai icon indicating copy to clipboard operation
spring-ai copied to clipboard

TokenTextSplitter fails with "Failed to deserialize the error entity"

Open khauser opened this issue 7 months ago • 0 comments
trafficstars

Bug description The chunking process of some "test execution"-files fails always at the same point. The file it's seems to be is about 8MB big testrunner.log.

Is this a problem of the Azure Search API or of spring-ai?

I tried several variations of TokenTextSplitter parameters.

2025-04-16T16:56:37.824+02:00[0;39m [32m INFO[0;39m [35m37196[0;39m [2m--- [ai] [nio-6060-exec-1] [0;39m[36mc.i.i.a.c.AnalyzeTestExecutionController[0;39m [2m:[0;39m {
  "id" : "d05a595b-1d1f-4a6e-ad5a-b193891e127c",
  "text" : "The testrunner is waiting for the AS: http://iste-run-tp5601-ts2450244-icm-as:7743/status/ReadinessProbe\n/intershop/bin/testrunner.sh: 15: waitforurl: not found\nAS was started: http://iste-run-tp5601-ts2450244-icm-as:7743/status/ReadinessProbe\n\nCommand Line:\njava -showversion -XX:+UseG1GC -Xms768m -Xmx1536m -Detest.home=/intershop/cartridges/etest/release -Dserver.name=testrunner -Djava.system.class.loader=com.intershop.beehive.runtime.IntershopClassLoader -Dintershop.ServerConfig=/intershop/serverdir.properties -javaagent:/intershop/cartridges/runtime/release/lib/ish-runtime.jar -Djdk.tls.client.protocols=TLSv1.2 -cp /intershop/cartridges/runtime/release/lib/ish-runtime.jar:/intershop/system-conf/cluster:/intershop/testrunner/lib/* -Dis.cartridgesClasspathLayout=release -Dis.cartridgeRepositories=/intershop/cartridges:/intershop/customizations -Dis.libraryRepositories=/intershop/lib:/intershop/customizations -Dis.cartridges=\"app_bo_test\" com.intershop.testrunner.IshTestrunner -s=tests.remote.com.intershop.smc.suite.SMC00TestSuite -o=/data/testplans/ff3c226c-31a4-46d3-b867-d2d2701397d7/testsuites/2450244/workspace/helm/icm-2.13.9/result\n\n[0.025s][warning][cds] Archived non-system classes are disabled because the java.system.class.loader property is specified (value = \"com.intershop.beehive.runtime.IntershopClassLoader\"). To use archived non-system classes, this property must not be set\nCan't find location '/intershop/customizations' for 'CARTRIDGES'\nCan't find location '/intershop/customizations' for 'LIBRARIES'\n243 library versions from library list files\nINFO: Agent Main of com.intershop.beehive.runtime.internal.agent.Agent first time executed\nopenjdk version \"21.0.6\" 2025-01-21 LTS\nOpenJDK Runtime Environment Temurin-21.0.6+7 (build 21.0.6+7-LTS)\nOpenJDK 64-Bit Server VM Temurin-21.0.6+7 (build 21.0.6+7-LTS, mixed mode, sharing)\n[2025-04-14 22:31:06.483 UTC] [main] INFO  c.intershop.testrunner.IshTestrunner - Testrunner output direcotry :/data/testplans/ff3c226c-31a4-46d3-b867-d2d2701397d7/testsuites/2450244/workspace/helm/icm-2.13.9/result was created :false\n[2025-04-14 22:31:06.574 UTC] [main] INFO  c.intershop.testrunner.SuiteHandler - Following tests were found in Suite: [tests.remote.com.intershop.smc.operation.servergroup.general.ServerGroupTest, tests.remote.com.intershop.smc.operation.site.general.SiteManagementTest, tests.remote.com.intershop.smc.smc.application.general.ApplicationTest, tests.remote.com.intershop.smc.smc.file.browsing.FileBrowserTest, tests.remote.com.intershop.smc.smc.file.download.FileDownloadTest, tests.remote.com.intershop.smc.smc.file.listing.FileListTest, tests.remote.com.intershop.smc.smc.installation.cluster.ClusterSnapshotTest, tests.remote.com.intershop.smc.smc.installation.files.InstallationFileTest, tests.remote.com.intershop.smc.smc.installation.heapdump.HeapdumpTest, tests.remote.com.intershop.smc.smc.installation.threaddump.ThreaddumpTest, tests.remote.com.intershop.smc.smc.orm.caching.ORMCacheTest, tests.remote.com.intershop.smc.smc.process.general.ProcessOverviewTest, tests.remote.com.intershop.smc.smc.process.locking.LockingTest, tests.remote.com.intershop.smc.smc.process.search.ProcessSearchTest, tests.remote.com.intershop.smc.smc.server.configuration.ApplicationServerConfigurationTest, tests.remote.com.intershop.smc.smc.server.logging.LoggingTest, tests.remote.com.intershop.smc.operation.servergroup.general.ServerGroupTest, tests.remote.com.intershop.smc.smc.serviceconfiguration.general.ServiceConfigurationTest, tests.remote.com.intershop.smc.smc.serviceconfiguration.listing.ServiceConfigurationListTest, tests.remote.com.intershop.smc.operation.site.general.SiteManagementTest, tests.remote.com.intershop.smc.smc.thread.listing.",
  "media" : null,
  "metadata" : {
    "fileName" : "istestrunner.log",
    "testPathId" : null,
    "filePath" : "workspace/helm/icm-2.13.9/result/istestrunner.log",
    "testSuiteId" : 2450244
  },
  "score" : null
}
[2m2025-04-16T17:05:13.306+02:00[0;39m [33m WARN[0;39m [35m37196[0;39m [2m--- [ai] [nio-6060-exec-1] [0;39m[36mc.a.c.i.s.HttpResponseBodyDecoder       [0;39m [2m:[0;39m Failed to deserialize the error entity.
Unsupported text Content-Type Type: class com.azure.search.documents.implementation.models.ErrorResponse
[2m2025-04-16T17:05:13.307+02:00[0;39m [31mERROR[0;39m [35m37196[0;39m [2m--- [ai] [nio-6060-exec-1] [0;39m[36mc.azure.search.documents.SearchClient   [0;39m [2m:[0;39m Status code 413, "The page was not displayed because the request entity is too large."
[2m2025-04-16T17:05:13.321+02:00[0;39m [32mDEBUG[0;39m [35m37196[0;39m [2m--- [ai] [nio-6060-exec-1] [0;39m[36mo.s.web.servlet.DispatcherServlet       [0;39m [2m:[0;39m Failed to complete request: com.azure.core.exception.HttpResponseException: Status code 413, "The page was not displayed because the request entity is too large."
[2m2025-04-16T17:05:13.327+02:00[0;39m [31mERROR[0;39m [35m37196[0;39m [2m--- [ai] [nio-6060-exec-1] [0;39m[36mo.a.c.c.C.[.[.[.[dispatcherServlet]     [0;39m [2m:[0;39m Servlet.service() for servlet [dispatcherServlet] in context with path [/vaadin-app] threw exception [Request processing failed: com.azure.core.exception.HttpResponseException: Status code 413, "The page was not displayed because the request entity is too large."] with root cause

java.lang.IllegalStateException: Unsupported text Content-Type Type: class com.azure.search.documents.implementation.models.ErrorResponse
	at com.azure.core.util.serializer.JacksonAdapter.deserializeText(JacksonAdapter.java:452) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.util.serializer.JacksonAdapter.lambda$deserialize$9(JacksonAdapter.java:353) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.util.serializer.JacksonAdapter.useAccessHelper(JacksonAdapter.java:488) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.util.serializer.JacksonAdapter.deserialize(JacksonAdapter.java:346) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.implementation.serializer.HttpResponseBodyDecoder.deserialize(HttpResponseBodyDecoder.java:176) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.implementation.serializer.HttpResponseBodyDecoder.deserializeBody(HttpResponseBodyDecoder.java:150) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.implementation.serializer.HttpResponseBodyDecoder.decodeByteArray(HttpResponseBodyDecoder.java:67) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.implementation.serializer.HttpResponseDecoder$HttpDecodedResponse.getDecodedBody(HttpResponseDecoder.java:93) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.implementation.http.rest.SyncRestProxy.ensureExpectedStatus(SyncRestProxy.java:131) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.implementation.http.rest.SyncRestProxy.handleRestReturnType(SyncRestProxy.java:211) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.implementation.http.rest.SyncRestProxy.invoke(SyncRestProxy.java:86) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.implementation.http.rest.RestProxyBase.invoke(RestProxyBase.java:124) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.http.rest.RestProxy.invoke(RestProxy.java:95) ~[azure-core-1.55.3.jar:1.55.3]
	at jdk.proxy2/jdk.proxy2.$Proxy146.indexSync(Unknown Source) ~[na:na]
	at com.azure.search.documents.implementation.DocumentsImpl.indexWithResponse(DocumentsImpl.java:778) ~[azure-search-documents-11.7.6.jar:11.7.6]
	at com.azure.search.documents.implementation.util.Utility.lambda$indexDocumentsWithResponse$1(Utility.java:160) ~[azure-search-documents-11.7.6.jar:11.7.6]
	at com.azure.search.documents.implementation.util.Utility.executeRestCallWithExceptionHandling(Utility.java:187) ~[azure-search-documents-11.7.6.jar:11.7.6]
	at com.azure.search.documents.implementation.util.Utility.indexDocumentsWithResponse(Utility.java:158) ~[azure-search-documents-11.7.6.jar:11.7.6]
	at com.azure.search.documents.SearchClient.indexDocumentsWithResponse(SearchClient.java:769) ~[azure-search-documents-11.7.6.jar:11.7.6]
	at com.azure.search.documents.SearchClient.uploadDocumentsWithResponse(SearchClient.java:422) ~[azure-search-documents-11.7.6.jar:11.7.6]
	at com.azure.search.documents.SearchClient.uploadDocuments(SearchClient.java:382) ~[azure-search-documents-11.7.6.jar:11.7.6]
	at org.springframework.ai.vectorstore.azure.AzureVectorStore.doAdd(AzureVectorStore.java:182) ~[spring-ai-azure-store-1.0.0-M7.jar:1.0.0-M7]
	at org.springframework.ai.vectorstore.observation.AbstractObservationVectorStore.lambda$add$1(AbstractObservationVectorStore.java:85) ~[spring-ai-vector-store-1.0.0-M7.jar:1.0.0-M7]
	at io.micrometer.observation.Observation.observe(Observation.java:498) ~[micrometer-observation-1.14.5.jar:1.14.5]
	at org.springframework.ai.vectorstore.observation.AbstractObservationVectorStore.add(AbstractObservationVectorStore.java:85) ~[spring-ai-vector-store-1.0.0-M7.jar:1.0.0-M7]
	at com.intershop.iste.ai.controller.AnalyzeTestExecutionController.processFileEntries(AnalyzeTestExecutionController.java:125) ~[main/:na]
	at com.intershop.iste.ai.controller.AnalyzeTestExecutionController.processFileEntries(AnalyzeTestExecutionController.java:94) ~[main/:na]
	at com.intershop.iste.ai.controller.AnalyzeTestExecutionController.processFileEntries(AnalyzeTestExecutionController.java:94) ~[main/:na]
	at com.intershop.iste.ai.controller.AnalyzeTestExecutionController.processFileEntries(AnalyzeTestExecutionController.java:94) ~[main/:na]
	at com.intershop.iste.ai.controller.AnalyzeTestExecutionController.processFileEntries(AnalyzeTestExecutionController.java:94) ~[main/:na]
	at com.intershop.iste.ai.controller.AnalyzeTestExecutionController.loadVectorData(AnalyzeTestExecutionController.java:68) ~[main/:na]
	at com.intershop.iste.ai.controller.AnalyzeTestExecutionController.loadTestPlan(AnalyzeTestExecutionController.java:78) ~[main/:na]
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:258) ~[spring-web-6.2.5.jar:6.2.5]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:191) ~[spring-web-6.2.5.jar:6.2.5]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118) ~[spring-webmvc-6.2.5.jar:6.2.5]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:986) ~[spring-webmvc-6.2.5.jar:6.2.5]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:891) ~[spring-webmvc-6.2.5.jar:6.2.5]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-6.2.5.jar:6.2.5]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089) ~[spring-webmvc-6.2.5.jar:6.2.5]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979) ~[spring-webmvc-6.2.5.jar:6.2.5]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) ~[spring-webmvc-6.2.5.jar:6.2.5]
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903) ~[spring-webmvc-6.2.5.jar:6.2.5]
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564) ~[tomcat-embed-core-10.1.39.jar:6.0]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) ~[spring-webmvc-6.2.5.jar:6.2.5]
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658) ~[tomcat-embed-core-10.1.39.jar:6.0]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) ~[tomcat-embed-websocket-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91) ~[spring-web-6.2.5.jar:6.2.5]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.2.5.jar:6.2.5]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-6.2.5.jar:6.2.5]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.2.5.jar:6.2.5]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-6.2.5.jar:6.2.5]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.2.5.jar:6.2.5]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:114) ~[spring-web-6.2.5.jar:6.2.5]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.2.5.jar:6.2.5]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-6.2.5.jar:6.2.5]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.2.5.jar:6.2.5]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:397) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:905) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]

[2m2025-04-16T17:05:13.346+02:00[0;39m [32mDEBUG[0;39m [35m37196[0;39m [2m--- [ai] [nio-6060-exec-1] [0;39m[36mo.s.web.servlet.DispatcherServlet       [0;39m [2m:[0;39m "ERROR" dispatch for GET "/vaadin-app/error?testPlanId=5601", parameters={masked}
[2m2025-04-16T17:05:13.371+02:00[0;39m [32mDEBUG[0;39m [35m37196[0;39m [2m--- [ai] [nio-6060-exec-1] [0;39m[36mo.s.w.s.v.ContentNegotiatingViewResolver[0;39m [2m:[0;39m Selected 'text/html' given [text/html, text/html;q=0.8]
[2m2025-04-16T17:05:13.381+02:00[0;39m [32mDEBUG[0;39m [35m37196[0;39m [2m--- [ai] [nio-6060-exec-1] [0;39m[36mo.s.web.servlet.DispatcherServlet       [0;39m [2m:[0;39m Exiting from "ERROR" dispatch, status 500

Environment Spring AI version: 1.0.0-M7 Java: Java(TM) SE Runtime Environment Oracle GraalVM 21.0.2+13.1 (build 21.0.2+13-LTS-jvmci-23.1-b30) Please provide as many details as possible: Spring AI version, Java version, which vector store you use if any, etc

Steps to reproduce build.gradle

    implementation group: 'com.azure', name: 'azure-search-documents', version: "11.7.6"
    implementation(group: 'org.springframework.ai', name: 'spring-ai-starter-model-azure-openai') {
        exclude group: 'com.azure', module: 'azure-search-documents'
    }
    implementation group: 'org.springframework.ai', name: 'spring-ai-autoconfigure-model-azure-openai'

    implementation group: 'org.springframework.ai', name: 'spring-ai-starter-vector-store-azure'
    implementation group: 'org.springframework.ai', name: 'spring-ai-autoconfigure-vector-store-azure'

    implementation group: 'org.springframework.ai', name: 'spring-ai-rag'
    implementation group: 'org.springframework.ai', name: 'spring-ai-azure-store'

    implementation group: 'com.azure', name: 'azure-core-http-okhttp', version: "1.12.10"

Controller.java

                try (InputStream inputStream = isteBIService.getLogFile(testSuiteId, entry.getPath());
                                ByteArrayOutputStream outputStream = new ByteArrayOutputStream())
                {

                    inputStream.transferTo(outputStream);

                    String content = outputStream.toString(StandardCharsets.UTF_8);

                    Document doc = Document.builder()
                                           .id("suite-" + testSuiteId + "/" + entry.getName())
                                           .text(content)
                                           .build();

                    //var tokenTextSplitter = TokenTextSplitter.builder().withChunkSize(800).build();
                    var tokenTextSplitter = new TokenTextSplitter(1024, 350, 5, 20000, true);
                    //var tokenTextSplitter = new TokenTextSplitter(512, 250, 5, 10000, true);
                    List<Document> splitDocuments = tokenTextSplitter.apply(List.of(doc));
                    for (Document splitDocument : splitDocuments)
                    {
                        splitDocument.getMetadata().put("fileName", entry.getName());
                        splitDocument.getMetadata().put("filePath", entry.getPath());
                        splitDocument.getMetadata().put("testSuiteId", testSuiteId);
                        splitDocument.getMetadata().put("testPathId", null);
                    }
                    log.info(new ObjectMapper().writerWithDefaultPrettyPrinter()
                                               .writeValueAsString(splitDocuments.get(0)));
                    store.add(splitDocuments);
                }
                catch(IOException e)
                {
                    throw new UncheckedIOException("Fehler beim Lesen des Log-Inhalts: " + entry.getPath(), e);
                }

Expected behavior A clear and concise description of what you expected to happen.

Minimal Complete Reproducible example Please provide a failing test or a minimal complete verifiable example that reproduces the issue. Bug reports that are reproducible will take priority in resolution over reports that are not reproducible.

khauser avatar Apr 16 '25 17:04 khauser