spring-cloud-dataflow icon indicating copy to clipboard operation
spring-cloud-dataflow copied to clipboard
verified publisher icon spring-cloud

Exception while running prometheus-rsocket-proxy with docker compose

Open bhcha opened this issue 1 year ago • 4 comments

Description: There is an exception raised from the prometheus rsocket proxy. This keeps on repeating at 10 sec intervals:

prometheus-rsocket-proxy    | 2024-12-23T05:08:17.202Z ERROR 1 --- [or-http-epoll-4] a.w.r.e.AbstractErrorWebExceptionHandler : [d5cc7e97-4]  500 Server Error for HTTP GET "/metrics/proxy"
prometheus-rsocket-proxy    |
prometheus-rsocket-proxy    | java.lang.ClassCastException: class io.prometheus.metrics.model.snapshots.SummarySnapshot$SummaryDataPointSnapshot cannot be cast to class io.prometheus.metrics.model.snapshots.HistogramSnapshot$HistogramDataPointSnapshot (io.prometheus.metrics.model.snapshots.SummarySnapshot$SummaryDataPointSnapshot and io.prometheus.metrics.model.snapshots.HistogramSnapshot$HistogramDataPointSnapshot are in unnamed module of loader org.springframework.boot.loader.launch.LaunchedClassLoader @372f7a8d)
prometheus-rsocket-proxy    |   at io.prometheus.metrics.expositionformats.PrometheusTextFormatWriter.writeHistogram(PrometheusTextFormatWriter.java:138) ~[prometheus-metrics-exposition-formats-1.2.1.jar:na]
prometheus-rsocket-proxy    |   Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
prometheus-rsocket-proxy    | Error has been observed at the following site(s):
prometheus-rsocket-proxy    |   *__checkpoint ? HTTP GET "/metrics/proxy" [ExceptionHandlingWebHandler]
prometheus-rsocket-proxy    | Original Stack Trace:
prometheus-rsocket-proxy    |           at io.prometheus.metrics.expositionformats.PrometheusTextFormatWriter.writeHistogram(PrometheusTextFormatWriter.java:138) ~[prometheus-metrics-exposition-formats-1.2.1.jar:na]
prometheus-rsocket-proxy    |           at io.prometheus.metrics.expositionformats.PrometheusTextFormatWriter.write(PrometheusTextFormatWriter.java:68) ~[prometheus-metrics-exposition-formats-1.2.1.jar:na]
prometheus-rsocket-proxy    |           at io.micrometer.prometheusmetrics.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:167) ~[micrometer-registry-prometheus-1.13.7.jar:1.13.7]
prometheus-rsocket-proxy    |           at io.micrometer.prometheusmetrics.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:163) ~[micrometer-registry-prometheus-1.13.7.jar:1.13.7]
prometheus-rsocket-proxy    |           at io.micrometer.prometheusmetrics.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:136) ~[micrometer-registry-prometheus-1.13.7.jar:1.13.7]
prometheus-rsocket-proxy    |           at io.micrometer.prometheusmetrics.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:124) ~[micrometer-registry-prometheus-1.13.7.jar:1.13.7]
prometheus-rsocket-proxy    |           at io.micrometer.prometheus.rsocket.PrometheusController.proxyMetrics(PrometheusController.java:156) ~[prometheus-rsocket-proxy-server-2.0.0-M2.jar:2.0.0-M2]
prometheus-rsocket-proxy    |           at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
prometheus-rsocket-proxy    |           at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
prometheus-rsocket-proxy    |           at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
prometheus-rsocket-proxy    |           at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
prometheus-rsocket-proxy    |           at org.springframework.web.reactive.result.method.InvocableHandlerMethod.lambda$invoke$0(InvocableHandlerMethod.java:198) ~[spring-webflux-6.1.14.jar:6.1.14]
prometheus-rsocket-proxy    |           at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:153) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:63) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:241) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:204) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.MonoFlatMap$FlatMapMain.onComplete(MonoFlatMap.java:189) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.Operators.complete(Operators.java:137) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:121) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.Mono.subscribe(Mono.java:4576) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:265) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:259) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:865) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:139) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.request(Operators.java:2331) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.request(FluxConcatMapNoPrefetch.java:339) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.MonoNext$NextSubscriber.request(MonoNext.java:108) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2241) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.MonoNext$NextSubscriber.onSubscribe(MonoNext.java:70) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onSubscribe(FluxConcatMapNoPrefetch.java:164) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:201) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:83) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.Mono.subscribe(Mono.java:4576) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:265) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.core.publisher.MonoDeferContextual.subscribe(MonoDeferContextual.java:55) ~[reactor-core-3.6.11.jar:3.6.11]
prometheus-rsocket-proxy    |           at reactor.netty.http.server.HttpServer$HttpServerHandle.onStateChange(HttpServer.java:1176) ~[reactor-netty-http-1.1.23.jar:1.1.23]
prometheus-rsocket-proxy    |           at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:716) ~[reactor-netty-core-1.1.23.jar:1.1.23]
prometheus-rsocket-proxy    |           at reactor.netty.transport.ServerTransport$ChildObserver.onStateChange(ServerTransport.java:481) ~[reactor-netty-core-1.1.23.jar:1.1.23]
prometheus-rsocket-proxy    |           at reactor.netty.http.server.HttpServerOperations.handleDefaultHttpRequest(HttpServerOperations.java:834) ~[reactor-netty-http-1.1.23.jar:1.1.23]
prometheus-rsocket-proxy    |           at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:779) ~[reactor-netty-http-1.1.23.jar:1.1.23]
prometheus-rsocket-proxy    |           at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:115) ~[reactor-netty-core-1.1.23.jar:1.1.23]
prometheus-rsocket-proxy    |           at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:262) ~[reactor-netty-http-1.1.23.jar:1.1.23]
prometheus-rsocket-proxy    |           at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) ~[netty-transport-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) ~[netty-transport-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799) ~[netty-transport-classes-epoll-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501) ~[netty-transport-classes-epoll-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399) ~[netty-transport-classes-epoll-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.114.Final.jar:4.1.114.Final]
prometheus-rsocket-proxy    |           at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]

Release versions:

{
  "featureInfo": {
    "analyticsEnabled": true,
    "streamsEnabled": true,
    "tasksEnabled": true,
    "schedulesEnabled": false,
    "monitoringDashboardType": "NONE"
  },
  "versionInfo": {
    "implementation": {
      "name": "spring-cloud-dataflow-server",
      "version": "2.11.3-SNAPSHOT"
    },
    "core": {
      "name": "Spring Cloud Data Flow Core",
      "version": "2.11.3-SNAPSHOT"
    },
    "dashboard": {
      "name": "Spring Cloud Dataflow UI",
      "version": "3.4.3-SNAPSHOT"
    },
    "shell": {
      "name": "Spring Cloud Data Flow Shell",
      "version": "2.11.3-SNAPSHOT",
      "url": "https://repo.spring.io/snapshot/org/springframework/cloud/spring-cloud-dataflow-shell/2.11.3-SNAPSHOT/spring-cloud-dataflow-shell-2.11.3-SNAPSHOT.jar"
    }
  },
  "securityInfo": {
    "authenticationEnabled": false,
    "authenticated": false,
    "username": null,
    "roles": []
  },
  "runtimeEnvironment": {
    "appDeployer": {
      "deployerImplementationVersion": "2.11.3-SNAPSHOT",
      "deployerName": "Spring Cloud Skipper Server",
      "deployerSpiVersion": "2.11.3-SNAPSHOT",
      "javaVersion": "11.0.19",
      "platformApiVersion": "",
      "platformClientVersion": "",
      "platformHostVersion": "",
      "platformSpecificInfo": {
        "default": "local"
      },
      "platformType": "Skipper Managed",
      "springBootVersion": "2.7.18",
      "springVersion": "5.3.34"
    },
    "taskLaunchers": [
      {
        "deployerImplementationVersion": "unknown",
        "deployerName": "LocalTaskLauncher",
        "deployerSpiVersion": "unknown",
        "javaVersion": "11.0.19",
        "platformApiVersion": "Linux 3.10.0-1160.95.1.el7.x86_64",
        "platformClientVersion": "3.10.0-1160.95.1.el7.x86_64",
        "platformHostVersion": "3.10.0-1160.95.1.el7.x86_64",
        "platformSpecificInfo": {},
        "platformType": "Local",
        "springBootVersion": "2.7.18",
        "springVersion": "5.3.34"
      }
    ]
  },
  "monitoringDashboardInfo": {
    "url": "",
    "refreshInterval": 15,
    "dashboardType": "NONE",
    "source": "default-scdf-source"
  },
  "gitAndBuildInfo": {
    "git": {
      "branch": "main",
      "commit": {
        "id": "ee585df",
        "time": "2024-05-21T11:43:47Z"
      }
    },
    "build": {
      "artifact": "spring-cloud-dataflow-server",
      "name": "Spring Cloud Data Flow Server",
      "time": "2024-05-22T02:30:35.832Z",
      "version": "2.11.3-SNAPSHOT",
      "group": "org.springframework.cloud"
    }
  },
  "_links": {
    "self": {
      "href": "http://localhost:9393/about"
    }
  }
}

Steps to reproduce: I used the information from the URL below verbatim, including the dc.sh. https://github.com/spring-cloud/spring-cloud-dataflow/tree/main/src/docker-compose

./dc.sh kafka mysql prometheus up

Screenshots: Uploading error.png…

Additional context:

bhcha avatar Dec 23 '24 05:12 bhcha

I would suggest using latest releases:

export DATAFLOW_VERSION=2.11.5
export SKIPPER_VERSION=2.11.5
./dc.sh kafka mysql prometheus up

corneil avatar Dec 30 '24 11:12 corneil

@corneil I did what you suggested, but I'm still having issues. I tried again with the image below

springcloud/spring-cloud-dataflow-server:2.11.5-SNAPSHOT
springcloud/spring-cloud-skipper-server:2.11.5-SNAPSHOT
or
springcloud/spring-cloud-dataflow-server:2.11.5-SNAPSHOT-jdk17
springcloud/spring-cloud-skipper-server:2.11.5-SNAPSHOT-jdk17

bhcha avatar Dec 31 '24 07:12 bhcha

You are using snapshot builds. Why don't you use a release build?

corneil avatar Dec 31 '24 11:12 corneil

Oh I'm so sorry, but I'm having the same problem with the release version instead of the snapshot version. It might be a problem on my server, so I'll try it on another server soon and let you know.

Happy New Year!!

bhcha avatar Dec 31 '24 11:12 bhcha