smallrye-reactive-messaging Kafka consume stopped without any exception log

Kafka consume stopped without any exception log

Open liuzhaozhao opened this issue 2 years ago • 11 comments

Problem description

My Kafka consumers will occasionally stop consuming, Consumers are distributed across multiple machine nodes, and each node is configured with multiple consuming threads (partitions>1). Once in a while, one of the machine nodes stops consuming (the other nodes do not stop). And there will be no consumer rebalancing. How can I eliminate this problem.

It happens about a couple of times a week, I have multiple topic comsumer, and it has happened in multiple topic comsumer.

Kafka Grafana Metric

Config

mp:
  messaging:
    incoming:
      location-traffic-in:
        connector: smallrye-kafka
        topic: location
        graceful-shutdown: true
        group:
          id: location-traffic
        key:
          deserializer: org.apache.kafka.common.serialization.StringDeserializer
        value:
          deserializer: org.apache.kafka.common.serialization.StringDeserializer
        auto:
          offset:
            reset: latest
        enable:
          auto:
            commit: true
        fetch:
          max:
            wait:
              ms: 10
        batch: true
        broadcast: false
        partitions: 3
        requests: 700
        max.poll.records: 200
        failure-strategy: ignore

Code

@Slf4j
@ApplicationScoped
public class LocationTrafficConsumer {
    @Incoming("location-traffic-in")
    @Acknowledgment(Acknowledgment.Strategy.NONE)
    @NonBlocking
    public Uni<List<Void>> consume(List<String> events) {
        try {
			// ....
            return Uni.join().all(uniList).andCollectFailures()
                    .onFailure().retry().atMost(3)
                    .onFailure().invoke(t -> LIMIT_EXCEPTION.tryAcquire(() -> log.warn("Deal Exception", t)))
                    .onFailure().recoverWithNull();
        } catch (Throwable e) {
            log.warn("Exception", e);
            return Uni.createFrom().nullItem();
        }
    }

}

Environment and dependency packages

System:Linux 4.4.0-116-generic
JDK:openjdk-17
Framework:Quarkus
Dependency:smallrye-reactive-messaging-kafka 3.13.0

Jul 13 '22 02:07 liuzhaozhao

Did you try removing the @Acknowledgment(Acknowledgment.Strategy.NONE). With that strategy, the messages are not backed, which means offsets are not committed. After a while, the application will consider you are facing a poison pill.

Also, producing a List<Void> is rather weird. Uni<Void> is probably what you want.

Jul 13 '22 04:07 cescoffier

Ok, I will modify and see the effect, thank you

Jul 13 '22 06:07 liuzhaozhao

The returning type Uni<List<Void>> is suspicious.

But the problem should not be the acknowledgement strategy. You are using enable.auto.commit=true so that's actually ok to use NONE. I'd be curious to see if you'd have the same behaviour with throttled commit strategy.

Plus if it's weird that you are not seeing any rebalances if the consuming stops.

Jul 13 '22 08:07 ozangunalp

@cescoffier I've modified it, but consumer is still stopped and there are no WARN, ERROR logs,Is there any other way to eliminate the problem?

@Slf4j
@ApplicationScoped
public class LocationTrafficConsumer {
    private static final RateLimit LIMIT_EXCEPTION = RateLimit.create(3);

    @Incoming("location-traffic-in")
    @NonBlocking
    public Uni<Void> consume(List<String> events) {
        try {
            List<Uni<Void>> uniList = new ArrayList<>();
            // some code
            return Uni.join().all(uniList).andCollectFailures().replaceWithVoid()
                    .onFailure().retry().atMost(3)
                    .onFailure().invoke(t -> LIMIT_EXCEPTION.tryAcquire(() -> log.warn("Deal Exception", t)))
                    .onFailure().recoverWithNull();
        } catch (Throwable e) {
            log.warn("Exception", e);
            return Uni.createFrom().voidItem();
        }
    }

}

Jul 13 '22 09:07 liuzhaozhao

@ozangunalp There are three nodes in total, and two of them are unconsumed, piling up and not being rebalanced. I'm going to add @Timeout(60000) to the consumption method to confirm if it's blocked in the business code.

location        46         4278824         4281325         2501            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        47         4024800         4027572         2772            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        48         3792332         3794083         1751            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        49         4292595         4294764         2169            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        50         3788027         3790107         2080            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        51         3827359         3828858         1499            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        52         3450834         3453229         2395            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        53         4621556         4623786         2230            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        54         3804134         3806224         2090            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        55         3781707         3783315         1608            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        56         4059864         4062223         2359            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        57         3977740         3979340         1600            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        58         3888166         3890252         2086            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        59         3628643         3630578         1935            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        60         4172245         4174164         1919            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        61         3812863         3814649         1786            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        62         4188870         4191503         2633            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        63         4225861         4227734         1873            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        64         3287542         3289051         1509            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        65         3862819         3864886         2067            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        66         3924515         3925978         1463            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        67         3343565         3344914         1349            kafka-consumer-location-traffic-in-0-a3b66c9c-692a-4e93-8a21-9b9bb9d8b3fe /10.202.14.10   kafka-consumer-location-traffic-in-0
location        112        3870943         3873304         2361            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        113        4027735         4029333         1598            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        114        3248248         3250567         2319            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        115        3713392         3714919         1527            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        116        3893649         3895859         2210            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        117        3967860         3969812         1952            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        118        3596639         3598356         1717            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        119        3502459         3504377         1918            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        120        3992535         3994313         1778            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        121        3518817         3520497         1680            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        122        3735594         3737273         1679            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        123        4467577         4470118         2541            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        124        3656595         3658490         1895            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        125        4305883         4307929         2046            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        126        3569122         3570813         1691            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        127        3796835         3798467         1632            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        128        3822095         3823658         1563            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        129        4010817         4015634         4817            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        130        3954040         3955517         1477            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        131        3735317         3737102         1785            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        132        3468510         3469966         1456            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        133        3738882         3741127         2245            kafka-consumer-location-traffic-in-1-ca6093d3-3d5f-41e6-bd42-5a5ea3a9004c /10.202.14.10   kafka-consumer-location-traffic-in-1
location        156        4283865         4286948         3083            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        157        3692689         3694787         2098            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        158        4434202         4436609         2407            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        159        4243867         4246072         2205            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        160        3317303         3319196         1893            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        161        3734397         3736535         2138            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        162        3381231         3382552         1321            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        163        3674888         3677016         2128            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        164        3684164         3686447         2283            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        165        3723886         3725830         1944            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        166        3711093         3713125         2032            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        167        3446924         3448910         1986            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        168        4098088         4099963         1875            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        169        3692472         3694165         1693            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        170        3640521         3642497         1976            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        171        3473453         3475000         1547            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        172        4013168         4015206         2038            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        173        4036263         4038447         2184            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        174        3735321         3737243         1922            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        175        4101616         4103979         2363            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        176        4274241         4277209         2968            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        177        4226410         4228414         2004            kafka-consumer-location-traffic-in-2-7f7026c3-5650-4382-805d-441ae4171563 /10.202.14.10   kafka-consumer-location-traffic-in-2
location        23         3818977         3818978         1               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        24         3471446         3471447         1               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        25         3630075         3630075         0               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        26         3450858         3450858         0               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        27         3846618         3846618         0               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        28         3781642         3781642         0               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        29         3412103         3412103         0               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        30         3467678         3467678         0               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        31         3444285         3444285         0               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        32         3531848         3531848         0               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        33         3600126         3600127         1               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        34         3213195         3213195         0               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        35         4185034         4185034         0               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        36         3681465         3681467         2               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        37         3339847         3339847         0               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        38         3771778         3771778         0               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        39         4199500         4199500         0               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        40         3942063         3942063         0               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        41         4029495         4029495         0               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        42         3707890         3707890         0               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        43         3467111         3467111         0               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        44         3671078         3671078         0               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        45         3700053         3700055         2               kafka-consumer-location-traffic-in-0-9b27ace9-1703-46d4-86d5-353b4c908a69 /10.202.14.11   kafka-consumer-location-traffic-in-0
location        68         4292808         4292808         0               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        69         4193630         4193630         0               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        70         3951871         3951871         0               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        71         3486437         3486440         3               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        72         3845154         3845157         3               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        73         4110218         4110218         0               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        74         3536306         3536306         0               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        75         3907729         3907731         2               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        76         3894714         3894714         0               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        77         3424257         3424257         0               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        78         3477416         3477416         0               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        79         4049843         4049844         1               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        80         3881161         3881162         1               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        81         3769417         3769417         0               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        82         4087392         4087392         0               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        83         3721245         3721245         0               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        84         3563035         3563035         0               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        85         3822875         3822875         0               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        86         4160157         4160157         0               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        87         3902806         3902807         1               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        88         4029498         4029498         0               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        89         3523406         3523406         0               kafka-consumer-location-traffic-in-1-643c56a3-1c5f-4f97-bc49-133d1deeb190 /10.202.14.11   kafka-consumer-location-traffic-in-1
location        134        4106249         4106249         0               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        135        3434190         3434190         0               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        136        3702670         3702670         0               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        137        3890294         3890295         1               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        138        3874990         3874991         1               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        139        4263445         4263445         0               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        140        4095948         4095948         0               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        141        3766815         3766816         1               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        142        3843828         3843829         1               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        143        3627205         3627205         0               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        144        3633239         3633239         0               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        145        4313211         4313211         0               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        146        3680403         3680403         0               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        147        4211879         4211879         0               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        148        4019564         4019564         0               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        149        3595004         3595004         0               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        150        4404499         4404500         1               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        151        4238058         4238058         0               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        152        4110814         4110815         1               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        153        3828031         3828033         2               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        154        4698773         4698773         0               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        155        3489979         3489979         0               kafka-consumer-location-traffic-in-2-374ce9fb-65b5-4ada-8a6b-cd5555b6216e /10.202.14.11   kafka-consumer-location-traffic-in-2
location        0          4091375         4093334         1959            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        10         3646356         3648196         1840            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        11         4058930         4061359         2429            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        12         3837605         3839332         1727            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        13         3515922         3517819         1897            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        1          4403110         4405258         2148            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        14         4332411         4334565         2154            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        15         3660145         3661418         1273            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        16         3876582         3878472         1890            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        17         3723267         3724730         1463            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        18         3583073         3585043         1970            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        19         3508683         3511126         2443            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        20         3506342         3508275         1933            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        21         3785803         3787698         1895            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        22         3863065         3864859         1794            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        2          3474667         3476749         2082            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        3          3548161         3549906         1745            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        4          3868823         3870572         1749            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        5          3955624         3957855         2231            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        6          4306242         4308539         2297            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        7          4267001         4269147         2146            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        8          4466247         4468623         2376            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        9          3551641         3553652         2011            kafka-consumer-location-traffic-in-0-280a13d7-88ce-43d7-a506-3d739c968ff4 /10.202.14.12   kafka-consumer-location-traffic-in-0
location        100        3640434         3642133         1699            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        101        3830056         3832394         2338            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        102        3534801         3536922         2121            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        103        3167506         3168837         1331            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        104        3492716         3494855         2139            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        105        3601980         3603545         1565            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        106        3800792         3803103         2311            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        107        3628906         3631336         2430            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        108        3848088         3849732         1644            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        109        4076314         4078748         2434            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        110        3899280         3901424         2144            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        111        3841329         3842580         1251            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        90         4075780         4078545         2765            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        91         3875897         3878221         2324            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        92         3738800         3740698         1898            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        93         3912474         3914320         1846            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        94         3864730         3866899         2169            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        95         3706019         3707461         1442            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        96         4043690         4045659         1969            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        97         3547743         3550044         2301            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        98         3817204         3818922         1718            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        99         3627514         3629283         1769            kafka-consumer-location-traffic-in-1-85e550cd-4950-4382-aca1-1d851e740f03 /10.202.14.12   kafka-consumer-location-traffic-in-1
location        178        4171096         4173122         2026            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        179        3462498         3464089         1591            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        180        3606194         3608068         1874            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        181        3788598         3789724         1126            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        182        3666659         3668227         1568            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        183        4440209         4442840         2631            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        184        3978404         3980702         2298            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        185        4155266         4157523         2257            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        186        3638633         3640449         1816            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        187        3961742         3964619         2877            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        188        3451711         3453613         1902            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        189        4041334         4043063         1729            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        190        4016676         4018492         1816            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        191        3982151         3983754         1603            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        192        4364896         4367444         2548            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        193        3846184         3848764         2580            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        194        3656020         3657859         1839            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        195        3740366         3742139         1773            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        196        3394225         3395977         1752            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        197        3511267         3512803         1536            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        198        3884761         3886491         1730            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
location        199        3667085         3668360         1275            kafka-consumer-location-traffic-in-2-a2642e41-21c8-4a23-8f1c-19b382db5f20 /10.202.14.12   kafka-consumer-location-traffic-in-2
TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                               HOST            CLIENT-ID

Jul 13 '22 09:07 liuzhaozhao

I think the .onFailure().invoke(t -> LIMIT_EXCEPTION.tryAcquire(() -> log.warn("Deal Exception", t))) can be an issue and you basically deadlock your system.

Your method is called on a single thread, so, if you block that thread, nothing else happens.

Jul 13 '22 09:07 cescoffier

.onFailure().invoke(t -> LIMIT_EXCEPTION.tryAcquire(() -> log.warn("Deal Exception", t))) This code does not block and is internally implemented to do no internal log printing if the limit is exceeded.

The problem does not recur after increasing @Timeout(60000), and needs to be observed for several days

@Slf4j
@ApplicationScoped
@Timeout(60000)
public class LocationTrafficConsumer {
    private static final RateLimit LIMIT_EXCEPTION = RateLimit.create(3);

    @Incoming("location-traffic-in")
    @NonBlocking
    public Uni<Void> consume(List<String> events) {
        try {
            List<Uni<Void>> uniList = new ArrayList<>();
            // some code
            return Uni.join().all(uniList).andCollectFailures().replaceWithVoid()
                    .onFailure().retry().atMost(3)
                    .onFailure().invoke(t -> LIMIT_EXCEPTION.tryAcquire(() -> log.warn("Deal Exception", t)))
                    .onFailure().recoverWithNull();
        } catch (Throwable e) {
            log.warn("Exception", e);
            return Uni.createFrom().voidItem();
        }
    }

}

Jul 15 '22 02:07 liuzhaozhao

@cescoffier The problem is still there, in order to make sure it is not a business code problem, I added the @incoming ("xxx-in") consume method Incoming quantity and outgoing (Uni) quantity. When some partition consume stops, the Incoming quantity of this part of partition is the same as the outgoing quantity, so the consume stop location is not blocked in the business code.

Before that, I tried to upgrade Quarkus version 2.11.1 (including the version upgrade of smallrye-reactive-messaging), but the problem was more serious, after a moment of startup, all partition consume stopped. I went back to Quarkus version 2.7.6 and tried to change the configuration several times. The current configuration is as follows:

mp:
  messaging:
    incoming:
      xxx-in:
        connector: smallrye-kafka
        health-enabled: true
        health-readiness-enabled: true
        tracing-enabled: false
        cloud-events: false
        topic: xxx-topic
        graceful-shutdown: true
        group:
          id: xxx-consumer
        key:
          deserializer: org.apache.kafka.common.serialization.StringDeserializer
        value:
          deserializer: org.apache.kafka.common.serialization.StringDeserializer
        auto:
          offset:
            reset: latest
        batch: true
        enable:
          auto:
            commit: true
        commit-strategy: ignore
        broadcast: false
        partitions: 6
        requests: 5
        max:
          poll:
            records: 500
        failure-strategy: ignore
        fail-on-deserialization-failure: false
        consumer-rebalance-listener:
          name: iov-event-topic.rebalancer

@Slf4j
@ApplicationScoped
public class XxxConsumer {
    @Incoming("xxx-in")
    @Acknowledgment(Acknowledgment.Strategy.POST_PROCESSING)
    @NonBlocking
    Uni<Void> consume(ConsumerRecords<String, String> events) {
        long startTime = System.currentTimeMillis();
        // statistics incoming here
        List<Uni<Void>> groupResults = new ArrayList<>();
	// some business code here
        return Uni.join().all(groupResults).andFailFast()
                .onFailure().retry().withBackOff(Duration.ofMillis(500), Duration.ofMillis(1000)).atMost(3)
                .ifNoItem().after(Duration.ofSeconds(40)).fail()
                .onFailure().invoke(t -> log.warn("Exception({}ms)：{}", System.currentTimeMillis() - startTime, topicPartitions, t))
                .onFailure().recoverWithNull()
                .invoke(v -> {
                    // statistics outgoing here
                });
    }
}

Do you have any suggestions for troubleshooting? There are many service consumption Topics, and this problem has appeared in many topics. The topic with the largest number of publish is about 8000 QPS.

Aug 06 '22 13:08 liuzhaozhao

I like the coding method of smallrye-reactive-messaging very much. Compared with using kafka api code, it is very concise, and it can achieve coding experience independent of mq server. If the team changes MQ coding method, it will generate more extra coding workload, so I very much hope that this problem can be solved.

Aug 06 '22 13:08 liuzhaozhao

@cescoffier @ozangunalp Analysis of jmap -dump and smallrye-reactive-messaging-kafka:3.13.0 shows that the resume condition is never met.

Aug 10 '22 03:08 liuzhaozhao

Aug 12 '22 08:08 liuzhaozhao

I think this is fixed by #1895

Oct 03 '22 16:10 ozangunalp

@ozangunalp hi. I've upgraded quarkus to 2.13.3.Final which has smallrye-reactive-messaging-kafka-3.21.0.jar But I still am having this issue.

Oct 27 '22 14:10 privetpoka

hi @privetpoka, we've just resolved another fix for this type of problem in #1956. It'll be included in 3.22.0. You can check it yourself, or If you'd have a reproducer I'd be glad to investigate it.

Oct 28 '22 14:10 ozangunalp

hi @ozangunalp , after I upgrade quarkus 2.16.7.Final(smallrye-reactive-messaging-kafka 3.22.1), the problem still exists.

I try to increase the log troubleshooting problem in KafkaRecordStreamSubscription,it was found that the client.pause() was executed and requested.get() is 0 for a long time, as a result, it cannot be executed to downstream.onItem(item) for a long time,This is my log code and log print:

// In class KafkaRecordStreamSubscription
private void pauseResume() {
        int size = queue.size();
        boolean print;
        if (size >= maxQueueSize && state.compareAndSet(STATE_POLLING, STATE_PAUSED)) {
            log.pausingChannel(channel, clientId, size, maxQueueSize);
            client.pause()
                    .subscribe().with(this::emptyConsumer, this::report);
            print = metric.pause();
        } else if (size <= halfMaxQueueSize && state.compareAndSet(STATE_PAUSED, STATE_POLLING)) {
            log.resumingChannel(channel, clientId, size, halfMaxQueueSize);
            client.resume()
                    .subscribe().with(this::emptyConsumer, this::report);
            print = metric.resume();
        } else {
            print = metric.other();
        }
        if (print) {
            logger.infof("Kafka metric info: %s(%s), size=%s,maxQueueSize=%s,halfMaxQueueSize=%s,wip=%s,state=%s,requested=%s",
                    channel, clientId,
                    size,
                    maxQueueSize, halfMaxQueueSize, wip.get(), state.get(), requested.get());
        }
    }


public class KafkaMetric {
    private final Logger log = Logger.getLogger(this.getClass());
    private final String channel;
    private final String clientId;

    private final AtomicLong pauseTimes = new AtomicLong(0);
    private final AtomicLong resumeTimes = new AtomicLong(0);
    private final AtomicLong otherTimes = new AtomicLong(0);
    private long printTime = 0;

    public KafkaMetric(String channel, String clientId) {
        this.channel = channel;
        this.clientId = clientId;
    }

    public boolean pause() {
        pauseTimes.incrementAndGet();
        return checkPrint();
    }

    public boolean resume() {
        resumeTimes.incrementAndGet();
        return checkPrint();
    }

    public boolean other() {
        otherTimes.incrementAndGet();
        return checkPrint();
    }

    private boolean checkPrint() {
        long now = System.currentTimeMillis();
        if ((now - printTime > 5000 && otherTimes.get() % 100 == 0) || now - printTime > 60000) {
            log.infof("Kafka metric:%s (%s), pause=%s,resume=%s,other=%s", channel, clientId, pauseTimes.get(),
                    resumeTimes.get(), otherTimes.get());
            printTime = now;
            return true;
        }
        return false;
    }

}

The code is to regularly print the number of pause/resume and other variable values.

The log:

2023-06-14 17:51:23.558[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:51:28.741[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=710,resume=709,other=514500
2023-06-14 17:51:28.741[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:51:33.810[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=710,resume=709,other=516300
2023-06-14 17:51:33.810[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:51:38.813[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=710,resume=709,other=518400
2023-06-14 17:51:38.813[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:51:43.900[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=710,resume=709,other=520200
2023-06-14 17:51:43.901[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:51:49.081[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=710,resume=709,other=522300
2023-06-14 17:51:49.082[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:51:54.115[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=710,resume=709,other=524000
2023-06-14 17:51:54.116[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:51:59.235[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=710,resume=709,other=526100
2023-06-14 17:51:59.236[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:52:04.430[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=710,resume=709,other=528000
2023-06-14 17:52:04.430[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:52:09.627[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=710,resume=709,other=530200
2023-06-14 17:52:09.628[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:52:14.798[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=710,resume=709,other=532000
2023-06-14 17:52:14.798[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:52:20.014[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=710,resume=709,other=534200
2023-06-14 17:52:20.014[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:52:25.124[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=710,resume=709,other=536000
2023-06-14 17:52:25.124[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:52:30.221[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=710,resume=709,other=538100
2023-06-14 17:52:30.222[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:52:35.437[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=710,resume=709,other=539800
2023-06-14 17:52:35.438[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:52:40.571[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=710,resume=709,other=541900
2023-06-14 17:52:40.571[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:52:45.811[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=710,resume=709,other=543700
2023-06-14 17:52:45.811[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:52:50.884[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=711,resume=710,other=545300
2023-06-14 17:52:50.884[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:52:56.068[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=711,resume=710,other=547100
2023-06-14 17:52:56.069[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:53:01.081[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=711,resume=710,other=548800
2023-06-14 17:53:01.081[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:53:06.108[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=711,resume=710,other=550800
2023-06-14 17:53:06.108[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:53:11.133[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=711,resume=710,other=552600
2023-06-14 17:53:11.134[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:53:16.149[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=711,resume=710,other=554600
2023-06-14 17:53:16.150[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:53:21.231[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=714,resume=713,other=556400
2023-06-14 17:53:21.232[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=2,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:53:26.469[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=729,resume=728,other=558400
2023-06-14 17:53:26.470[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:53:31.666[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=744,resume=743,other=560200
2023-06-14 17:53:31.666[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=3,state=2,requested=0
2023-06-14 17:53:36.726[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=757,resume=756,other=561900
2023-06-14 17:53:36.727[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:53:41.917[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=771,resume=770,other=563700
2023-06-14 17:53:41.919[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=2,maxQueueSize=10,halfMaxQueueSize=0,wip=1,state=2,requested=0
2023-06-14 17:53:47.079[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=786,resume=786,other=565700
2023-06-14 17:53:47.079[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=3,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=1,requested=0
2023-06-14 17:53:52.112[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=800,resume=799,other=567400
2023-06-14 17:53:52.113[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=2,maxQueueSize=10,halfMaxQueueSize=0,wip=2,state=2,requested=0
2023-06-14 17:53:57.165[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=812,resume=811,other=568900
2023-06-14 17:53:57.165[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=3,state=2,requested=0
2023-06-14 17:54:02.349[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=826,resume=825,other=571000
2023-06-14 17:54:02.350[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=2,maxQueueSize=10,halfMaxQueueSize=0,wip=1,state=2,requested=1
2023-06-14 17:54:07.408[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=838,resume=837,other=572700
2023-06-14 17:54:07.408[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=2,maxQueueSize=10,halfMaxQueueSize=0,wip=1,state=2,requested=0
2023-06-14 17:54:12.613[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=850,resume=849,other=574800
2023-06-14 17:54:12.614[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:54:17.808[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=861,resume=860,other=576600
2023-06-14 17:54:17.809[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:54:23.013[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=868,resume=867,other=578800
2023-06-14 17:54:23.014[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:54:28.061[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=868,resume=867,other=580600
2023-06-14 17:54:28.061[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:54:33.122[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=868,resume=867,other=582700
2023-06-14 17:54:33.129[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:54:38.179[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=868,resume=868,other=584300
2023-06-14 17:54:38.180[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=1,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=1,requested=1
2023-06-14 17:54:43.982[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=869,resume=868,other=586400
2023-06-14 17:54:43.982[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:54:49.160[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=869,resume=868,other=588500
2023-06-14 17:54:49.160[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:54:54.168[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=869,resume=868,other=590200
2023-06-14 17:54:54.168[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:54:59.370[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=869,resume=868,other=592300
2023-06-14 17:54:59.370[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:55:04.419[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=869,resume=868,other=594000
2023-06-14 17:55:04.420[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:55:09.424[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=869,resume=868,other=596000
2023-06-14 17:55:09.424[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:55:14.664[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=874,resume=873,other=597800
2023-06-14 17:55:14.664[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:55:19.743[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=887,resume=887,other=599800
2023-06-14 17:55:19.743[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=7,maxQueueSize=10,halfMaxQueueSize=0,wip=1,state=1,requested=0
2023-06-14 17:55:24.923[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=898,resume=897,other=601500
2023-06-14 17:55:24.924[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=2,maxQueueSize=10,halfMaxQueueSize=0,wip=3,state=2,requested=0
2023-06-14 17:55:30.167[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=903,resume=902,other=603600
2023-06-14 17:55:30.168[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:55:35.408[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=903,resume=902,other=605400
2023-06-14 17:55:35.409[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:55:40.494[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=904,resume=904,other=607400
2023-06-14 17:55:40.495[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=3,maxQueueSize=10,halfMaxQueueSize=0,wip=3,state=1,requested=1
2023-06-14 17:55:45.734[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=916,resume=915,other=609100
2023-06-14 17:55:45.735[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:55:50.814[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=923,resume=922,other=611200
2023-06-14 17:55:50.814[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:55:56.009[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=926,resume=925,other=613000
2023-06-14 17:55:56.010[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:56:01.251[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=932,resume=931,other=615200
2023-06-14 17:56:01.251[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=2,maxQueueSize=10,halfMaxQueueSize=0,wip=1,state=2,requested=0
2023-06-14 17:56:06.339[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=939,resume=938,other=616900
2023-06-14 17:56:06.339[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=3,state=2,requested=0
2023-06-14 17:56:11.398[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=987,resume=986,other=619600
2023-06-14 17:56:11.398[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:56:16.540[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1058,resume=1058,other=622400
2023-06-14 17:56:16.540[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=4,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=1,requested=0
2023-06-14 17:56:21.702[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1138,resume=1138,other=625600
2023-06-14 17:56:21.702[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=1,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=1,requested=1
2023-06-14 17:56:26.767[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1196,resume=1196,other=630000
2023-06-14 17:56:26.767[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=1,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=1,requested=1
2023-06-14 17:56:31.844[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1245,resume=1245,other=634500
2023-06-14 17:56:31.844[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=1,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=1,requested=0
2023-06-14 17:56:37.486[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1298,resume=1297,other=638500
2023-06-14 17:56:37.486[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=2,maxQueueSize=10,halfMaxQueueSize=0,wip=4,state=2,requested=0
2023-06-14 17:56:42.535[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1352,resume=1352,other=643200
2023-06-14 17:56:42.536[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=1,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=1,requested=1
2023-06-14 17:56:47.616[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1420,resume=1420,other=647400
2023-06-14 17:56:47.616[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=1,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=1,requested=1
2023-06-14 17:56:52.753[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1466,resume=1466,other=652100
2023-06-14 17:56:52.754[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=1,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=1,requested=1
2023-06-14 17:56:57.808[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1526,resume=1526,other=656700
2023-06-14 17:56:57.809[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=7,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=1,requested=0
2023-06-14 17:57:02.944[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=660200
2023-06-14 17:57:02.944[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:57:08.153[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=662400
2023-06-14 17:57:08.153[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:57:13.394[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=664700
2023-06-14 17:57:13.394[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:57:18.579[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=666900
2023-06-14 17:57:18.579[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:57:23.792[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=669200
2023-06-14 17:57:23.793[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:57:28.934[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=671400
2023-06-14 17:57:28.935[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:57:33.974[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=673600
2023-06-14 17:57:33.975[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:57:39.187[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=675400
2023-06-14 17:57:39.188[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:57:44.236[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=677600
2023-06-14 17:57:44.236[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:57:49.312[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=679800
2023-06-14 17:57:49.312[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:57:54.435[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=682000
2023-06-14 17:57:54.438[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:57:59.444[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=684200
2023-06-14 17:57:59.444[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:58:04.548[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=686400
2023-06-14 17:58:04.548[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:58:09.578[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=688600
2023-06-14 17:58:09.578[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:58:14.725[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=690800
2023-06-14 17:58:14.725[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:58:19.787[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=693000
2023-06-14 17:58:19.788[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:58:24.991[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=695200
2023-06-14 17:58:24.991[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:58:30.048[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=697400
2023-06-14 17:58:30.048[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:58:35.234[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=699600
2023-06-14 17:58:35.234[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:58:40.394[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=701900
2023-06-14 17:58:40.394[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:58:45.484[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=703800
2023-06-14 17:58:45.484[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:58:50.647[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=706100
2023-06-14 17:58:50.647[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:58:55.813[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=708300
2023-06-14 17:58:55.813[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:59:00.955[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=710600
2023-06-14 17:59:00.956[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:59:05.991[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=712700
2023-06-14 17:59:05.992[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:59:11.132[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=714900
2023-06-14 17:59:11.133[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:59:16.367[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=717000
2023-06-14 17:59:16.368[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:59:21.483[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=719200
2023-06-14 17:59:21.483[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:59:26.501[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=720800
2023-06-14 17:59:26.501[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:59:31.637[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=723100
2023-06-14 17:59:31.638[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:59:36.743[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=725300
2023-06-14 17:59:36.744[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:59:41.887[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=727600
2023-06-14 17:59:41.888[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:59:46.942[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=729700
2023-06-14 17:59:46.942[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:59:52.161[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=732000
2023-06-14 17:59:52.162[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 17:59:57.281[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=734100
2023-06-14 17:59:57.281[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:00:02.462[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=736400
2023-06-14 18:00:02.463[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:00:07.516[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=738500
2023-06-14 18:00:07.516[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:00:12.714[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=740800
2023-06-14 18:00:12.715[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:00:17.915[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=743000
2023-06-14 18:00:17.915[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:00:22.928[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=745200
2023-06-14 18:00:22.928[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:00:28.128[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=747100
2023-06-14 18:00:28.128[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:00:33.346[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=749400
2023-06-14 18:00:33.346[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:00:38.361[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=751500
2023-06-14 18:00:38.361[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:00:43.447[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=753700
2023-06-14 18:00:43.448[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:00:48.580[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=755900
2023-06-14 18:00:48.580[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:00:53.587[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=758100
2023-06-14 18:00:53.588[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:00:58.736[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=760300
2023-06-14 18:00:58.736[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:01:03.760[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=762500
2023-06-14 18:01:03.760[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:01:08.791[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=764600
2023-06-14 18:01:08.791[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:01:13.904[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=766800
2023-06-14 18:01:13.904[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:01:19.010[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=768600
2023-06-14 18:01:19.010[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:01:24.078[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=770800
2023-06-14 18:01:24.078[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:01:29.267[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=773000
2023-06-14 18:01:29.267[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:01:34.426[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=775200
2023-06-14 18:01:34.426[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:01:39.494[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=777400
2023-06-14 18:01:39.494[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:01:44.699[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=779600
2023-06-14 18:01:44.699[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:01:49.841[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=781800
2023-06-14 18:01:49.841[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:01:55.078[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=784000
2023-06-14 18:01:55.078[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:02:00.103[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=786200
2023-06-14 18:02:00.103[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:02:05.335[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=788400
2023-06-14 18:02:05.335[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:02:10.396[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=790600
2023-06-14 18:02:10.396[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:02:15.581[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=792400
2023-06-14 18:02:15.581[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:02:20.632[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=794600
2023-06-14 18:02:20.633[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:02:25.690[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=796700
2023-06-14 18:02:25.691[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:02:30.767[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=798900
2023-06-14 18:02:30.767[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:02:35.776[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=800900
2023-06-14 18:02:35.776[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:02:40.829[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=803100
2023-06-14 18:02:40.829[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:02:45.892[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=805200
2023-06-14 18:02:45.893[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:02:51.055[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=807100
2023-06-14 18:02:51.055[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:02:56.206[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=809300
2023-06-14 18:02:56.207[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:03:01.390[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=811600
2023-06-14 18:03:01.391[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:03:06.549[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=813700
2023-06-14 18:03:06.550[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:03:11.756[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=816000
2023-06-14 18:03:11.757[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:03:16.816[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=818100
2023-06-14 18:03:16.816[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:03:21.880[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=820300
2023-06-14 18:03:21.881[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:03:26.987[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=822400
2023-06-14 18:03:26.988[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:03:32.021[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=824300
2023-06-14 18:03:32.022[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:03:37.226[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=826500
2023-06-14 18:03:37.227[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:03:42.430[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=828800
2023-06-14 18:03:42.430[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:03:47.523[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=830900
2023-06-14 18:03:47.523[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:03:52.564[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=833100
2023-06-14 18:03:52.565[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:03:57.743[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=835300
2023-06-14 18:03:57.743[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:04:02.963[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=837600
2023-06-14 18:04:02.964[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:04:08.058[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=839700
2023-06-14 18:04:08.059[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:04:13.283[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=842000
2023-06-14 18:04:13.284[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:04:18.448[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=843900
2023-06-14 18:04:18.449[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:04:23.569[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=846100
2023-06-14 18:04:23.569[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:04:28.699[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=848300
2023-06-14 18:04:28.700[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:04:33.761[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=850500
2023-06-14 18:04:33.761[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:04:38.881[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=852700
2023-06-14 18:04:38.881[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=1,state=2,requested=0
2023-06-14 18:04:44.090[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=854900
2023-06-14 18:04:44.091[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:04:49.174[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=857100
2023-06-14 18:04:49.175[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:04:54.367[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=859100
2023-06-14 18:04:54.374[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:04:59.453[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=861300
2023-06-14 18:04:59.453[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:05:04.684[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=863500
2023-06-14 18:05:04.684[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:05:09.720[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=865700
2023-06-14 18:05:09.720[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:05:14.747[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=867800
2023-06-14 18:05:14.747[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:05:19.756[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=870000
2023-06-14 18:05:19.756[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:05:24.935[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=872200
2023-06-14 18:05:24.936[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:05:30.149[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=874200
2023-06-14 18:05:30.150[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:05:35.325[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=876400
2023-06-14 18:05:35.326[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:05:40.519[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=878700
2023-06-14 18:05:40.519[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:05:45.587[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=880800
2023-06-14 18:05:45.588[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:05:50.766[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=883100
2023-06-14 18:05:50.767[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:05:55.835[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=885300
2023-06-14 18:05:55.836[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:06:01.025[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=887600
2023-06-14 18:06:01.026[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:06:06.158[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=889800
2023-06-14 18:06:06.158[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:06:11.341[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=892100
2023-06-14 18:06:11.341[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:06:16.343[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=893900
2023-06-14 18:06:16.343[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:06:21.544[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=896200
2023-06-14 18:06:21.545[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:06:26.601[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=898300
2023-06-14 18:06:26.602[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:06:31.818[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=900600
2023-06-14 18:06:31.818[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:06:36.827[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=902700
2023-06-14 18:06:36.827[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:06:42.029[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=905000
2023-06-14 18:06:42.030[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:06:47.096[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=906900
2023-06-14 18:06:47.096[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:06:52.250[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=909200
2023-06-14 18:06:52.251[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:06:57.431[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=911400
2023-06-14 18:06:57.431[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:07:02.560[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=913600
2023-06-14 18:07:02.561[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:07:07.737[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=915800
2023-06-14 18:07:07.738[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:07:12.894[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=918100
2023-06-14 18:07:12.894[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:07:18.075[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=920300
2023-06-14 18:07:18.075[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:07:23.302[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=922600
2023-06-14 18:07:23.302[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:07:28.435[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=924500
2023-06-14 18:07:28.436[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:07:33.572[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=926700
2023-06-14 18:07:33.573[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:07:38.750[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=928900
2023-06-14 18:07:38.751[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:07:43.882[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=931100
2023-06-14 18:07:43.882[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:07:48.891[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=933300
2023-06-14 18:07:48.892[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:07:53.953[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=935500
2023-06-14 18:07:53.953[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:07:58.993[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=937700
2023-06-14 18:07:58.993[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:08:04.238[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=939900
2023-06-14 18:08:04.238[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:08:09.363[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=941900
2023-06-14 18:08:09.363[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:08:14.548[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=944100
2023-06-14 18:08:14.548[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:08:19.770[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=946400
2023-06-14 18:08:19.770[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:08:24.976[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=948600
2023-06-14 18:08:24.976[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:08:29.983[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=950800
2023-06-14 18:08:29.983[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:08:35.202[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=953000
2023-06-14 18:08:35.203[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:08:40.387[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=955300
2023-06-14 18:08:40.388[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:08:45.457[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=957100
2023-06-14 18:08:45.457[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:08:50.602[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=959400
2023-06-14 18:08:50.602[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:08:55.695[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=961600
2023-06-14 18:08:55.695[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:09:00.819[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=963900
2023-06-14 18:09:00.819[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:09:05.926[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=966100
2023-06-14 18:09:05.926[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:09:11.101[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=968400
2023-06-14 18:09:11.101[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:09:16.200[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=970600
2023-06-14 18:09:16.200[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:09:21.655[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=972800
2023-06-14 18:09:21.655[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:09:26.825[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=975000
2023-06-14 18:09:26.826[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:09:32.026[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=977300
2023-06-14 18:09:32.027[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:09:37.160[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=979500
2023-06-14 18:09:37.161[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:09:42.339[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=981800
2023-06-14 18:09:42.340[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:09:47.430[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=984000
2023-06-14 18:09:47.430[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:09:52.595[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=986300
2023-06-14 18:09:52.595[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:09:57.600[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=988200
2023-06-14 18:09:57.600[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:10:02.775[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=990500
2023-06-14 18:10:02.776[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:10:07.826[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=992700
2023-06-14 18:10:07.827[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:10:12.960[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=995000
2023-06-14 18:10:12.960[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:10:18.014[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=997200
2023-06-14 18:10:18.014[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:10:23.251[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=999500
2023-06-14 18:10:23.252[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:10:28.322[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1001700
2023-06-14 18:10:28.323[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:10:33.599[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1003800
2023-06-14 18:10:33.599[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:10:38.749[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1006100
2023-06-14 18:10:38.749[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:10:43.949[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1008400
2023-06-14 18:10:43.949[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:10:49.032[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1010700
2023-06-14 18:10:49.032[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:10:54.259[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1013000
2023-06-14 18:10:54.259[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:10:59.390[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1015300
2023-06-14 18:10:59.391[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:11:04.559[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1017500
2023-06-14 18:11:04.559[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:11:09.619[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1019500
2023-06-14 18:11:09.619[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:11:14.644[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1021700
2023-06-14 18:11:14.644[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:11:19.778[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1024000
2023-06-14 18:11:19.778[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:11:24.870[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1026200
2023-06-14 18:11:24.871[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:11:30.087[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1028500
2023-06-14 18:11:30.087[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:11:35.251[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1030700
2023-06-14 18:11:35.252[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:11:40.345[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1033000
2023-06-14 18:11:40.345[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:11:45.573[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1035000
2023-06-14 18:11:45.574[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:11:50.770[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1037300
2023-06-14 18:11:50.776[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:11:55.965[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1039500
2023-06-14 18:11:55.965[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:12:00.984[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1041700
2023-06-14 18:12:00.985[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:12:06.179[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1043900
2023-06-14 18:12:06.179[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:12:11.386[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1046200
2023-06-14 18:12:11.386[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:12:16.738[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1048300
2023-06-14 18:12:16.738[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:12:21.887[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1050600
2023-06-14 18:12:21.887[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:12:26.963[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1052800
2023-06-14 18:12:26.964[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:12:32.189[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1055100
2023-06-14 18:12:32.189[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:12:37.269[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1057300
2023-06-14 18:12:37.280[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:12:42.473[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1059600
2023-06-14 18:12:42.474[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:12:47.478[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1061700
2023-06-14 18:12:47.478[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:12:52.680[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1063800
2023-06-14 18:12:52.680[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaRecordStreamSubscription(172)]: Kafka metric info: 5-iov-event-in(kafka-consumer-5-iov-event-in-0), size=10,maxQueueSize=10,halfMaxQueueSize=0,wip=0,state=2,requested=0
2023-06-14 18:12:57.810[INFO]-[smallrye-kafka-consumer-thread-0]-[i.s.r.m.k.i.KafkaMetric(40)]: Kafka metric:5-iov-event-in (kafka-consumer-5-iov-event-in-0), pause=1561,resume=1560,other=1066000

From the log, it can be seen that the consumption has been suspended for a long time many times, and in the end it has been suspended all the time, which can rule out the problem of slow consumption.

If consume configuration partitions greater than 1 , more likely to cause problems.

Jun 14 '23 10:06 liuzhaozhao

After adjusting according to the following configuration, there has been no problem of stopping consumption recently:

mp:
  messaging:
    incoming:
      xxx-in:
        ......
        commit-strategy: throttled
        throttled:
          unprocessed-record-max-age:
            ms: ${consumer.offset-commit-interval-ms:500000}
        max:
          poll:
            interval:
              ms: ${consumer.max-poll-interval-ms:300000}
        ......

If not configured, the default is:

# Stop consumption when the site submission interval exceeds (check the log to see that rebalancing is not triggered)
unprocessed-record-max-age.ms = 60000
# Trigger rebalancing after the interval between two pulls exceeds this configuration (consumption will not be stopped)
max.poll.interval.ms = 300000

The default value unprocessed-record-max-age.ms is smaller than max.poll.interval.ms, causing consumption to stop before triggering rebalancing, hi @ozangunalp @cescoffier , do you need to adjust the default value of unprocessed-record-max-age.ms to avoid this problem.

Jul 03 '23 09:07 liuzhaozhao

Is it possible that processing of some records takes more than 60s? Therefore setting 500s to unprocessed max age takes you to the safe side of that check?

Jul 03 '23 11:07 ozangunalp

Is it possible that processing of some records takes more than 60s? Therefore setting 500s to unprocessed max age takes you to the safe side of that check?

Yes, the database will be operated when consuming data. Because of the network environment or database problems, a certain batch of records may be processed for a long time (through @Timed(..., longTask = true), it can be found that it does exist)；

unprocessed-record- max-age.ms=500000 just configures a value greater than max.poll.interval.ms, the specific value is not important, it is used to ensure rebalancing instead of stopping consumption during long-term consumption, so it is hoped that the default configuration of smallrye-reactive-messaging is that unprocessed-record-max-age.ms is greater than max.poll.interval.ms to avoid the occasional problem of stopping consumption when users use the default configuration;

Jul 04 '23 09:07 liuzhaozhao

smallrye-reactive-messaging smallrye-reactive-messaging copied to clipboard

Kafka consume stopped without any exception log

Problem description

Kafka Grafana Metric

Config

Code

Environment and dependency packages

smallrye-reactive-messaging
smallrye-reactive-messaging copied to clipboard