"Local: Invalid argument or configuration" when building as 64 bit

[JoaoFranciscoCarvalhoNeto]

Description

Following code runs when building as Any CPU, but when building as 64 bit, it returns a "Local: Invalid argument or configuration" error. Running with Debug all enabled, I can see that it reverts to api v 0.9.0.0

%7|1558374888.039|APIVERSION|rdkafka#producer-1| [thrd:localhost:19092/bootstrap]: localhost:19092/bootstrap: Using (configuration fallback) 0.9.0 protocol features

I could not find this error anywhere on the internet. I am working with some libraries only built for 64 bit, so building as Any CPU is not a feasible workaround.

How to reproduce

    static void Main(string[] args)
    {
        // Generate a byte array via google protobuf
       Byte[] b = new Byte[100];
        rnd.NextBytes(b)

        var message = new Message<Null, byte[]> { Value = packet.ToByteArray() };
        Action<DeliveryReport<Null, byte[]>> handler = r =>
        Console.WriteLine(!r.Error.IsError
        ? $"Delivered message to {r.TopicPartitionOffset}"
        : $"Delivery Error: {r.Error.Reason}");

        ProducerConfig conf = new ProducerConfig { BootstrapServers = "localhost:19092,localhost:29092,localhost:39092" };


        using (var p = new ProducerBuilder<Null, byte[]>(conf).Build())
        {
            for (int i = 0; i < 100; i++)
            {
                try
                {
                    p.Produce("Test Topic", message, handler);
                    Console.WriteLine("Sent packet");
                    Thread.Sleep(5000);
                }
                catch (KafkaException e)
                {
                    Console.WriteLine(e.Message);
                    throw;
                }
            }
        }
    }

Checklist

Please provide the following information:

• [ ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• [ ] Confluent.Kafka nuget version. v 1.0.0
• [ ] Apache Kafka version. v2.2.0
• [ ] Client configuration. BootstrapServers only
• [ ] Operating system. Windows 10 64 bit
• [ ] Provide logs (with "debug" : "..." as necessary in configuration).
• [ ] Provide broker log excerpts.
• [ ] Critical issue.

[JoaoFranciscoCarvalhoNeto]

Updating the issue:

Solved when downgrading to 0.9.5

[mhowlett]

What does Library.VersionString report in the case that isn't working? Do you have librdkafka installed on your machine elsewhere? It sounds like the library loader may be finding and loading an old version of the library for some reason.

[andreas-soroko]

Quite similar problem but happens in x86 and x64 confluent-kafka-dotnet: 1.2.0 librdkafka (Library.VersionString): 1.2.0 (only happens when EnableAutoCommit = false )

Local: Invalid argument or configuration
   at Confluent.Kafka.Impl.SafeKafkaHandle.StoreOffsets(IEnumerable`1 offsets)
   at Confluent.Kafka.Consumer`2.StoreOffset(TopicPartitionOffset offset)
   at Confluent.Kafka.Consumer`2.StoreOffset(ConsumeResult`2 result)
   ....

[MZ-Q]

Same issue during ProduceAsync call on Docker. Confluent.Kafka nuget version. v 1.3.0 Docker runtime: mcr.microsoft.com/dotnet/core/runtime:3.1-bionic

On the other hand, out of Docker container - no issues found for Ubuntu 18.04.

librdkafka-dev versions are same for both Docker start and for Ubuntu 18.04: Installed: 0.11.3-1build1 Candidate: 0.11.3-1build1

Any advises are much appreciated.

[mhowlett]

don't explicitly install librdkafka - librdkafka 0.11.3 is not compatible with Confluent.Kafka 1.3.0.

[MZ-Q]

don't explicitly install librdkafka - librdkafka 0.11.3 is not compatible with Confluent.Kafka 1.3.0.

I'm using: RUN apt-get update -y && apt-get install -y librdkafka-dev In Docker file. Because It looks like librdkafka is absent in runtime:3.1-bionic. Should I use something else?

[mhowlett]

the librdkafka.redist nuget package is a dependency of Confluent.Kafka, it should automatically use the librdkafka version in that.

[MZ-Q]

the librdkafka.redist nuget package is a dependency of Confluent.Kafka, it should automatically use the librdkafka version in that.

Awesome. Previously I had System.DllNotFoundException: Failed to load the librdkafka native library. exception without explicitly install librdkafka on my Docker container.

Looks like update to Confluent.Kafka 1.3 solved it. Thank you very much for help, friend.

[rcollina]

@mhowlett I have the same issue when invoking Consumer.StoreOffsets.

Auto commit is set to true. Auto commit offsets is set to false.

Is there any kind of thread affinity that is tripping the offset store call?

Thank you for any help.

[felipeogarcia]

@mhowlett I have the same issue when invoking Consumer.StoreOffsets.

Auto commit is set to true. Auto commit offsets is set to false.

Is there any kind of thread affinity that is tripping the offset store call?

Thank you for any help.

I was having the same exact issue. In my case I was extending the Confluent.Kafka.ConsumerConfig class overriding with some of my default values. They seem to have conflicted with the base class so I just created my own class and mapped properties back to the ConsumerConfig class. It fixed for me.

[e-jlion]

Quite similar problem but happens in x86 and x64 confluent-kafka-dotnet: 1.2.0 librdkafka (Library.VersionString): 1.2.0 (only happens when EnableAutoCommit = false )

Local: Invalid argument or configuration
   at Confluent.Kafka.Impl.SafeKafkaHandle.StoreOffsets(IEnumerable`1 offsets)
   at Confluent.Kafka.Consumer`2.StoreOffset(TopicPartitionOffset offset)
   at Confluent.Kafka.Consumer`2.StoreOffset(ConsumeResult`2 result)
   ....

I also encountered the problem of Invalid argument or configuration, how did you solve it later? Confluent.Kafka 1.7.0

[benjambe]

I had this issue and it was fixed by adding"enableAutoOffsetStore": false, in config

[vijaybw]

Do we have any workaround for this?

[mhowlett]

@vijaybw what version of .net and confluent-kafka-dotnet are you using?

[vijaybw]

@mhowlett
librdkafka: stable 1.8.2 .net 6 Confluent.kafka 1.8.2 System: Macbook Pro M1

[mhowlett]

M1 librdkafka builds are not in librdkafka.redist yet (I believe we plan to add this in v1.9) and apparently the x86 osx build isn't loading on that platform. You could use Library.Load to load a custom build of librdkafka. sorry this isn't ideal!

https://github.com/edenhill/librdkafka#build-from-source

[vijaybw]

Thank you for the reply. I followed steps from here https://github.com/confluentinc/confluent-kafka-dotnet/issues/1755#issuecomment-1039598261 I am able to run the application now which is better than "Failed to load the librdkafka". But, it fails when it is trying to produce/consume the message.

[mhowlett]

https://github.com/confluentinc/confluent-kafka-dotnet/issues/1769

[vijaybw]

Thank you tons. Hope it works in target release.

[wafeelka]

I added paramenter Acks = Acks.Leader into my ProducerConfig. When i use local build (windows x64) it works fine, message sended. But if i use my service in docker in CentOs i've got this error in method ProduceAsync(). Confluent.Kafka version 18.2 and 17.0

[wafeelka]

Я добавил параментер Acks = Acks.Leader в свой ProducerConfig. Когда я использую локальную сборку (windows x64), она работает нормально, сообщение отправлено. Но если я использую свой сервис в docker в CentOs, я получаю эту ошибку в методе ProduceAsync(). Confluent.Kafka версии 18.2 и 17.0

UPD this problem is not due to parameter Acks. without this parameter still errror

UPD My mistake, in docker topic was null. Don't repeat my mistakes

[sathoril]

Did anyone got a fix/workaroung for this? The Local : Invalid argument or configuration still happens to me.

I'm using a MacBook Air M1 - Monterey 12.3.1.

EDIT:

It worked for me using the pre-release version shown on #1769. Does anyone know when a stable version will be available with this fix?

[bmitioglov]

I had this problem when my topicName was null