Global options no longer work via rpk connect

[trahim]

We have found that global options that were in the Benthos binary no longer work when calling a command.

For example this command we used to run.

docker run ghcr.io/benthosdev/benthos -c /benthos.yaml --resources '/resources/*.yaml' streams '/streams/*.yaml'

This starts fine in streams mode. When running now with the new new redpanda image

docker run docker.redpanda.com/redpandadata/redpanda connect -c /benthos.yaml --resources '/resources/*.yaml' streams '/streams/*.yaml'

We get

Incorrect Usage: flag provided but not defined: -c

NAME:
   redpanda-connect streams - Run Redpanda Connect in streams mode

USAGE:
   redpanda-connect streams command [command options]

DESCRIPTION:
   Run Redpanda Connect in streams mode, where multiple pipelines can be executed in a
   single process and can be created, updated and removed via REST HTTP
   endpoints.

     redpanda-connect streams
     redpanda-connect -c ./root_config.yaml streams
     redpanda-connect streams ./path/to/stream/configs ./and/some/more
     redpanda-connect -c ./root_config.yaml streams ./streams/*.yaml

   In streams mode the stream fields of a root target config (input, buffer,
   pipeline, output) will be ignored. Other fields will be shared across all
   loaded streams (resources, metrics, etc).

   For more information check out the docs at:
   https://docs.redpanda.com/redpanda-connect/guides/streams_mode/about

COMMANDS:
   help, h  Shows a list of commands or help for one command

OPTIONS:
   --no-api                   Disable the HTTP API for streams mode (default: false)
   --prefix-stream-endpoints  Whether HTTP endpoints registered by stream configs should be prefixed with the stream ID (default: true)
   --help, -h                 show help

Same issues when running the streams command in help for streams above. We have found that using the global -X flag seems to work.

docker run docker.redpanda.com/redpandadata/redpanda connect -X /benthos.yaml -X '/resources/*.yaml' streams '/streams/*.yaml'

Any idea if the plan is to use -X going forward or will the old Benthos global flags be usable again in the rpk binary?

[Jeffail]

Hey @trahim, is this with the latest version of the redpanda docker container? The issue has been fixed since the first release of rpk connect, but it's possible there are still some rough edges in build artifacts.

[trahim]

Hey @Jeffail apologies thought I but the versions in the above but yes with the latest version: v24.1.8

[emmanuelbertho]

Hello, I'm having the same issue. It is working with the last version of ghcr.io/benthosdev/benthos but not the last version of ocker.redpanda.com/redpandadata/redpanda:latest

[Jeffail]

Just a quick update, this has been fixed in a newer version of rpk, we also have stand alone docker images just containing connect without rpk as an alternative. I've also updated the cli for the next release of connect so that all flags can be made after the subcommand. This will make it possible to run connect streams -r ./foo.yaml ./streams/*.yaml and so on, which is much more intuitive and will be what we recommend in the future.

[tbnguyen1407]

@Jeffail I tested latest redpanda-connect binary and the issue still exists:

$ redpanda-connect --version
Version: 4.31.0
Date: 2024-07-19T14:44:05Z

$ redpanda-connect streams -c config.yaml -r resources/*.yaml streams/*.yaml
Incorrect Usage: flag provided but not defined: -c
...

[trahim]

I also get the same on the latest

redpanda@1bbe6351af6d:/$ rpk connect --version
Version: 4.32.1
Date: 2024-07-26T22:53:24Z
redpanda@1bbe6351af6d:/$ rpk connect -c /benthos.yaml --resources '/resources/*.yaml' streams '/streams/*.yaml'
Incorrect Usage: flag provided but not defined: -c

[mihaitodor]

There are some documentation updates in progress, but you should be able to use rpk connect streams -o /benthos.yaml --resources '/resources/*.yaml' '/streams/*.yaml'. Please try that and report back if it doesn't work.

[brknstrngz]

@mihaitodor unfortunately this does still not seem to work with 4.32.1:

Running as: redpanda-connect streams -o ./config.yaml --resources './resources/one/cache.yaml' streams './streams/one/stream.yaml'

Exits with: ERRO stream configuration file read error: failed to load config 'streams/one/stream.yaml': stream id (one_stream) collision from file: streams/one/stream.yaml @service=benthos

In the resources directory there is only one file with a memory cache resource referenced in the stream definition.

HTH

[mihaitodor]

@brknstrngz Oops, my bad! 🙈 I forgot to remove the second streams. Would you mind trying this?

> redpanda-connect streams -o ./config.yaml --resources './resources/one/cache.yaml' './streams/one/stream.yaml'

[brknstrngz]

Thank you @mihaitodor, it works!