Proposal to introduce notion of environments

[ekozynin]

It should be possible to define multiple remotes for the same environments. For example, Kafka cluster will consist of number of remote urls:

• broker(s)
• schema registry
• connector instance(s)
• KSql instance(s)

The above set of remotes (that logically forms a Kafka cluster) will be repeated for different environments (dev, prod).

Also we should be able to specify some common remote systems (for example Secret manager) per environment. 'servers' can de defined under the environment as well.

Do you think a concept of 'environment' should be introduced on the root level?

environment:
  localDev:
    servers:
        # servers here
    remotes:
      broker:
          # url, protocol, etc
      schemaRegistry:
          # url, protocol, etc
      connect:
          # url, protocol, etc
      secretManager:
          # url, protocol, etc

  staging:
    servers:
        # servers here
    remotes:
      broker:
          # url, protocol, etc
      schemaRegistry:
          # url, protocol, etc
      connect:
          # url, protocol, etc
      secretManager:
          # url, protocol, etc

[github-actions[bot]]

Welcome to AsyncAPI. Thanks a lot for reporting your first issue. Please check out our contributors guide and the instructions about a basic recommended setup useful for opening a pull request.

Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue.

[ekozynin]

Issue 65 proposes to solve the same problem by grouping servers.

I would say it may be used complimentary to the environment proposal.

Example: Kafka cluster usually defines a list of brokers, and any of them may be used by a client application to connect to a cluster. For this problem I would use grouping to specify that number of remotes (within an environment) actually represent brokers.

I would still use environment(s) to distinguish what group of brokers belongs to which environment.

environment:
  staging:
    remotes:
      broker-1:
          group: brokers
          # url, protocol, etc
      broker-2:
          group: brokers
          # url, protocol, etc
      schemaRegistry:
          # url, protocol, etc
      connect:
          # url, protocol, etc
      secretManager:
          # url, protocol, etc

  production:
    remotes:
      broker-1:
          group: brokers
          # url, protocol, etc
      broker-2:
          group: brokers
          # url, protocol, etc
      broker-3:
          group: brokers
          # url, protocol, etc
      schemaRegistry:
          # url, protocol, etc
      connect:
          # url, protocol, etc
      secretManager:
          # url, protocol, etc

[ekozynin]

One more use of environments is to specify variables to substitute in other definitions. Runtime value for the 'environment' will need to be passed to generator and other tools.

In this example 'x-replicas' value will be different depending on the environment.

environments:
  local:
    variables:
      example-channel-replicas: 1
  dev:
    variables:
      example-channel-replicas: 3

channels:
  example-channel:
    x-messageCompatibility: 'FULL' 
    x-message:
      $ref: '#/components/messages/ExampleMessage'
    bindings:
      kafka:
        x-partitions: 9
        x-replicas: 
          $ref: '#/environments/{environment}/variables/example-channel-replicas'
        x-configs:
          cleanup.policy: compact
          delete.retention.ms: 0
          confluent.value.schema.validation: 'true'

[magicmatatjahu]

@ekozynin Hi! Have in mind that https://github.com/asyncapi/spec/pull/531 will most likely be included in the next version of AsyncAPI, so how would you like to combine your idea with the idea of linking channels to servers?

Another thing: Server Reusability. If a server does not change between environments at all, would the user then have to duplicate the server definition for each environment? This seems to me to be something that would create a lot of boilerplate for the user.

The case with variables: I am not in favor of this solution because you are trying to tie variables (perhaps for generation etc) to a specification. Currently you already have this possibility in the current version (two options):

• parse the document and paste in the appropriate tokens (like {environment}) the data from the input
• use an extension that can be read in the tools.

How about marking the environment at the server level?

servers:
  brokerProd:
    environment: prod
    # url, protocol, etc
  schemaRegistryDev:
    environment: dev
    # url, protocol, etc

Or if you need also grouping then we can introduces labels or annotations (proposal copied from @GeraldLoeffler from first draft of proposal from https://github.com/asyncapi/spec/pull/531 PR) to define dynamically annotations:

brokerProd:
  labels:
    environment: prod
    group: brokers
  # url, protocol, etc
schemaRegistryDev:
  labels:
    environment: dev
    group: schemaRegistries
  # url, protocol, etc

[ekozynin]

Sorry my example wasn't detailed enough, server definitions are different between environments. Consider the below example.

Linking servers to channels will not help in this scenario, as in case of Kafka, channels (topics) are partitioned and located on different servers by Kafka internals. We need to define multiple brokers in asyncapi because this is how client applications connect to Kafka cluster. In case one broker is down, they will connect to any other broker in the list.

Introducing labels (as in @magicmatatjahu example above) for servers will solve this particular problem, as we will be able clearly mark server's type and environment.

environment:
  local:
    remotes:
      broker:
          group: brokers
          endpoint: localhost:9091
      schemaRegistry:
          endpoint: http://localhost:9080

  production:
    remotes:
      broker-1:
          group: brokers
          endpoint: production1.cloud
          security:
              - confluentBroker: []
      broker-2:
          group: brokers
          endpoint: production2.cloud
          security:
              - confluentBroker: []
      broker-3:
          group: brokers
          endpoint: production3.cloud
          security:
              - confluentBroker: []
      schemaRegistry:
          endpoint: schema-registry-prod.cloud
          security:
              - confluentSchemaRegistry: []

[ekozynin]

Since the spec has a notion of servers/remotes, means we are already including our environments/infrastructure into asyncapi file.

Example 1: Our servers/remotes configuration usually remains the same across multiple environments, except of some small changes, such as remote endpoint. In this scenario, and in case of few environments, we have to copy paste the same definition, just changing one or two attributes. Amount of boilerplate code increases. Using variables we can define different values for different environments, while keeping a single server/remote definition.

Example 2: In case of kafka, when we define kafka bindings for the channel as part of asyncapi spec, we may want to create only a single partition for local, while many more partitions for production use. I don't think it is do-able with the current version of asyncapi, nor I think there is anything in place for upcoming version 3. Using variables we can define different values for different environments, and using them in the bindings section.

For the latter example, I can use custom extension to define variables, parse and substitute them. The question is, should it become part of the asyncapi specification. We already can define different servers/remotes in the specification (aka different environments) in the spec, so I think we should think how to make it more re-usable and flexible for different use-cases.

[dalelane]

I definitely see a lot of value in being able to group related server types together - and, for schema registries in particular, having a proper place to put these details would be fantastic.

In my opinion, though, I don't think Kafka Connect belongs in here. Connect is really just an application, so I think should be treated the same as any other client application.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity :sleeping:

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience :heart:

[smoya]

@ekozynin I recently updated https://github.com/asyncapi/spec/issues/654, becoming mostly a similar suggestion on what you did.

WDYT about @fmvilas suggestion here: https://github.com/asyncapi/spec/issues/654#issuecomment-1074424027 Do you think it makes sense?

I'm happy to move forward and join efforts with you, so we can include this into v2.4.0 release next month.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity :sleeping:

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience :heart:

[smoya]

@fmvilas @derberg could you please remove the stale label? https://github.com/asyncapi/spec/issues/654 might fix this.

[fmvilas]

Done. Weird, supposedly, just leaving a comment here should remove the stale label 🤔 Or maybe I dreamed it 😄

[derberg]

@fmvilas not dream, it is true 😄 . Just remember that stale workflow runs just once a day. So we have remove-stale-when-updated but you need to wait until next schedule 🤷🏼

Example: https://github.com/asyncapi/spec/issues/738#issuecomment-1185510041

So better to unstale by comment as it works and kinda makes you think about the update info

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity :sleeping:

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience :heart: