OpenAPI spec issue

[fabianofpena]

Environment

• Airbyte version: Helm Chart version 0.43.22
• Deployment: Kuberentes (Helm Chart)
• Source Connector and version: Seems to be to any. Same behavior with Oracle, Google Sheets and MySQL
• Step where error happened: The error happened during the import using octavia CLI.

Current Behavior

I've upgraded from version 0.40.10 to 0.43.22. When I tried to import my existing connections (sources and destinations didn't fail the import)

Expected Behavior

Should be able to import my existing connections.

Logs

airbyte_api_client.exceptions.ApiAttributeError: AirbyteStreamConfiguration has no attribute 'suggested' at ['received_data']['sync_catalog']['streams'][0]['config']['suggested']

Steps to Reproduce

1. Create any source, destination and a connection using version 0.40.10.
2. Upgrade your Airbyte helm chart to 0.43.22
3. Try to import the the existing connection (octavia import)

[BenoitFayolle]

I think I'm running into a similar issue with a docker build

Environment

• Airbyte version: 0.40.30
• Deployment: Docker
• Source Connector and version: Google Sheets 0.2.31
• Step where error happened: Running octavia generate connection

Current Behavior

Local build of airbyte with docker. After successfully creating a Redshift destination and a Google Sheets source, the octavia generate connection command errors out

Expected Behavior

Successfully creates the connection

Logs

🐙 - Octavia is targetting your Airbyte instance running at http://localhost:8000 on workspace 9986ad57-4f20-4e60-8907-0184cd045691.
Traceback (most recent call last):
  File "/usr/local/bin/octavia", line 8, in <module>
    sys.exit(octavia())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/base_commands.py", line 54, in invoke
    raise e
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/base_commands.py", line 51, in invoke
    result = super().invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/generate/commands.py", line 76, in connection
    output_path = connection_renderer.write_yaml(project_path=".")
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/generate/renderers.py", line 197, in write_yaml
    rendered_yaml = self._render()
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/generate/renderers.py", line 294, in _render
    yaml_catalog = self.catalog_to_yaml(self.source.catalog)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/apply/resources.py", line 518, in catalog
    schema = self.api_instance.discover_schema_for_source(self.source_discover_schema_request_body)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api/source_api.py", line 1073, in discover_schema_for_source
    return self.discover_schema_for_source_endpoint.call_with_http_info(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api_client.py", line 849, in call_with_http_info
    return self.api_client.call_api(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api_client.py", line 410, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api_client.py", line 225, in __call_api
    return_data = self.deserialize(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api_client.py", line 326, in deserialize
    deserialized_data = validate_and_convert_types(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1570, in validate_and_convert_types
    converted_instance = attempt_convert_item(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1454, in attempt_convert_item
    return deserialize_model(input_value, valid_class,
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1374, in deserialize_model
    return model_class._new_from_openapi_data(**kw_args)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 370, in _new_from_openapi_data
    return cls._from_openapi_data(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model/source_discover_schema_read.py", line 203, in _from_openapi_data
    setattr(self, var_name, var_value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 185, in __setattr__
    self[attr] = value
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 510, in __setitem__
    self.set_attribute(name, value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 157, in set_attribute
    value = validate_and_convert_types(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1570, in validate_and_convert_types
    converted_instance = attempt_convert_item(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1454, in attempt_convert_item
    return deserialize_model(input_value, valid_class,
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1374, in deserialize_model
    return model_class._new_from_openapi_data(**kw_args)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 370, in _new_from_openapi_data
    return cls._from_openapi_data(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model/airbyte_catalog.py", line 174, in _from_openapi_data
    self.streams = streams
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 185, in __setattr__
    self[attr] = value
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 510, in __setitem__
    self.set_attribute(name, value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 157, in set_attribute
    value = validate_and_convert_types(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1620, in validate_and_convert_types
    input_value[index] = validate_and_convert_types(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1570, in validate_and_convert_types
    converted_instance = attempt_convert_item(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1454, in attempt_convert_item
    return deserialize_model(input_value, valid_class,
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1374, in deserialize_model
    return model_class._new_from_openapi_data(**kw_args)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 370, in _new_from_openapi_data
    return cls._from_openapi_data(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model/airbyte_stream_and_configuration.py", line 177, in _from_openapi_data
    setattr(self, var_name, var_value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 185, in __setattr__
    self[attr] = value
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 510, in __setitem__
    self.set_attribute(name, value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 157, in set_attribute
    value = validate_and_convert_types(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1570, in validate_and_convert_types
    converted_instance = attempt_convert_item(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1454, in attempt_convert_item
    return deserialize_model(input_value, valid_class,
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1374, in deserialize_model
    return model_class._new_from_openapi_data(**kw_args)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 370, in _new_from_openapi_data
    return cls._from_openapi_data(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model/airbyte_stream_configuration.py", line 201, in _from_openapi_data
    setattr(self, var_name, var_value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 185, in __setattr__
    self[attr] = value
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 510, in __setitem__
    self.set_attribute(name, value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 134, in set_attribute
    raise ApiAttributeError(
airbyte_api_client.exceptions.ApiAttributeError: AirbyteStreamConfiguration has no attribute 'suggested' at ['received_data']['catalog']['streams'][0]['config']['suggested']

[BenoitFayolle]

Same thing happening with Salesforce and MySQL source connectors

[moszutij]

I'm running into a similar issue with a docker build

Environment

• Airbyte version: 0.40.30
• Deployment: Docker
• Source Connector and version: S3 0.1.27
• Step where error happened: Running octavia import connection

Current Behavior

I've upgraded from version 0.40.26 to 0.40.30. When I tried to import my existing connections (sources and destinations didn't fail the import)

Expected Behavior

Should be able to import my existing connections.

Logs

🐙 - Octavia is targetting your Airbyte instance running at http://localhost:8000 on workspace af4f9d13-d672-44f4-ae52-47fd1c657d84.
Traceback (most recent call last):
  File "/usr/local/bin/octavia", line 8, in <module>
    sys.exit(octavia())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/base_commands.py", line 54, in invoke
    raise e
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/base_commands.py", line 51, in invoke
    result = super().invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/check_context.py", line 91, in wrapper
    f(ctx, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/_import/commands.py", line 156, in connection
    click.echo(import_connection(ctx.obj["API_CLIENT"], ctx.obj["WORKSPACE_ID"], resource))
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/_import/commands.py", line 120, in import_connection
    new_configuration_path = connection_renderer.import_configuration(".", remote_configuration)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/generate/renderers.py", line 314, in import_configuration
    rendered = self._render()
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/generate/renderers.py", line 294, in _render
    yaml_catalog = self.catalog_to_yaml(self.source.catalog)
  File "/usr/local/lib/python3.9/site-packages/octavia_cli/apply/resources.py", line 505, in catalog
    schema = self.api_instance.discover_schema_for_source(self.source_discover_schema_request_body)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api/source_api.py", line 970, in discover_schema_for_source
    return self.discover_schema_for_source_endpoint.call_with_http_info(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api_client.py", line 849, in call_with_http_info
    return self.api_client.call_api(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api_client.py", line 410, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api_client.py", line 225, in __call_api
    return_data = self.deserialize(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/api_client.py", line 326, in deserialize
    deserialized_data = validate_and_convert_types(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1570, in validate_and_convert_types
    converted_instance = attempt_convert_item(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1454, in attempt_convert_item
    return deserialize_model(input_value, valid_class,
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1374, in deserialize_model
    return model_class._new_from_openapi_data(**kw_args)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 370, in _new_from_openapi_data
    return cls._from_openapi_data(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model/source_discover_schema_read.py", line 203, in _from_openapi_data
    setattr(self, var_name, var_value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 185, in __setattr__
    self[attr] = value
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 510, in __setitem__
    self.set_attribute(name, value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 157, in set_attribute
    value = validate_and_convert_types(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1570, in validate_and_convert_types
    converted_instance = attempt_convert_item(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1454, in attempt_convert_item
    return deserialize_model(input_value, valid_class,
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1374, in deserialize_model
    return model_class._new_from_openapi_data(**kw_args)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 370, in _new_from_openapi_data
    return cls._from_openapi_data(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model/airbyte_catalog.py", line 174, in _from_openapi_data
    self.streams = streams
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 185, in __setattr__
    self[attr] = value
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 510, in __setitem__
    self.set_attribute(name, value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 157, in set_attribute
    value = validate_and_convert_types(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1620, in validate_and_convert_types
    input_value[index] = validate_and_convert_types(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1570, in validate_and_convert_types
    converted_instance = attempt_convert_item(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1454, in attempt_convert_item
    return deserialize_model(input_value, valid_class,
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1374, in deserialize_model
    return model_class._new_from_openapi_data(**kw_args)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 370, in _new_from_openapi_data
    return cls._from_openapi_data(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model/airbyte_stream_and_configuration.py", line 177, in _from_openapi_data
    setattr(self, var_name, var_value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 185, in __setattr__
    self[attr] = value
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 510, in __setitem__
    self.set_attribute(name, value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 157, in set_attribute
    value = validate_and_convert_types(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1570, in validate_and_convert_types
    converted_instance = attempt_convert_item(
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1454, in attempt_convert_item
    return deserialize_model(input_value, valid_class,
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 1374, in deserialize_model
    return model_class._new_from_openapi_data(**kw_args)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 370, in _new_from_openapi_data
    return cls._from_openapi_data(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 46, in wrapped_init
    return fn(_self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model/airbyte_stream_configuration.py", line 201, in _from_openapi_data
    setattr(self, var_name, var_value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 185, in __setattr__
    self[attr] = value
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 510, in __setitem__
    self.set_attribute(name, value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 134, in set_attribute
    raise ApiAttributeError(
airbyte_api_client.exceptions.ApiAttributeError: AirbyteStreamConfiguration has no attribute 'suggested' at ['received_data']['catalog']['streams'][0]['config']['suggested']

[piotr-struzik]

Same issue Managed to narrow it down to importing connections (Postgres -> Redshift) Source and Destination imports work but

octavia import connection "xxxxxxx-xxxx-xxxx-xxxxxxxxxxxx" Fails with:

  File "/usr/local/lib/python3.9/site-packages/airbyte_api_client/model_utils.py", line 134, in set_attribute
    raise ApiAttributeError(
airbyte_api_client.exceptions.ApiAttributeError: AirbyteStreamConfiguration has no attribute 'suggested' at ['received_data']['catalog']['streams'][0]['config']['suggested']

Webapp version: 0.40.32 Octavia CLI version: 0.40.32

[mattppal]

Interesting, for me this fails on the destination Redshift. Connections and sources work just fine:

TypeError: _from_openapi_data() missing 2 required positional arguments: 'supports_dbt' and 'normalization_config'

Webapp version: 0.40.32 Octavia CLI version: 0.40.32

[avi-kipi]

I have successfully created sources and destination in my Airbyte instakce using Octavia CLI but while generating connection configuration.yml file getting error. " raise ApiAttributeError( airbyte_api_client.exceptions.ApiAttributeError: AirbyteStreamConfiguration has no attribute 'suggested' at ['received_data']['catalog']['streams'][0]['config']['suggested']"

[fabianofpena]

I've found the fallowing solution:

1. Saved the install.sh in my local environment (https://raw.githubusercontent.com/airbytehq/airbyte/master/octavia-cli/install.sh), since the version 0.42.0 does not exists in the registry
2. Changed the $VERSION to last octavia cli docker image available on dockerhub (VERSION=0.42.1)
3. Then executed the install.sh file to update my octavia cli docker image

[octavia-squidington-iii]

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.

[octavia-squidington-iii]

This issue was closed because it has been inactive for 20 days since being marked as stale.