airbyte
airbyte copied to clipboard
airbytehq
Connectors: Rename to increase user experience (AWS connector)
Tell us about the documentation you'd like us to add or update.
As a user it might be problematic to find certain connectors because of their deisplay name. For example, there is a Amazon SQS, S3 and an AWS CloudTrail Source.
The naming should be reworked in a way, that a user can easily find the AWS connector by adding the prefix AWS.
If applicable, add links to the relevant docs that should be updated
Screenshots addressing this issue
Turns out, that the changes are minimal. Although I would like to gather your thoughts on this topic.
In addition to the renaming in the specs, the documentation and all references to them need to be updated, too. Before I start this (relatively) time consuming task: Thoughts please (@sajarin @marcosmarxm & others ofc!)
I think it is a good idea maybe taking a look for Google and Azure too.
@YowanR @misteryeo what are your thoughts here?
I think it is a good idea maybe taking a look for Google and Azure too.
Yeah, definitely searching for further cases in following steps. I just wanted to state a very obvious case here at first and gather your thoughts about this.👍🏼
@juweins - this is a great idea and something we're already thinking about, a concept of 'aliases'.
I'm tagging in @evantahler as his team is already thinking about this - to see if a short term implementation here might be helpful or not considering our longer term plans!
I read this as both an inconsistent naming convention problem and a search problem.
"Aliases" is a term mean to create an additional connector offering sharing the same underlying code, eg "AWS RDS Postgres" and "Google Cloud SQL Postgres" both could be aliases of "source-postgres" to aid discovery. But, in this case, there would be entries for each of these connectors:
search term: "postgres"
results:
* postgres
* AWS RDS Postgres
* AWS Aurora Postgres
* Google Cloud SQL Postgres
I don't think that's what we want here.
What's happening here is that our connector search should consider more than just the connector's name. We have docs and tags for connectors (coming soon with the "connector metadata" project), but we don't use this information in search. We want all of these searches to produce a result which contains the source-s3 connector
search term: "s3"
search term: "amazon"
search term: "aws"
Also, I agree we should have consistent connector naming, eg. AWS _name_of_service, so by that rule: Amazon SQS --> AWS SQL and S3 --> AWS S3 should be renamed. I think in the past we didn't want to say "AWS S3" because the connector is compatible with any S3 compatible destination (e.g. GCS), but I agree that the name is probably more confusing that not.
Tagging @bechurch as well, who is thinking about connector metadata, and @sherifnada who thinks about connector names.
@misteryeo Thanks for feedback. @evantahler Thanks for the insight! In fact, I totally agree with you.
Please let me know if I can be of any help in this refactoring. :)
@juweins we'd love some help renaming things! The good news is that I think the changes would be localized to the connector's name in this file and the docs. We won't need to release a new version or change the directory structure.
Would you like to propose a more consistent naming scheme?
Tagging @airbytehq/frontend as there are some good thoughts on "connector search" in this issue
@juweins we'd love some help renaming things! The good news is that I think the changes would be localized to the connector's name in this file and the docs. We won't need to release a new version or change the directory structure.
Would you like to propose a more consistent naming scheme?
I can verify, that the source_definitions.yaml and destination_definitions.yaml are the only files affected in a simple renaming. However, transferring these changes may imply tedious adjustments in the docs for consistency.
Regarding naming scheme: I will think about this, I do not have any specific in mind yet. 👍🏼
So I finally had the time to think about this, here is my point of view - discussion appreciated!
Here are my thoughts on the building blocks:
direction: The intention of the connector stays clear even if abbreviated. In the current repo, I face some extra long connector names, which may be annoying to developers. Using a clear 3 char direction indicator maintains readability and makes space for the important naming blocks.
vendor: Currently we are facing this issue with an inconsitent naming scheme. How do we categorize connectors and achieve a distinction between them? I find it useful to agree on a 3 char abbreviation for certain vendors. This way, we can differentiate clearly between vendors in certain markets (amazon, aws | google, gcp | ...) The vendor part may be skipped for single products, e.g. appfollow, Appsflyer, Appstore-singer (Thoughts?), and may be introduced when there are 2+ connectors for a vendor.
product: The current free naming should be limited from 3-10 (12,14?) chars in total. (Well, s3 may be a valid exception) This way, we get concise names for present and upcoming connectors.
version: How do we handle new api versions, but maintain older ones? This part will be important in the future for example, as some vendors might still support e.g. api version v1, but release a v2. We can maintain backward compatibility for existing pipelines and increase our "freedom" for new/otherwise breaking features. I have faced this upcoming issue by working on an issue for the atlassian confluence connector. They have two api version and the current connector is built on v1. see Atlassian Developer
suffix: There are a few connectors which are specific to certain capaibilites (e.g. *-strict-encrypt). This can be included in the naming scheme by aggreeing on a selection of suffixes (e.g. "strict", "batch", "stream" ....)
That being said, I am happy to hear your feedback & thoughts on this! :) @evantahler , @misteryeo, @marcosmarxm & all others reading/thinking about this!
Note: This is only focusing on the naming convention assuming that the connector search (in the UI) is not connected to those names.
BTW: What are the search features/indexes in the UI as of now? There should be some kind of (automatic?) key word tagging for the search field in the future. Example: Apart from the connector name, let's say "Database" should filter for all databases that are currently available in my airbyte version.
If you are referring to this component:
then it's only searching the display names at the moment AFAICT
+1 on the expanding the ability to search in that component for at least the connector name. Happy to create another issue if it diverges too much from the original issue.
What's happening here is that our connector search should consider more than just the connector's name. We have docs and tags for connectors (coming soon with the "connector metadata" project), but we don't use this information in search.
@bnchrch are tags being added as part of the connector metadata? I checked a couple of metadata.yaml files and didn't see a tags field. That would be a prerequisite for us to add the functionality to search connectors by these tags in the Airbyte UI
There has been discussions about adding tags (in a few different forms), but no issue created yet, so lets do that!
#25741
This is a very small amount of work so I imagine we can add this really quickly
@bnchrch Thanks for creating the issue, that would be a great addition to the connector metadata to allow things like searching by tags or even better telemetry
At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.
