Add custom Parameters from Glue into the Custom Properties tab

[AboutManuel]

Is your feature request related to a problem? Please describe. Nope.

Describe the solution you'd like We are currently ingesting tables using the ATHENA connector. (Not the Glue connector). This is tricky because Athena and Glue share a lot but if you use both connectors you can end up with duplicated information.

So, we are using the ATHENA connector and from this we are getting tables and their Lake Formation tags, lineage, etc. We would also like to get the Custom Parameters for both the tables and the columns that Glue has.

This could be an optional configuration in the connector.

Why would we need this? Because our workflow starts with the Data Producer doing a first ingestion into Athena / Glue and loading the metadata directly to Glue using a custom service we made.

Parameters – A map array of key-value pairs.

Each key is a Key string, not less than 1 or more than 255 bytes long, matching the Single-line string pattern.

Each value is a UTF-8 string, not more than 512000 bytes long.

These key-value pairs define properties associated with the table.

I've worked with the Glue and Lake Formation before so I can give some insights if needed.

With this:

# List all tables in the Data Catalog
response = glue_client.get_tables(DatabaseName='DATABASE_NAME')
tables = response['TableList']

You get a json with all the table data, including the JSONs with the parameters (for columns and tables)

Like this (all the fields inside "Parameters" here are custom fields created by us):

(...)
    {'Name': 'FIELD_NAME',
     'Type': 'string',
     'Comment': 'FIELD DESCRIPTION HERE',
     'Parameters': {'field_pk': 'TRUE',
      'field_description': 'FIELD DESCRIPTION HERE',
      'field_dq_rangedvalues': 'FALSE',
      'field_dq_business_logic_description': '',
      'field_dq_format': 'FALSE',
(...)

Describe alternatives you've considered Fetch custom metadata from a json file and ingest it through an API service.

Additional context https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-tables.html#aws-glue-api-catalog-tables-Column