Use botocore models to automatically define AWS nodes and relationships

[achantavy]

Description:

Describe your idea. Please be detailed. If a feature request, please describe the desired behavior, what scenario it enables, and how it would be used.

AWS defines object model documents in botocore, e.g. https://github.com/boto/botocore/blob/develop/botocore/data/ec2/2016-11-15/service-2.json. These documents can help us automatically define sync jobs.

We should try to prototype this for one AWS resource type first and then try to generalize.

[optional Relevant Links:]

Any extra documentation required to understand the issue.

GCP has a similar concept called service discovery - in future we can take advantage of this too.

[ramonpetgrave64]

Here's an example, getting the boto shape/schema and documentation from python.

n [1]: from pprint import pprint
   ...: import botocore
   ...: import boto3
   ...: 
   ...: ec2 = boto3.resource('ec2')
   ...: pprint(ec2.get_available_subresources())
   ...: 
   ...: shape = botocore.session.Session().get_service_model('ec2').shape_for('KeyPair')
   ...: 
   ...: members = shape.members
   ...: pprint(shape.members)
   ...: 
   ...: prop = members['KeyMaterial']
   ...: pprint(prop.serialization)
   ...: pprint(prop.type_name)
   ...: pprint(prop.documentation)
['ClassicAddress',
 'DhcpOptions',
 'Image',
 'Instance',
 'InternetGateway',
 'KeyPair',
 'NetworkAcl',
 'NetworkInterface',
 'NetworkInterfaceAssociation',
 'PlacementGroup',
 'Route',
 'RouteTable',
 'RouteTableAssociation',
 'SecurityGroup',
 'Snapshot',
 'Subnet',
 'Tag',
 'Volume',
 'Vpc',
 'VpcAddress',
 'VpcPeeringConnection']
OrderedDict([('KeyFingerprint', <StringShape(String)>),
             ('KeyMaterial', <StringShape(SensitiveUserData)>),
             ('KeyName', <StringShape(String)>),
             ('KeyPairId', <StringShape(String)>),
             ('Tags', <ListShape(TagList)>)])
{'name': 'keyMaterial'}
'string'
'<p>An unencrypted PEM encoded RSA or ED25519 private key.</p>'

[trodery]

Here's an example, getting the boto shape/schema and documentation from python.

I was able to auto-generate some basic models using the example you provided. The big issue I ran into is that only a few resources are available. Trying to use boto3.resource("ecr") results in the below exception message:

The available resources are:
   - cloudformation
   - cloudwatch
   - dynamodb
   - ec2
   - glacier
   - iam
   - opsworks
   - s3
   - sns
   - sqs

Consider using a boto3.client('ecr') instead of a resource for 'ecr'

Is there a way to get the service models from a client rather than a session? I didn't see anything obvious.

[ramonpetgrave64]

I hadn't realized that, but this file might offer some clues. https://github.com/boto/botocore/blob/master/botocore/model.py