detecting and warning about non-optimal hydration

[judell]

Query 1 is fast.

select 
  title
from 
  aws_iam_policy 
where
  title = 'IAMFullAccess'

+---------------+
| title         |
+---------------+
| IAMFullAccess |
+---------------+

Time: 254.027107ms

Query 2 is slow.

select 
  title,
  policy_std
from 
  aws_iam_policy 
where
  title = 'IAMFullAccess'

+---------------+----------------------------------------
| title         | policy_std                             
+---------------+----------------------------------------
| IAMFullAccess | {"Statement":[{"Action":["iam:*","organ
+---------------+----------------------------------------

Time: 33.509729367s

There are ~920 policies in the table, under the covers we call the getIamPolicy hydrator for each, even though we only need to call it once.

Query 3 is fast again.

select 
  title,
  policy_std
from 
  aws_iam_policy 
where
  arn = 'arn:aws:iam::aws:policy/IAMFullAccess'

+---------------+----------------------------------------
| title         | policy_std                             
+---------------+----------------------------------------
| IAMFullAccess | {"Statement":[{"Action":["iam:*","organ
+---------------+----------------------------------------

Time: 424.731557ms

Looking at the implementation in table_aws_iam_policy.go I think I see what's going on.

	var arn string

	if h.Item != nil {
		policy := h.Item.(*iam.Policy)
		arn = *policy.Arn
	} else {
		arn = d.KeyColumnQuals["arn"].GetStringValue()
	}

	// Create Session
	svc, err := IAMService(ctx, d)
	if err != nil {
		return nil, err
	}

	params := &iam.GetPolicyInput{
		PolicyArn: &arn,
	}

The API call is keyed to arn, so if we know the arn we can avoid all the unnecessary hydration calls.

Clearly it would be helpful to surface the quals in the hub documentation. I'm also wondering about a general way to

• detect queries like query 2 as they happen

• tell the user that query 3 would be better