Getting an intermittent error while using SDK's RetryHydrate

[Subhajit97]

I am working on a table where I want to retry the list hydrate function for certain errors. Hence, I have defined the RetryConfig in table's ListConfig

List: &plugin.ListConfig{
	Hydrate: tableDNSRecordList,
	RetryConfig: &plugin.RetryConfig{
		ShouldRetryErrorFunc: shouldRetryError(),
	},
	KeyColumns: plugin.KeyColumnSlice{
		{Name: "domain", Require: plugin.Required, Operators: []string{"="}},
		{Name: "type", Require: plugin.Optional, Operators: []string{"="}},
		{Name: "dns_server", Require: plugin.Optional, Operators: []string{"="}, CacheMatch: "exact"},
	},
},

func shouldRetryError() plugin.ErrorPredicateWithContext {
	return func(ctx context.Context, d *plugin.QueryData, h *plugin.HydrateData, err error) bool {
		if err != nil {
			panic(d.Table.Name)
			return true
		}
		return false
	}
}

Note: For the test, it returns true for any error

While running the below query (see attachment) multiple times test.sql.zip

• For a successful run

➜  test cat test.sql | steampipe query
+---------------+--------+--------------------------------------------------+
| resource      | status | reason                                           |
+---------------+--------+--------------------------------------------------+
| github.com    | ok     | github.com name servers have no CNAME record.    |
| microsoft.com | ok     | microsoft.com name servers have no CNAME record. |
+---------------+--------+--------------------------------------------------+

Time: 1915.2ms. Rows fetched: 53. Hydrate calls: 0.

• If an error occurs (returns panic, since I added the panic in shouldRetryError function)

➜  test cat test.sql | steampipe query
Warning: executeQueries: query 1 of 1 failed: ERROR: rpc error: code = Internal desc = list call tableDNSRecordList failed with panic net_dns_record (SQLSTATE HV000)

• For the same query, sometimes it returns the error (usually it should return the panic that I have added)

➜  test cat test.sql | steampipe query
Warning: executeQueries: query 1 of 1 failed: ERROR: rpc error: code = Unknown desc = read udp 192.168.1.103:54954->8.8.8.8:53: i/o timeout (SQLSTATE HV000)

Steps to reproduce:

• Use branch test-retry in Net plugin
• Clone Net Insights repo, and run steampipe check benchmark.dns_best_practices

[Subhajit97]

Here is the SQL query that I'm using

with domain_list as (
    select distinct domain from net_dns_record where domain in ('github.com', 'microsoft.com') order by domain
  ),
  domain_ns_records as (
    select domain, type, target from net_dns_record where domain in (select domain from domain_list) and type = 'NS' order by domain
  ),
  ns_ips as (
    select domain, ip, type, target, host(ip) as ip_text from net_dns_record where domain in (select target from domain_ns_records) and type = 'A' order by domain
  ),
  ns_record_with_ip as (
    select
      domain_ns_records.domain,
      domain_ns_records.target as name_server,
      host(ns_ips.ip) as ip_text
    from
      domain_ns_records
      left join ns_ips on domain_ns_records.target = ns_ips.domain
    where
      domain_ns_records.type = 'NS'
      and ns_ips.ip is not null
    order by domain_ns_records.target
  ),
  ns_record_with_record_count_stats as (
    select
      domain,
      name_server,
      (select count(*) from net_dns_record where domain = ns_record_with_ip.domain and dns_server = ns_record_with_ip.ip_text and type = 'CNAME') as cname_record_count,
      (select count(*) from net_dns_record where domain = ns_record_with_ip.domain and dns_server = ns_record_with_ip.ip_text and type not in ('CNAME')) as non_cname_record_count
    from
      ns_record_with_ip
  ),
  ns_record_with_cname_other as (
    select distinct domain from ns_record_with_record_count_stats where cname_record_count > 0 and non_cname_record_count > 0 order by domain
  )
  select
    domain_list.domain as resource,
    case
      when ns_record_with_cname_other is null then 'ok'
      else 'alarm'
    end as status,
    case
      when ns_record_with_cname_other is null then domain_list.domain || ' name servers have no CNAME record.'
      else domain_list.domain || ' name servers have CNAME record along with NS (or any other) records.'
    end as reason
  from
    domain_list
    left join ns_record_with_cname_other on domain_list.domain = ns_record_with_cname_other.domain;