Getting null for column ip_address_type in aws_ec2_gateway_load_balancer table.

[bigdatasourav]

Describe the bug Getting null for column ip_address_type even if there is a valid value available.

[ParthaI]

@bigdatasourav The API doesn't return IpAddressType value for Gateway Load Balancer, but we are getting the IpAddressType value for Application Load Balancer.

Both AWS SDK V1 Go API, and AWS SDK V2 Go API does not return any value for the ip_address_type column.

Raised a support request in AWS V2 GO SDK

[cjsrkd3321]

Same issue on me. aws_iam_user, aws_iam_group, aws_vpc_security_group etc.. tables are return NULL or very less information to me. I think it's problem that occured because it updated to aws sdk v2. I submit 2 pictures below.(left is mac m1(arm) / right is linux server(amd))

[LalitLab]

@cjsrkd3321 Appreciate raising the issue. We will test it out in the Linux server(AMD) and revert to you.

[cjsrkd3321]

@LalitTurbot Thank you. Additionaly, I'm using docker on linux server(amd). And version 0.71.0 operated well like below.

[LalitLab]

@cjsrkd3321 If possible, could you please share the version of the AWS plugin and Steampipe CLI you are using on the Linux machine?

steampipe plugin list to get the plugin version steampipe --version to get the CLI version

[cjsrkd3321]

This version is well operated.

steampipe@0407b8210445:/workspace/steampipe-mod-aws-thrifty$ steampipe --version
steampipe version 0.16.1

steampipe@0407b8210445:/workspace/steampipe-mod-aws-thrifty$ steampipe plugin list
+--------------------------------------------------+---------+-----------------------------------------------------------------------------------------------+
| Name                                             | Version | Connections                                                                                   |
+--------------------------------------------------+---------+-----------------------------------------------------------------------------------------------+
| hub.steampipe.io/plugins/turbot/[email protected]       | 0.71.0  | aws_prod,aws_log_dns,aws_sec_dns,aws_mas,aws_test,aws_ct_dns,aws_bak_dns,aws_sec,aws_sg,aws_a |
|                                                  |         | ll,aws_dev_dns,aws_prod_dns,aws_ct,aws_log,aws_dev,aws_bak,aws_dns,aws_mas_dns,aws_test_dns   |
| hub.steampipe.io/plugins/turbot/steampipe@latest | 0.5.0   | steampipe                                                                                     |
+--------------------------------------------------+---------+-----------------------------------------------------------------------------------------------+

This version is not well operated. ( INCLUDE VERSION 0.72.0 )

steampipe@5e2bf5420090:/workspace$ steampipe -v
steampipe version 0.16.1

steampipe@5e2bf5420090:/workspace$ steampipe plugin list
+--------------------------------------------------+---------+-----------------------------------------------------------------------------------------------+
| Name                                             | Version | Connections                                                                                   |
+--------------------------------------------------+---------+-----------------------------------------------------------------------------------------------+
| hub.steampipe.io/plugins/turbot/aws@latest       | 0.76.0  | aws_test_dns,aws_dev,aws_dns,aws_ct,aws_bak_dns,aws_test,aws_prod,aws_ct_dns,aws_prod_dns,aws |
|                                                  |         | _all,aws_dev_dns,aws_log_dns,aws_sec,aws_bak,aws_sg,aws_mas,aws_mas_dns,aws_log,aws_sec_dns   |
| hub.steampipe.io/plugins/turbot/steampipe@latest | 0.5.0   | steampipe                                                                                     |
+--------------------------------------------------+---------+-----------------------------------------------------------------------------------------------+

[LalitLab]

Hey @cjsrkd3321 If you could try running select * from aws_account, does it work for you? The aws_account table is not yet migrated to aws SDK v2. Also, did you find anything in the steampipe logs?

I am unable to replicate the issue on my side. I have tried running it on the Linux AMD and the Mac Arm. It works fine for me.

Could you share the details of your AWS steampipe config with secrets commented and the docker configuration so I can try the same on my side?

[cjsrkd3321]

@LalitTurbot

1. select * from aws_account -> Yes. It works well.

2. plugin logs are not same Left side return result but Right side return nothing Both are same version of plugin that 0.76.0.

3. steampipe aws.spc file

connection "aws_all" {
  plugin     = "aws"
  type        = "aggregator"
  connections = ["aws_ct", "aws_log", "aws_dev", "aws_test", "aws_prod", "aws_mas", "aws_sec", "aws_bak"]

  options "connection" {
    cache     = true # true, false
    cache_ttl = 3600  # expiration (TTL) in seconds
  }

  max_error_retry_attempts = 7
  min_error_retry_delay = 100

  ignore_error_codes = ["AccessDenied"]
}

......

connection "aws_ct" {
  plugin      = "aws"
  profile     = "default"
  regions     = ["ap-northeast-2"]

  options "connection" {
    cache     = false # true, false
    cache_ttl = 300  # expiration (TTL) in seconds
  }

  max_error_retry_attempts = 7
  min_error_retry_delay = 100

  ignore_error_codes = ["AccessDenied"]

......
}

4. docker configuration file

FROM turbot/steampipe:0.16.1

USER root

# Certificates for outbound connection
COPY our.crt /etc/ssl/certs/our.crt
COPY our.crt /usr/local/share/ca-certificates/our.crt
RUN update-ca-certificates

RUN apt-get update -y && apt-get install -y git curl unzip less

USER steampipe:0

RUN steampipe plugin install steampipe
RUN steampipe plugin install aws

WORKDIR /workspace

# expose postgres service default port
EXPOSE 9193

COPY docker-entrypoint.sh /usr/local/bin
COPY config /home/steampipe/.aws/config
COPY aws.spc /home/steampipe/.steampipe/config/aws.spc

ENTRYPOINT [ "docker-entrypoint.sh" ]
# dcker-entrypoint.sh : steampipe service start --foreground

[e-gineer]

@cjsrkd3321 I'd like to confirm what the parameters are that are causing the problem for you.

Could you please share the value of Works? (yes or no) for each row of this table?

OS	Steampipe	AWS plugin	Works?
MacOS	v0.16.x	v0.71.x	?
MacOS	v0.16.x	v0.76.x	?
Linux	v0.16.x	v0.71.x	?
Linux	v0.16.x	v0.76.x	?
Docker on Linux	v0.16.x	v0.71.x	?
Docker on Linux	v0.16.x	v0.76.x	?

Specifically, we're trying to determine if this is a general issue, or related to docker specifically?

Thanks!

[cjsrkd3321]

@e-gineer

OS	Steampipe	AWS plugin	Works?
MacOS	v0.16.x	v0.71.x	WORKS
MacOS	v0.16.x	v0.76.x	WORKS
Docker on Linux	v0.16.x	v0.71.x	WORKS
Docker on Linux	v0.16.x	v0.76.x	NOT WORKS

Unfortunately, I can't test on Linux now.(Maybe I can test next week) And, I submit picture that I had missed(or skipped?).

Both are 0.76.0, left is Mac M1 / right is Docker on Linux

aws_account query works well, but aws_vpc_security_group query not works that I intended(right side).

[cbruno10]

@cjsrkd3321 Can you please also run the same tests as above (on ARM and Docker on Linux), but with v0.72.0 as well? I'd be interested in the results as they may help narrow down root cause, i.e., if it's an AWS SDK update or something else we've changed in the plugins during or after v0.72.0. Thanks!

[cjsrkd3321]

[0.72.0] aws_vpc_security_group query works well.(ARM and AMD) But, I can't test aws_iam_user or aws_iam_group query bacause server(AMD) couldn't connect internet now.

Anyhow, additional tests can test next week 🥲 But, I think aws_iam_user and aws_iam_group queries will fail on 'Docker on Linux' because with this kind of problem I had didn't update after 0.71.0 on my server.

[cbruno10]

Thanks @cjsrkd3321 for the additional testing!

When you're able to again, on ARM and AMD, can you please test the following tables with AWS plugin versions v0.72.0 and v0.76.0:

• aws_ec2_key_pair (migrated to AWS SDK v2 in v0.75.0)
• aws_iam_group (migrated to AWS SDK v2 in v0.72.0)
• aws_s3_bucket (migrated to AWS SDK v2 in v0.72.0)
• aws_sns_topic (migrated to AWS SDK v2 in v0.72.0)
• aws_vpc (migrated to AWS SDK v2 in v0.75.0)

The aws_vpc_security_group table may have worked on v0.72.0 since it wasn't migrated to AWS SDK v2 until v0.75.0, so from the table set above, if all tables migrated in v0.72.0 fail in v0.72.0 and v0.75.0, and all tables migrated in v0.75.0 succeed in v0.72.0 and fail in v0.75.0, then it does seem very likely that the migration to AWS SDK v2 is the main culprit.

We're still attempting to reproduce from our side, but have not been able to yet. We'll post an update here if we're able to though.

[cjsrkd3321]

v0.72.0(abnormal) vs v0.71.0(normal)

Table	State
aws_ec2_key_pair	works well
aws_iam_group	not working
aws_iam_user	not working
aws_s3_bucket	abnormally working (different to real, it's including many duplicated)
aws_vpc	works well
aws_sns_topic	abnormally working (different to real, it's including many duplicated)

I can't test v0.75.0 now. Next time, I'll bring testing results on pure Linux machine and results for v0.75.0.

[cjsrkd3321]

Not working on Local Linux machine.. v0.78.0 I don't know why this. Anyhow, We should use on ARM machine after v0.72.0.

[github-actions[bot]]

'This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.'

[github-actions[bot]]

'This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.'

[github-actions[bot]]

'This issue was closed because it has been stalled for 90 days with no activity.'