terraformer
terraformer copied to clipboard
GoogleCloudPlatform
AWS import iam fails with repeated RPC errors
My Env
terraformer-dir % terraform --version
Terraform v1.1.9
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v4.6.0
terraformer-dir % terraformer --version
version v0.8.19
terraformer-dir % ls -l ~/.terraform.d/plugins/registry.terraform.io/hashicorp/aws/4.6.0/
total 0
drwxr-xr-x 3 bgriffis staff 96 Mar 23 06:29 darwin_arm64
terraformer-dir % terraformer import aws --resources=iam --regions=us-east-2 --profile=someprofile
2022/05/11 14:43:33 aws importing default region
2022/05/11 14:43:33 aws importing... iam
2022/05/11 14:43:38 aws done importing iam
2022/05/11 14:43:38 Number of resources for service iam: 67
2022/05/11 14:43:38 Refreshing state... aws_iam_user.tfer--xxxxx
2022/05/11 14:43:38 Refreshing state... aws_iam_user_group_membership.tfer--v-002E-xxxxx
2022/05/11 14:43:38 Refreshing state... aws_iam_role_policy_attachment.tfer--xxxxx
2022/05/11 14:43:38 Refreshing state... aws_iam_role.tfer--AWSServiceRoleFoxxxxx
2022/05/11 14:43:38 Refreshing state... aws_iam_user.tfer--AIDxxxxx
2022/05/11 14:43:38 Refreshing state... aws_iam_user.tfer--AIDxxxxxx
2022/05/11 14:43:38 Refreshing state... aws_iam_role_policy_attachment.tfer--xxxxx
2022/05/11 14:43:38 Refreshing state... aws_iam_role.tfer--AWSServiceRoleForElasticLoadBalancing
2022/05/11 14:43:38 Refreshing state... aws_iam_instance_profile.tfer--ecsInstanceRole
2022/05/11 14:43:38 Refreshing state... aws_iam_user_policy_attachment.tfer--v-002xxxxx
2022/05/11 14:43:38 Refreshing state... aws_iam_role.tfer--AWSServiceRoleForAutoScaling
2022/05/11 14:43:38 Refreshing state... aws_iam_user.tfer--AIDxxxxx
2022/05/11 14:43:38 Refreshing state... aws_iam_user_policy_attachment.tfer--a-002E-xxxxx
2022/05/11 14:43:38 Refreshing state... aws_iam_role_policy_attachment.tfer--AWSServiceRoleForElasticLoadBalancing_AWSElasticLoadBalancingServiceRolePolicy
2022/05/11 14:43:38 Refreshing state... aws_iam_policy.tfer--xxxxxx
2022/05/11 14:43:38 rpc error: code = Unavailable desc = error reading from server: EOF
2022/05/11 14:43:38 WARN: Fail read resource from provider, wait 300ms before retry
2022/05/11 14:43:38 rpc error: code = Unavailable desc = error reading from server: EOF
2022/05/11 14:43:38 WARN: Fail read resource from provider, wait 300ms before retry
2022/05/11 14:43:38 rpc error: code = Unavailable desc = error reading from server: EOF
Those last two lines repeat A LOT! Occasionally I'll get some of these:
2022/05/11 14:45:13 Fail read resource from provider, trying import command
2022/05/11 14:45:13 Refreshing state... aws_iam_role.tfer--AWSServiceRoleForECS
2022/05/11 14:45:13 Fail read resource from provider, trying import command
and finally
2022/05/11 14:45:19 Filtered number of resources for service iam: 0
2022/05/11 14:45:19 aws Connecting....
2022/05/11 14:45:19 aws save iam
2022/05/11 14:45:19 Can't find provider version. Ensure that you are following https://www.terraform.io/docs/configuration/providers.html#plugin-names-and-versions.
2022/05/11 14:45:19 aws save tfstate for iam
terraformer-dir % tree generated/aws/iam
generated/aws/iam
├── provider.tf
└── terraform.tfstate
terraformer-dir % cat generated/aws/iam/provider.tf
provider "aws" {
region = "us-east-1"
}
terraform {
required_providers {
aws = {
version = ""
}
}
}
But when I set --resources=* it seemed to pull IAM resources ok.
terraformer-dir % tree generated/aws/iam
iam
├── iam_group.tf
├── iam_group_policy_attachment.tf
├── iam_instance_profile.tf
├── iam_policy.tf
├── iam_role.tf
├── iam_role_policy_attachment.tf
├── iam_user.tf
├── iam_user_group_membership.tf
├── iam_user_policy_attachment.tf
├── outputs.tf
├── provider.tf
└── terraform.tfstate
Am I missing something?
Having the same issue as OP
Using same version of terraform and terraformer, but I am using AWS providers version 4.13.0
It seems that terraformer found a ton of resources... things were looking good...
2022/05/12 17:53:24 aws done importing xray
2022/05/12 17:53:24 Number of resources for service cloudwatch: 177
2022/05/12 17:53:24 Number of resources for service servicecatalog: 0
2022/05/12 17:53:24 Number of resources for service vpn_connection: 0
2022/05/12 17:53:24 Number of resources for service waf_regional: 7
2022/05/12 17:53:24 Number of resources for service api_gateway: 22
2022/05/12 17:53:24 Number of resources for service wafv2_cloudfront: 0
2022/05/12 17:53:24 Number of resources for service alb: 513
2022/05/12 17:53:24 Number of resources for service devicefarm: 0
2022/05/12 17:53:24 Number of resources for service rds: 55
2022/05/12 17:53:24 Number of resources for service efs: 11
2022/05/12 17:53:24 Number of resources for service workspaces: 0
2022/05/12 17:53:24 Number of resources for service ecr: 732
2022/05/12 17:53:24 Number of resources for service ecrpublic: 0
2022/05/12 17:53:24 Number of resources for service redshift: 9
2022/05/12 17:53:24 Number of resources for service route_table: 125
2022/05/12 17:53:24 Number of resources for service ssm: 3242
2022/05/12 17:53:24 Number of resources for service wafv2_regional: 0
2022/05/12 17:53:24 Number of resources for service nacl: 6
2022/05/12 17:53:24 Number of resources for service nat: 6
2022/05/12 17:53:24 Number of resources for service ses: 2
2022/05/12 17:53:24 Number of resources for service config: 6
2022/05/12 17:53:24 Number of resources for service firehose: 6
2022/05/12 17:53:24 Number of resources for service opsworks: 10
2022/05/12 17:53:24 Number of resources for service swf: 0
2022/05/12 17:53:24 Number of resources for service waf: 1
2022/05/12 17:53:24 Number of resources for service acm: 16
2022/05/12 17:53:24 Number of resources for service appsync: 0
2022/05/12 17:53:24 Number of resources for service ebs: 848
2022/05/12 17:53:24 Number of resources for service ec2_instance: 405
2022/05/12 17:53:24 Number of resources for service eni: 855
2022/05/12 17:53:24 Number of resources for service sqs: 1000
2022/05/12 17:53:24 Number of resources for service transit_gateway: 0
2022/05/12 17:53:24 Number of resources for service docdb: 25
2022/05/12 17:53:24 Number of resources for service es: 3
2022/05/12 17:53:24 Number of resources for service securityhub: 0
2022/05/12 17:53:24 Number of resources for service subnet: 114
2022/05/12 17:53:24 Number of resources for service vpn_gateway: 2
2022/05/12 17:53:24 Number of resources for service codepipeline: 0
2022/05/12 17:53:24 Number of resources for service elasticache: 64
2022/05/12 17:53:24 Number of resources for service elb: 61
2022/05/12 17:53:24 Number of resources for service igw: 6
2022/05/12 17:53:24 Number of resources for service logs: 211
2022/05/12 17:53:24 Number of resources for service organization: 25
2022/05/12 17:53:24 Number of resources for service qldb: 0
2022/05/12 17:53:24 Number of resources for service cloud9: 0
2022/05/12 17:53:24 Number of resources for service codecommit: 0
2022/05/12 17:53:24 Number of resources for service ecs: 1
2022/05/12 17:53:24 Number of resources for service iam: 2962
2022/05/12 17:53:24 Number of resources for service iot: 0
2022/05/12 17:53:24 Number of resources for service media_store: 0
2022/05/12 17:53:24 Number of resources for service codedeploy: 0
2022/05/12 17:53:24 Number of resources for service datapipeline: 0
2022/05/12 17:53:24 Number of resources for service dynamodb: 20
2022/05/12 17:53:24 Number of resources for service sfn: 0
2022/05/12 17:53:24 Number of resources for service sg: 669
2022/05/12 17:53:24 Number of resources for service vpc_peering: 54
2022/05/12 17:53:24 Number of resources for service s3: 309
2022/05/12 17:53:24 Number of resources for service cloudhsm: 0
2022/05/12 17:53:24 Number of resources for service codebuild: 1
2022/05/12 17:53:24 Number of resources for service cognito: 4
2022/05/12 17:53:24 Number of resources for service emr: 0
2022/05/12 17:53:24 Number of resources for service kinesis: 8
2022/05/12 17:53:24 Number of resources for service lambda: 62
2022/05/12 17:53:24 Number of resources for service route53: 3159
2022/05/12 17:53:24 Number of resources for service sns: 26
2022/05/12 17:53:24 Number of resources for service budgets: 26
2022/05/12 17:53:24 Number of resources for service customer_gateway: 0
2022/05/12 17:53:24 Number of resources for service eks: 0
2022/05/12 17:53:24 Number of resources for service elastic_beanstalk: 0
2022/05/12 17:53:24 Number of resources for service glue: 28
2022/05/12 17:53:24 Number of resources for service secretsmanager: 0
2022/05/12 17:53:24 Number of resources for service xray: 1
2022/05/12 17:53:24 Number of resources for service media_package: 0
2022/05/12 17:53:24 Number of resources for service auto_scaling: 6
2022/05/12 17:53:24 Number of resources for service batch: 0
2022/05/12 17:53:24 Number of resources for service cloudformation: 6
2022/05/12 17:53:24 Number of resources for service cloudfront: 2
2022/05/12 17:53:24 Number of resources for service cloudtrail: 1
2022/05/12 17:53:24 Number of resources for service eip: 700
2022/05/12 17:53:24 Number of resources for service kms: 18
2022/05/12 17:53:24 Number of resources for service msk: 0
2022/05/12 17:53:24 Number of resources for service resourcegroups: 0
2022/05/12 17:53:24 Number of resources for service vpc: 6
2022/05/12 17:53:24 Refreshing state... aws_
Continues like that for a while with this happening every now and then...
2022/05/12 17:53:25 WARN: Fail read resource from provider, wait 300ms before retry
2022/05/12 17:53:25 Fail read resource from provider, trying import command
2022/05/12 17:53:25 Fail read resource from provider, trying import command
2022/05/12 17:53:25 Fail read resource from provider, trying import command
2022/05/12 17:53:25 Fail read resource from provider, trying import command
2022/05/12 17:53:25 rpc error: code = Canceled desc = context canceled
2022/05/12 17:53:25 Fail read resource from provider, trying import command
2022/05/12 17:53:25 rpc error: code = Canceled desc = context canceled
2022/05/12 17:53:25 Fail read resource from provider, trying import command
2022/05/12 17:53:25 rpc error: code = Canceled desc = context canceled
2022/05/12 17:53:25 Refreshing state... aws_ssm_parameter.tfer---
Continues for another 30 minutes like this then
`2022/05/12 18:29:25 ERROR: Unable to refresh resource tfer--
Any ideas as to why it keeps failing out? Doesn't matter if I use resources="*" or specify a resource...
Also fails with this at the end:
goroutine 1 [running]: github.com/GoogleCloudPlatform/terraformer/terraformutils.(*Resource).ParseTFstate(0xc0030145a0, {0xbd2faa0, 0xc00457a230}, {{0xbf40c88, 0xc004529530}}) github.com/GoogleCloudPlatform/terraformer/terraformutils/resource.go:151 +0xff github.com/GoogleCloudPlatform/terraformer/terraformutils.(*Resource).ConvertTFstate(0xc0030145a0, 0xc000c31f10) github.com/GoogleCloudPlatform/terraformer/terraformutils/resource.go:176 +0x397 github.com/GoogleCloudPlatform/terraformer/terraformutils.(*ProvidersMapping).ConvertTFStates(0xc004449690, 0x54) github.com/GoogleCloudPlatform/terraformer/terraformutils/providers_mapping.go:151 +0xab github.com/GoogleCloudPlatform/terraformer/cmd.Import({_, _}, {{0xc000ff9800, 0x54, 0x80}, {0x129ac258, 0x0, 0x0}, {0xaf7a310, 0x1e}, ...}, ...) github.com/GoogleCloudPlatform/terraformer/cmd/import.go:102 +0x2c9 github.com/GoogleCloudPlatform/terraformer/cmd.importRegionResources({{0xc000635ff0, 0x1, 0x1}, {0x129ac258, 0x0, 0x0}, {0xaf7a310, 0x1e}, {0xaee2b02, 0x9}, ...}, ...) github.com/GoogleCloudPlatform/terraformer/cmd/provider_cmd_aws.go:121 +0x278 github.com/GoogleCloudPlatform/terraformer/cmd.newCmdAwsImporter.func1(0xc000b8a2c0, {0xaed5235, 0x7, 0x7}) github.com/GoogleCloudPlatform/terraformer/cmd/provider_cmd_aws.go:58 +0x4b0 github.com/spf13/cobra.(*Command).execute(0xc000b8a2c0, {0xc0004bc4d0, 0x7, 0x7}) github.com/spf13/[email protected]/command.go:850 +0x60e github.com/spf13/cobra.(*Command).ExecuteC(0xc000ff62c0) github.com/spf13/[email protected]/command.go:958 +0x3ad github.com/spf13/cobra.(*Command).Execute(...) github.com/spf13/[email protected]/command.go:895 github.com/GoogleCloudPlatform/terraformer/cmd.Execute() github.com/GoogleCloudPlatform/terraformer/cmd/root.go:36 +0x1e main.main() github.com/GoogleCloudPlatform/terraformer/main.go:39 +0x35
I also face a similar problem as well when i try to import AWS iam together with another AWS resource like sns, codecommit etc. Importing iam alone and filtering it by tags works fine. See error below.
2022/07/02 08:02:09 WARN: Fail read resource from provider, wait 300ms before retry 2022/07/02 08:02:09 Fail read resource from provider, trying import command panic: interface conversion: interface {} is nil, not string
goroutine 1 [running]: github.com/GoogleCloudPlatform/terraformer/providers/aws.(*IamGenerator).PostConvertHook(0xc0009f8420) github.com/GoogleCloudPlatform/terraformer/providers/aws/iam.go:371 +0x4fd github.com/GoogleCloudPlatform/terraformer/providers/aws.(*AwsFacade).PostConvertHook(0xc002f2b328?) github.com/GoogleCloudPlatform/terraformer/terraformutils/providers_mapping.go:175 +0xb4 github.com/GoogleCloudPlatform/terraformer/cmd.Import({_, _}, {{0xc0006a1120, 0x2, 0x2}, {0x11e48aa0, 0x0, 0x0}, {0xad37714, 0x1e}, ...}, ...) github.com/GoogleCloudPlatform/terraformer/cmd/import.go:104 +0x2d2 github.com/GoogleCloudPlatform/terraformer/cmd.importRegionResources({{0xc0006a1080, 0x2, 0x2}, {0x11e48aa0, 0x0, 0x0}, {0xad37714, 0x1e}, {0xac9d73d, 0x9}, ...}, ...) github.com/GoogleCloudPlatform/terraformer/cmd/provider_cmd_aws.go:121 +0x272 github.com/GoogleCloudPlatform/terraformer/cmd.newCmdAwsImporter.func1(0xc0000a3080?, {0xac8fc10?, 0x2?, 0x2?}) github.com/GoogleCloudPlatform/terraformer/cmd/provider_cmd_aws.go:66 +0x128 github.com/spf13/cobra.(*Command).execute(0xc0000a3080, {0xc0006a0fa0, 0x2, 0x2}) github.com/spf13/[email protected]/command.go:850 +0x67c github.com/spf13/cobra.(*Command).ExecuteC(0xc0000a22c0) github.com/spf13/[email protected]/command.go:958 +0x39c github.com/spf13/cobra.(*Command).Execute(...) github.com/spf13/[email protected]/command.go:895 github.com/GoogleCloudPlatform/terraformer/cmd.Execute() github.com/GoogleCloudPlatform/terraformer/cmd/root.go:36 +0x1e main.main() github.com/GoogleCloudPlatform/terraformer/main.go:39 +0x50 zsh: command not found: --regions=us-east-1
any update , same error :
with enviroment Terraformer v0.8.21 Terraform v1.2.6 on darwin_amd64
- provider registry.terraform.io/hashicorp/aws v3.75.2
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue was closed because it has been stalled for 7 days with no activity.