Allow filtering of resources

[iragsdale]

Thanks so much for writing this, it looks like it's gonna be awesome for migrating existing resources into Terraform!

However, I would prefer to migrate in smaller chunks, so I think it would be awesome if we could specify some basic parameter matching on the command line so that we only generate files for specific resources.

If you're interested in a pull request, I might give this a shot myself.

[nrcxcia]

This would also really help me. +1

[dtan4]

Like this?

$ terraforming s3 --resource hoge
resource "aws_s3_bucket" "hoge" {
    bucket = "hoge"
    acl    = "private"
}

It sounds good and useful if it exists. @iragsdale Could you send a pull request if you already have an idea of implementation?

[iragsdale]

Something like that, yeah. I'll see if I can take a crack at it soon.

[iragsdale]

So this isn't quite ready for a pull request, but it's functional for EC2 resources and illustrates a relatively simple approach.

https://github.com/iragsdale/terraforming/commit/d82159c7e44f7022f9cd0e8e2280ac71b028dca2

It lets you specify multiple patterns like:

terraforming ec2 --limit instance_type=m3.medium tags=~whatever placement.availability_zone=us-east-1b

I ran into some trouble with the existing specs, this blows up around 25 of them for some reason and I'm a bit too tired to figure out why at the moment. (Plus my ruby is a bit rusty).

Anyway, if the approach makes sense I don't think it should be too difficult to implement the basic matching for the remaining resource types.

It's also a bit confusing because we're matching on the objects being returned by the SDK but printing out the values expected by Terraform, so the output shows availability_zone but the matcher requires placement.availability_zone. There are some other minor matching issues like that where some cleverness could make it way easier to use (tags are one spot).

Anyway, if you could give this a once over and maybe point me in the right direction on the specs, I could maybe clean it up enough to submit a pull request.

[dtan4]

Sorry for the late reply ... :bow: :bow: :bow: It seems good creating Matcher class :smile:

I understood your problem. placement.availability_zone= is not a good option, because user should know about AWS API.

I think maybe constructing the struct which has the same fields as Terraform resource before executing filtering is very helpful... e.g. EC2) SDK returns Aws::EC2::Types::Instance instance, then convert it to the struct like Terraform's instance resource. After that Terraforming executes filtering.

Because this approach may be so heavy, however, how about implement a simple filter to select by resource name?

[chriskilding]

We are using Terraforming on our project as well, and have to reverse engineer a giant pile of EC2 resources. Filtering, especially by things like instance tag, would be incredibly helpful. What is the status of this work in progress? Can it be turned into a pull request within a reasonably speedy timeframe? ;)

[bassrock]

I think filtering by tag would be really helpful. Currently I am trying to transform multiple CloudFormation stacks into terraform environments.

[kbroughton]

+1

[iragsdale]

This was an incredibly naive implementation but it worked for my purposes. Our internal priorities changed so I haven't had much time to try to turn it into a suitable PR, but hopefully it'll be helpful for someone: https://github.com/dtan4/terraforming/compare/master...iragsdale:master

[kbroughton]

Thanks, I pulled your branch master.

I was able to get filtering with terraforming ec2 --limit placement.availability_zone=us-west-2a

The options suggest you can filter by tag, but I'm not sure how to Options: [--merge=MERGE] # tfstate file to merge [--tfstate], [--no-tfstate] # Generate tfstate [--limit=image_id=ami-xxxx placement.availability_zone=us-east-1b tags=~name] # List matching resources. Arguments using =~ match using regular expressions.

I would expect --limit tags=<tag_key>=<tag_value> or --limit tags="<tag_key>,tag_value" so that you can specify the filtering to only match when supplied key,value is a match.

Could you supply the correct syntax?

[iragsdale]

@kbroughton I'm sorry, I don't think I'm gonna be able to provide much in the way of support right now. The syntax sort of depends on the structure of the objects being matched, and I'm a bit too busy at work to get set up to figure that out right now. :(

[soapy1]

I ran into a similar problem at work, where I was trying to use terraforming to describe only a small subset of aws resources. I approached the problem by using leveraging the aws sdk opposed to matching. I was wondering what people thought of this approach and if it would be useful to anybody if I submitted a pr with this feature?

[kbroughton]

That's a great approach. Is there a way to factor it out so you don't have to touch every single file? It seems the current architecture makes it difficult to get the hook into the klass on execute.

I would also be fine for now with just being able to filter after, but i'm so new to ruby, i don't know how to access the dict items from output like resource "aws_instance" "control" { blah: blah tags: { dict i want access to } }

Could you suggest a code snippet to get at tags for me?

[kbroughton]

@soapy1 if you have a beta version of your PR i am working on this issue for the next day or two.

[soapy1]

@kbroughton I pushed up my implementation to https://github.com/dtan4/terraforming/compare/master...soapy1:pre-filtering-resources

You can use it like this:

bin/terraforming ec2 --filters '[{"name":"instance-id", "values":["i-0bdd70830cd1a3ac0"]}]'

so far, it's only able to filter ec2 instances.

I haven't written any tests for it yet. If people like the implementation then I will flush it out a bit more, add tests and make a pr.

[kbroughton]

Thanks a lot soapy! I'm using it now. Nice implementation. I've written some automation to add the snippet equivalents for each resource that is taggable.

          attributes.merge!(tags_attributes_of(instance))

and

      def tags_attributes_of(instance)
        tags = instance.tags
        attributes = { "tags.#" => tags.length.to_s }
        tags.each { |tag| attributes["tags.#{tag.key}"] = tag.value }
        attributes
      end

Now i'm working on spreading the filtering you did for ec2.rb across all resources.

[kbroughton]

Hi soapy1,

I got a little further extending your work to other modules and adding better tag support. https://github.com/dtan4/terraforming/pull/215

[mhemken-nyt]

@kbroughton any chance we can merge this? I would really benefit from the --filter flag.

[kbroughton]

I'm afraid I haven't touched this for years, and won't have time to work on it this month. Happy to have someone fork it and re-submit.

[skambiente]

@kbroughton I pushed up my implementation to master...soapy1:pre-filtering-resources

You can use it like this:

bin/terraforming ec2 --filters '[{"name":"instance-id", "values":["i-0bdd70830cd1a3ac0"]}]'

so far, it's only able to filter ec2 instances.

I haven't written any tests for it yet. If people like the implementation then I will flush it out a bit more, add tests and make a pr.

Hi Soapy1, Not sure if the filtering was implemented but I really need this feature as I only want to do terraforming with single vpc (prod). pls let me know the resolution on this, thanks a lot.

[soapy1]

@skambiente, I'm in a similar boat at kbroughton here. I haven't touched this in a long time, and I'm not involved in the project, so I don't know. It seems like the PR that add this feature is still open #215. You might be able to use a check out of that branch to use the feature.