Split fastly_service_v1 to smaller resources to enable advanced scenarios in terraform and to improve the design

[JiriKovar]

Terraform Version

1.0.11

Affected Resource(s)

fastly_service_v1

Issue

We are creating lots of short lived Fastly services that serve various sets of temporary QA environment (a.k.a. backends). Now we would like to avoid creating and deleting so many Fastly services (according to our findings, we are creating 500+ services/day). Secondarily, we would like to lower the number of requests being run on Fastly APIs (currently we are running about 3000+ requests/hour in peaks, standard API rate limitation is 1000req/h). Ideally, we would love to have an option to integrate all of the environments under a small numbed of services versioned with the changes in our terraform repository. The problem is that currently, the whole Fastly service is implemented as a single terraform resource, which does not support this workflow.

Current Configuration

Here is an extremely simplified configuration that shows what we currently do (please notice the variables used):

resource "fastly_service_v1" "ui_service" {
  name = var.name

  domain {
    name = var.domain
  }

  backend {
    name                  = "ui_hosting"
    address               = var.backend_address
    port                  = 443
    use_ssl               = true
    ssl_cert_hostname     = var.backend_cert_hostname
    shield                = var.backend_shield
    auto_loadbalance      = false
    between_bytes_timeout = 10000
    connect_timeout       = 1000
    first_byte_timeout    = 15000
  }
  
  condition {
    name      = ".well-known/security.txt"
    priority  = 1
    statement = "req.http.host != \"${var.domain}\" && req.url.path == \"/.well-known/security.txt\""
    type      = "REQUEST"
  }

  response_object {
    name              = ".well-known/security.txt"
    content           = "Contact: mailto:[email protected]\nEncryption: https://${var.domain}/pgp-key.txt"
    content_type      = "text/plain"
    request_condition = ".well-known/security.txt"
    response          = "OK"
    status            = 200
  }

  force_destroy = true
}

Current Behavior

There is no way to integrate multiple environments under one Fastly service via this terraform provider - we cant import it and "add" second domain, backend, condition and response_object to an preexisting Fastly service. The best we could do to avoid creating and deleting the services is not to delete them, but leave them be, import them in version 1 into the next environment and terraform everything inside them anew. However that does not solve the issue with API calls (which, according to the Fastly is the smaller issue, but its still far from optimal solution the terraform enables us to use).

Proposed Behavior

I would like to use azurerm logic_app_workflow as an example. The solution there is to have the service created as a separate resource (and as a data resource as well) and each action as a separate resource as well. By breaking the configuration into these chunks, we would be able to use the following workflow:

• have a data resource for the Fastly service that would detect whether there is a compatible service to add to or not (a naming convention on our end would suffice for the versioning purposes). Please beware that data resource does not allow 404 responses (which is widely worked around by querying for the list of resources that could be empty, i.e. azurerm_resources)
• if the data resource detects that there is no compatible service, we can create it through service resource
• if the data resource detects that there is a compatible service, we can use its ID for the purpose of adding domains, backends, conditions and response_objects to an existing service, otherwise, we would use ID of the newly created service from the 2nd step
• once the environment is no longer needed, it can be destroyed together with its parts of the service
• the only issue is that the environment that created the service must not destroy it (that we can either try to solve by removing it from the state, or it would require a flag similar to force_destroy but with an opposite meaning = prevent deactivation)
• we can cleanup services no longer used later on (we already do that for some of the environments)

Proposed Configuration

locals {
  create_new = length(data.fastly_services.existing.resources) == 0 ? true : false
  service_id = local.create_new ? one(fastly_service_v1.ui_service[*].id) : one(data.azurerm_resources.logic_app_resource.resources[*].id)
}

data "fastly_services_v2" "existing" { // query for services by name/regex, returns list of potential finds
  name = var.name 
}

resource "fastly_service_v2" "ui_service" { // creates the service if necessary
  count = local.create_new ? 1 : 0 
  name = var.name
  force_active = true // ensures that the service is not deactivated, or destroyed when the resource is being destroyed
}

resource "fastly_domain_v2" {
  service_id = local.service_id // link to the service
  name = var.domain
}

resource "fastly_backend_v2" {
  service_id = local.service_id
  name                  = "${var.domain}_ui_hosting"
  address               = var.backend_address
  port                  = 443
  use_ssl               = true
  ssl_cert_hostname     = var.backend_cert_hostname
  shield                = var.backend_shield
  auto_loadbalance      = false
  between_bytes_timeout = 10000
  connect_timeout       = 1000
  first_byte_timeout    = 15000
}
  
resource "fastly_condition_v2" {
  service_id = local.service_id
  name      = "${var.domain}_.well-known/security.txt"
  priority  = 1
  statement = "req.http.host != \"${var.domain}\" && req.url.path == \"/.well-known/security.txt\""
  type      = "REQUEST"
}

resource "fastly_response_object_v2" {
  service_id = local.service_id
  name              = "${var.domain}_.well-known/security.txt"
  content           = "Contact: mailto:[email protected]\nEncryption: https://${var.domain}/pgp-key.txt"
  content_type      = "text/plain"
  request_condition = ".well-known/security.txt"
  response          = "OK"
  status            = 200
}

Additional Considerations

• It is possible that with this approach, we might run into limits of the size of the service (i.e. the number of domains), but afaik these limits can be increased and it should be less of an issue compared to the number of services created and the API requests
• It might sound like you are solving an Fastly issue since I came up with this proposal to work around the Fastly limitations, but I believe that having an option to utilize a service provider (Fastly) properly according to its limitations is the first responsibility of every terraform provider. Also please know that I'm discussing this issue with Fastly and they are investigating their options.
• Making this change would move the design of this terraform provider several levels higher - having everything implemented as a single resource is not a great design and it brings many limitations that at the end of the day hurts either Fastly, or the customer. Bottom line is that this change is desirable no matter the result from Fastly, or how we change our way of working.

[JiriKovar]

Oh, and I forgot to mention that this will also solve 2 issues that were bothering us:

• the issue with recreating all the parts of the service whenever order changes. I.e.: if I add / remove a condition and it affects the internal order of the conditions, everything after the changed condition will be recreated.
• the dependencies between conditions and parts relying on the condition

The first is something that has been bothering us for quite a while, because a small change in the terraform definition could create some extremely long terraform plan results (thousands of lines), which are hard to check to make sure that it includes the intended changes.

The second is an issue because if you make a mistake (i.e. typo in the name of the condition, condition missing), terraform can't detect that and it will fail when its being applied to the Fastly service (which is way too late)

The proposed solution would obviously solve this, because there is no ordering - every part of the service is a separate terraform resource and thus planned & applied separately and they would have explicit dependencies enabling the terraform to detect the aforementioned typo in the names / missing parts.

[Integralist]

Hi @JiriKovar

Thanks for opening this issue.

Looking back over your problem statement it sounds like you require the Fastly Terraform Provider to support managing a pool of services, where for each service you are able to rotate the backends/domains those services manage and the Terraform configuration is otherwise the same.

Would that be correct?

I wonder if a combination of the following concepts might help reach this goal:

• Configurability with dynamic blocks
• Multiple environments using directories

I would tend to agree that the current Terraform resource structure isn't ideal from a consumer perspective, but it's coupled to how Fastly's API and Services are currently modelled, due to the CRUD lifecycle of objects always being attached to a service version and isn't something that can be manipulated in isolation. This is something we've experimented at length with ourselves previously.

That said we have long-term plans to make more objects "customer level" instead of "service level". Which will facilitate the separation of resources.

Hope this information helps.

[JiriKovar]

Hi @Integralist

No, that is not my proposal (however I see you are well informed :) ) - a pool of recyclable services is something we are discussing with Fastly as a workaround as it is the only option we currently have (we have a plan for that and currently I'm working on a PoC).

However, this is going to be a hack I would love to avoid in the long run - simply said, terraform and the current Fastly terraform provider were not built around recycling services from some previous runs (we are going to bend the terraform workflow through imports from the previous runs, creating an external dependency on our custom code, making the terraform harder to run from local, presenting a race conditions that needs to be handled on our side and complicating the overall management of the Fastly services and the tooling around them).

My proposal above is wrapped around versioning of the terraform definition and sharing the compatible services for multiple environments without the aforementioned hack (no pool, no custom code, but longer lived services shared between all the environments of the compatible version. For example 1 service can serve a persistent QA environment + 3 UI test environments + 2 Integration tests environments + 5 custom dev environments). However both approaches of sharing the versioned services and a pool of the reusable services, are valid options that can utilize the changes I suggest.

I understand that the Fastly service workflow and versioning is the issue there - my simplified take on the problem would be that, you need to:

• pick a version to edit / create a new version (which is tricky, because you want to support various workflows)
• make changes in it
• activate the version (the question is when and if)

Well, my proposal to that would be another separation - you are trying to "automagically" handle the versioning, which is not easy and might lead to an unexpected behavior (we've experienced that as well). I think it would be great to give the customers a way to handle the versioning by having additional resources of fastly_service_version_v2 through which it would be possible to configure the source version and the triggers for the purpose of creating a new version (similar to the null_resource triggers ). And a fastly_service_activation_v2 resource that could be explicitly used to activate/deactivate the version instead of having switches for that (and with the definition of the dependencies so that it is activated at the correct time). And behind that, a default behavior of modifying a latest version if its not active / creating one from the latest active version and not activating as part of the terraform apply.

Please let me stress that no matter which of the two aforementioned approaches we choose (sharing or pool), this will give us options to remove the hack we are about to make.

Thank you very much for looking into this issue!

[JiriKovar]

Oh and for the record, we are using both dynamic blocks and workspaces + directories. For example:

• dynamic blocks are used for some blocks of the services handling routing to the backend in the correct datacenter (as the number of the datacenters vary between types of environments)
• directories are used for types of the environments (QA / Production / UI Tests / Integration Tests / On Demand environment used by our DEVs, etc.)
• workspaces are used for the "clones" of the same environment type (i.e. currently we have 42 running On Demand environments)

But I don't see how we can use these to have a "clean" implementation of recycling a services from the previous runs. For that, we would need at least some kind of data resource that would enable us to search for a service that can be recycled, but even then I can't imagine how we would handle the race conditions between multiple terraform runs at the same time fighting over the same services...

[Integralist]

Just circling back to this issue, I believe the release of https://github.com/fastly/terraform-provider-fastly/releases/tag/v2.3.0 and the introduction of the reuse attribute would help your testing workflows mention in this issue.

Is there anything outstanding that is preventing us from closing off this issue? My feeling is we should close this issue and, if necessary, re-open smaller, more focused issues as that might be easier to digest and make progress on (with the understanding that it's unlikely we're going to be able to change the nested resource structure due to coupling with the current Fastly service model).

Thanks.

[Integralist]

Going to close off this issue for now. As mentioned, due to Fastly's data model it's unlikely we're going to be able to resolve the issue of separating out the nested blocks into their own resources.