Charlotte AWS deployment

[dkatzz]

Description

Charlotte has some custom resources in their AWS environment, which we enabled support for in https://github.com/civiform/civiform/issues/7378. This issue tracks working with them to deploy after that change and track any additional requirements.

https://github.com/civiform/civiform/issues/7852 is related as well

Describe the solution you'd like.

Charlotte AWS is deployed alongside their custom resources

What priority is this?

High, since this blocks the launch for Charlotte

Associated civic entity

Charlotte

[dkatzz]

We met with Charlotte on Monday to go through the new deployment, but realized they were still on version 1.58.0. They also didn't have their security team at the meeting, who could answer questions about the new VPC. Instead of working on the new deployment, we spent the time upgrading their existing deployment. We have a thread now to ask questions about the VPC and will plan to have a follow up meeting to deploy within their system using the updates from https://github.com/civiform/cloud-deploy-infra/pull/330, and https://github.com/civiform/cloud-deploy-infra/pull/335

[dkatzz]

I have been testing with this branch. I was getting pretty close then got this error:

│ Error: creating ELBv2 application Load Balancer (charlotte-vpc2-civiform-lb): InvalidSubnet: VPC vpc-0f18a363a62cc1675 has no internet gateway
│    status code: 400, request id: 7c02e087-3e0d-4174-9b79-161db5642108
│ 
│  with module.ecs_fargate_service.aws_lb.civiform_lb,
│  on ../../modules/ecs_fargate_service/main.tf line 26, in resource "aws_lb" "civiform_lb":
│  26: resource "aws_lb" "civiform_lb" {

I then made this change to make the lb internal and then it made it to the end of the deployment but just failed the health check. I then tried to disable the health check then I got the error:

module.ecs_fargate_service.aws_lb_target_group.lb_https_tgs: Modifying... [id=arn:aws:elasticloadbalancing:us-east-2:381492017277:targetgroup/charlotte-vpc3-https-9000/42ed693af2892f21]
╷
│ Error: modifying Target Group: InvalidConfigurationRequest: Health check enabled must be true for target groups with target type 'ip'
│    status code: 400, request id: 2de1cd13-7f34-4968-81af-9163014d5477
│ 
│  with module.ecs_fargate_service.aws_lb_target_group.lb_https_tgs,
│  on ../../modules/ecs_fargate_service/main.tf line 114, in re

[dkatzz]

Got error after update (https://github.com/civiform/cloud-deploy-infra/commit/c6a151a40b897a56f64ab270a38b82f30af053c5):

╷
│ Error: creating ELBv2 Listener (arn:aws:elasticloadbalancing:us-east-2:381492017277:loadbalancer/net/charlotte-vpc4-civiform-lb/743e0c4d75301971): ValidationError: Listener protocol 'HTTP' must be one of 'TCP_UDP, TCP, UDP, TLS'
│       status code: 400, request id: 894d923a-82b8-4310-9a6f-666c4ebfa838
│ 
│   with module.ecs_fargate_service.aws_lb_listener.lb_http_listeners,
│   on ../../modules/ecs_fargate_service/main.tf line 163, in resource "aws_lb_listener" "lb_http_listeners":
│  163: resource "aws_lb_listener" "lb_http_listeners" {
│ 
╵
╷
│ Error: creating ELBv2 Listener (arn:aws:elasticloadbalancing:us-east-2:381492017277:loadbalancer/net/charlotte-vpc4-civiform-lb/743e0c4d75301971): ValidationError: Listener protocol 'HTTPS' must be one of 'TCP_UDP, TCP, UDP, TLS'
│       status code: 400, request id: 4725a613-a084-4517-84e1-0c9ca26ff471
│ 
│   with module.ecs_fargate_service.aws_lb_listener.lb_https_listeners,
│   on ../../modules/ecs_fargate_service/main.tf line 188, in resource "aws_lb_listener" "lb_https_listeners":
│  188: resource "aws_lb_listener" "lb_https_listeners" {
│ 
╵
╷
│ Error: creating ECS Service (charlotte-vpc4-civiform-service): InvalidParameterException: The target group with targetGroupArn arn:aws:elasticloadbalancing:us-east-2:381492017277:targetgroup/charlotte-vpc4-https-9000/4eab270cd26fb3da does not have an associated load balancer.
│ 
│   with module.ecs_fargate_service.aws_ecs_service.service,
│   on ../../modules/ecs_fargate_service/main.tf line 212, in resource "aws_ecs_service" "service":
│  212: resource "aws_ecs_service" "service" {
│ 
╵

[dkatzz]

https://github.com/civiform/cloud-deploy-infra/compare/main...clt-test2?expand=1 this version deploys, but the task shows: ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secret from asm: service call has been retried 5 time(s): failed to fetch secret arn:aws:secretsmanager:us-east-2:381492017277:secret:charlotte-vpc8-civiform_adfs_client_id-9WJNgJ from secrets manager: RequestCanceled: request context canceled caused by: context deadline exceeded. Please check your task network configuration.

[dkatzz]

https://github.com/civiform/cloud-deploy-infra/compare/main...clt-test2?expand=1 this version deploys, but the task shows: ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secret from asm: service call has been retried 5 time(s): failed to fetch secret arn:aws:secretsmanager:us-east-2:381492017277:secret:charlotte-vpc8-civiform_adfs_client_id-9WJNgJ from secrets manager: RequestCanceled: request context canceled caused by: context deadline exceeded. Please check your task network configuration.

^ that issue was because the security group didn't have the inbound allowed on port 433

[dkatzz]

@wukimmy made updates to fix issues with fetching secrets and also with issues reaching dockerhub:

Task is stopping
CannotPullContainerError: pull image manifest has been retried 5 time(s): failed to resolve ref [docker.io/civiform/aws-metrics-scraper:latest](http://docker.io/civiform/aws-metrics-scraper:latest): failed to do request: Head "https://registry-1.docker.io/v2/civiform/aws-metrics-scraper/manifests/latest": dial tcp [34.226.69.105:443](http://34.226.69.105:443/): i/o timeout

[dkatzz]

Now there are issues with initializing ebean:

Caused by: java.lang.NoClassDefFoundError: Could not initialize class io.ebean.DB at repository.SettingsGroupRepository.<init>(SettingsGroupRepository.java:23) at repository.SettingsGroupRepository$$FastClassByGuice$$ccd7bc.GUICE$TRAMPOLINE(<generated>) at repository.SettingsGroupRepository$$FastClassByGuice$$ccd7bc.apply(<generated>) at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300) at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:40) at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:60) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300) at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1148) at modules.SettingsMigrationModule$SettingsMigrator.lambda$new$0(SettingsMigrationModule.java:37) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48) at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source) at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source) at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)

[wukimmy]

Environment updates I've made:

1. Created a NAT Gateway on the Security VPC for the docker.io access
2. Created a VPC Gateway for Secret Manager access

[dkatzz]

Tried updating HTTPS -> HTTP

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place
+/- create replacement and then destroy

Terraform will perform the following actions:

  # module.ecs_fargate_service.aws_ecs_service.service will be updated in-place
  ~ resource "aws_ecs_service" "service" {
        id                                 = "arn:aws:ecs:us-east-2:381492017277:service/charlotte-vpc8-civiform/charlotte-vpc8-civiform-service"
        name                               = "charlotte-vpc8-civiform-service"
        tags                               = {
            "Name" = "charlotte-vpc8-civiform-ecs-tasks-sg"
            "Type" = "Civiform Fargate Service"
        }
        # (17 unchanged attributes hidden)

      - load_balancer {
          - container_name   = "charlotte-vpc8-civiform" -> null
          - container_port   = 9000 -> null
          - target_group_arn = "arn:aws:elasticloadbalancing:us-east-2:381492017277:targetgroup/charlotte-vpc8-https-9000/4183cc1e34fe7388" -> null
            # (1 unchanged attribute hidden)
        }
      + load_balancer {
          + container_name   = "charlotte-vpc8-civiform"
          + container_port   = 9000
          + target_group_arn = (known after apply)
            # (1 unchanged attribute hidden)
        }

        # (3 unchanged blocks hidden)
    }

  # module.ecs_fargate_service.aws_lb_listener.lb_https_listeners will be updated in-place
  ~ resource "aws_lb_listener" "lb_https_listeners" {
        id                = "arn:aws:elasticloadbalancing:us-east-2:381492017277:listener/app/charlotte-vpc8-civiform-lb/b14031ebc73c49c4/5d68bbfdb4a381e3"
        tags              = {
            "Name" = "charlotte-vpc8 Civiform Fargate Service"
            "Type" = "Civiform Fargate Service"
        }
        # (7 unchanged attributes hidden)

      ~ default_action {
          ~ target_group_arn = "arn:aws:elasticloadbalancing:us-east-2:381492017277:targetgroup/charlotte-vpc8-https-9000/4183cc1e34fe7388" -> (known after apply)
            # (2 unchanged attributes hidden)
        }
    }

  # module.ecs_fargate_service.aws_lb_target_group.lb_https_tgs must be replaced
+/- resource "aws_lb_target_group" "lb_https_tgs" {
      ~ arn                                = "arn:aws:elasticloadbalancing:us-east-2:381492017277:targetgroup/charlotte-vpc8-https-9000/4183cc1e34fe7388" -> (known after apply)
      ~ arn_suffix                         = "targetgroup/charlotte-vpc8-https-9000/4183cc1e34fe7388" -> (known after apply)
      ~ id                                 = "arn:aws:elasticloadbalancing:us-east-2:381492017277:targetgroup/charlotte-vpc8-https-9000/4183cc1e34fe7388" -> (known after apply)
      ~ ip_address_type                    = "ipv4" -> (known after apply)
      ~ load_balancing_cross_zone_enabled  = "use_load_balancer_configuration" -> (known after apply)
        name                               = "charlotte-vpc8-https-9000"
      + preserve_client_ip                 = (known after apply)
      ~ protocol                           = "HTTPS" -> "HTTP" # forces replacement
      ~ protocol_version                   = "HTTP1" -> (known after apply)
        tags                               = {
            "Name" = "charlotte-vpc8-civiform-https-9000"
            "Type" = "Civiform Fargate Service"
        }
        # (10 unchanged attributes hidden)

      ~ health_check {
          ~ protocol            = "HTTPS" -> "HTTP"
            # (8 unchanged attributes hidden)
        }

      - stickiness {
          - cookie_duration = 86400 -> null
          - enabled         = false -> null
          - type            = "lb_cookie" -> null
            # (1 unchanged attribute hidden)
        }

      - target_failover {}
    }

Plan: 1 to add, 2 to change, 1 to destroy.

╷
│ Error: ELBv2 Target Group (charlotte-vpc8-https-9000) already exists
│ 
│   with module.ecs_fargate_service.aws_lb_target_group.lb_https_tgs,
│   on ../../modules/ecs_fargate_service/main.tf line 114, in resource "aws_lb_target_group" "lb_https_tgs":
│  114: resource "aws_lb_target_group" "lb_https_tgs" {
│ 

[wukimmy]

Added a VPCe for the RDS access

[wukimmy]

Created some networking Reachability Analysis in Network Manager for easier troubleshooting. Just click on re-analyse when needed to check if it is a networking problem

[dkatzz]

Still getting error around ebean connection after vpce added - looking into other reasons why this may not be working

Caused by: java.lang.NoClassDefFoundError: Could not initialize class io.ebean.DB at repository.QuestionRepository.<init>(QuestionRepository.java:45) at repository.QuestionRepository$$FastClassByGuice$$ae54e0.GUICE$TRAMPOLINE(<generated>) at repository.QuestionRepository$$FastClassByGuice$$ae54e0.apply(<generated>) at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300) at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:40) at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:60) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300) at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:40) at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:60) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300) at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1148) at modules.DatabaseSeedModule$DatabaseSeedScheduler.lambda$new$0(DatabaseSeedModule.java:36) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)

[wukimmy]

Created a new inbound rule on RDS to accept from ecs security group

[wukimmy]

Service it's running, but it appears to be going to pending after a while. Got the error Illegal request, responding with status '400 Bad Request': Unsupported HTTP method: The HTTP method started with 0x16 rather than any known HTTP method from 10.126.152.169:22072. Perhaps this was an HTTPS request sent to an HTTP endpoint?

[dkatzz]

Trying now with a new app prefix (charlotte-vpc9) and running from the clt-test4 branch, which changes the protocol back from HTTPS to HTTP (https://github.com/civiform/cloud-deploy-infra/commit/ecf3218de8ef49c330338fd6ca40ba3f42e735f6)

[dkatzz]

After the most recent deployment, we aren't seeing errors in the logs and the task is running and RDS has connections - I think we now just have to make a configuration update to get a public facing URL. The normal URL we would use is the ELB URL (http://internal-charlotte-vpc9-civiform-lb-979264705.us-east-2.elb.amazonaws.com./), but that's now internal

[dkatzz]

Seeing unhealthy on the NLB. Tried navigating to NLb-Testing-fixing-IP-205c34e73dd3af17.elb.us-east-2.amazonaws.com and NLb-Testing-fixing-IP-205c34e73dd3af17.elb.us-east-2.amazonaws.com:443 with no luck

[dkatzz]

@wukimmy did a curl in an ec2 instance on the same VPC and got the same results as doing a curl to exygy-civiform-lb-1702473953.us-east-1.elb.amazonaws.com.

Based on this, I'll ensure the changes from the test branch are checked in appropriately and then we can do a test with charlotte

[dkatzz]

Latest issue we got was ResourceInitializationError: unable to pull secrets or registry auth: unable to retrieve secret from asm: There is a connection issue between the task and AWS Secrets Manager - we're waiting on the Charlotte team to update their firewall or create an endpoint between resources.

[dkatzz]

We were getting that secrets error a few times, but most recently we got past that and then got an error: CannotPullContainerError: The task cannot pull [docker.io/civiform/civiform@sha256:d47cebfda1b31c70e78e1e10f959fb8c9aaba1f0ce1fe7a3c72ec9dc2b586f5d](http://docker.io/civiform/civiform@sha256:d47cebfda1b31c70e78e1e10f959fb8c9aaba1f0ce1fe7a3c72ec9dc2b586f5d) from the registry [docker.io/civiform/civiform@sha256:d47cebfda1b31c70e78e1e10f959fb8c9aaba1f0ce1fe7a3c72ec9dc2b586f5d](http://docker.io/civiform/civiform@sha256:d47cebfda1b31c70e78e1e10f959fb8c9aaba1f0ce1fe7a3c72ec9dc2b586f5d). There is a connection issue between the task and the registry. Check your task network configuration. : failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://docker-images-prod.s3.dualstack.us-east-1.amazonaw..

[dkatzz]

Now the error is: ResourceInitializationError: failed to validate logger args: The task cannot find the Amazon CloudWatch log group defined in the task definition. There is a connection issue between the task and Amazon CloudWatch. Check your network configuration. : signal: killed - They're going to add this to their firewall

[dkatzz]

This was the most recent error:

Container
August 08, 2024 at 07:50 (UTC-4:00)
	
2024-08-08T11:50:26.868Z info [email protected]/collector.go:250 Received signal from OS {"signal": "terminated"}
	
clt-staging-metrics-scraper
August 08, 2024 at 07:50 (UTC-4:00)
	
2024-08-08T11:50:26.869Z info [email protected]/service.go:178 Starting shutdown...
	
clt-staging-metrics-scraper
August 08, 2024 at 07:50 (UTC-4:00)
	
2024-08-08T11:50:26.869Z info healthcheck/handler.go:132 Health Check state change {"kind": "extension", "name": "health_check", "status": "unavailable"}
	
clt-staging-metrics-scraper
August 08, 2024 at 07:50 (UTC-4:00)
	
2024-08-08T11:50:26.869Z info extensions/extensions.go:50 Stopping extensions...
	
clt-staging-metrics-scraper
August 08, 2024 at 07:50 (UTC-4:00)
	
2024-08-08T11:50:26.869Z info [email protected]/zpagesextension.go:98 Unregistered zPages span processor on tracer provider {"kind": "extension", "name": "zpages"}
	
clt-staging-metrics-scraper
August 08, 2024 at 07:50 (UTC-4:00)
	
2024-08-08T11:50:26.869Z info [email protected]/service.go:192 Shutdown complete.
	
clt-staging-metrics-scraper
August 08, 2024 at 07:50 (UTC-4:00)
	
Caused by: java.net.SocketTimeoutException: Read timed out at java.base/sun.nio.ch.NioSocketImpl.timedRead(Unknown Source) at java.base/sun.nio.ch.NioSocketImpl.implRead(Unknown Source) at java.base/sun.nio.ch.NioSocketImpl.read(Unknown Source) at java.base/sun.nio.ch.NioSocketImpl$1.read(Unknown Source) at java.base/java.net.Socket$SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.net.[www.protocol.https.HttpsClient.afterConnect](http://www.protocol.https.httpsclient.afterconnect/)(Unknown Source) at java.base/sun.net.[www.protocol.https.AbstractDelegateHttpsURLConnection.connect](http://www.protocol.https.abstractdelegatehttpsurlconnection.connect/)(Unknown Source) at java.base/sun.net.[www.protocol.http.HttpURLConnection.getInputStream0](http://www.protocol.http.httpurlconnection.getinputstream0/)(Unknown Source) at java.base/sun.net.[www.protocol.http.HttpURLConnection.getInputStream](http://www.protocol.http.httpurlconnection.getinputstream/)(Unknown Source) at java.base/sun.net.[www.protocol.https.HttpsURLConnectionImpl.getInputStream](http://www.protocol.https.httpsurlconnectionimpl.getinputstream/)(Unknown Source) at com.nimbusds.jose.util.DefaultResourceRetriever.getInputStream(DefaultResourceRetriever.java:361) at com.nimbusds.jose.util.DefaultResourceRetriever.retrieveResource(DefaultResourceRetriever.java:264) at org.pac4j.oidc.config.OidcConfiguration.internalInit(OidcConfiguration.java:198) ... 41 more
	
clt-staging-civiform
August 08, 2024 at 07:50 (UTC-4:00)
	
Caused by: org.pac4j.core.exception.TechnicalException: java.net.SocketTimeoutException: Read timed out at org.pac4j.oidc.config.OidcConfiguration.internalInit(OidcConfiguration.java:201) at org.pac4j.core.util.InitializableObject.init(InitializableObject.java:56) at org.pac4j.oidc.client.OidcClient.internalInit(OidcClient.java:48) at org.pac4j.core.util.InitializableObject.init(InitializableObject.java:56) at org.pac4j.core.util.InitializableObject.init(InitializableObject.java:33) at auth.oidc.OidcClientProvider.get(OidcClientProvider.java:239) at auth.oidc.OidcClientProvider.get(OidcClientProvider.java:31) at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:86) at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:72) at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:60) at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:59) at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:40) at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:60) at com.google.inject.internal.ProviderMethod.doProvision(ProviderMethod.java:171) at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.provision(InternalProviderInstanceBindingImpl.java:185) at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.get(InternalProviderInstanceBindingImpl.java:162) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:169) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45) at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:40) at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:60) at com.google.inject.internal.ProviderMethod.doProvision(ProviderMethod.java:171) at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.provision(InternalProviderInstanceBindingImpl.java:185) at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.get(InternalProviderInstanceBindingImpl.java:162) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:169) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45) at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:50) at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:146) at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:101) at com.google.inject.internal.Initializer$InjectableReference.get(Initializer.java:256) at com.google.inject.internal.Initializer.injectAll(Initializer.java:153) at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180) ... 9 more

[dkatzz]

Adding auth0 and okta to the firewall rules fixed that issue ^

[dkatzz]

Marking this as complete since the deployment was successful. The charlotte team is still working on networking, but that involves less input from the CiviForm team. I'll create a new issue or re open this if there are any follow ups during the networking conversations.