Allow reuse of HTTP connection

[nickabousselam]

trafficstars

Terraform CLI and Provider Versions

% terraform --version Terraform v1.5.7 on darwin_arm64

• provider registry.terraform.io/go-gitea/gitea v0.3.0
• provider registry.terraform.io/hashicorp/http v3.4.0
• provider registry.terraform.io/hashicorp/random v3.5.1
• provider registry.terraform.io/hashicorp/time v0.9.1

Use Cases or Problem Statement

We are using this HTTP provider to make multiple requests to the same API in a single Terraform sync.

We have found that when these requests are made, a new TCP connection is created for each request instead of the connection re-use/pooling that I would typically expect in Go. This causes a problem for us - we unavoidably have to make several hundred requests to that API and the connections limit on the remote server is exhausted as a result of not having connection reuse.

Proposal

The reason for the problem I believe is that the HTTP client transport is being cloned each time in the provider's Read() method instead of relying directly on the default shared transport http.DefaultTransport (in order to guard against changes to the shared value).

I propose that we move the HTTP client and transport creation to the initialisation of the data source so we can take advantage of connection reuse. Consecutive calls to Read() can use the same instance of the client instead of building its own each time.

Incidentally, the go-retryablehttp client whose transport is replaced by the clone above itself relies on a library that creates its own instance of a http client and transport (go-cleanhttp), so if you were to use the client by itself you would have the same issue.

How much impact is this issue causing?

Medium

Additional Information

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct