terraform-provider-http icon indicating copy to clipboard operation
terraform-provider-http copied to clipboard

Published 20 hours ago • verified publisher icon hashicorp

GET not following 307 redirects

Open louhnmn opened this issue 2 years ago • 1 comments
trafficstars

Terraform CLI and Provider Versions

Terraform v1.4.6 on darwin_amd64

  • provider registry.terraform.io/hashicorp/http v3.3.0

Terraform Configuration

terraform {
  required_version = ">= 1.4.0"
}

variable "gsheet" {
  type = object({
    base_url = string
    pub_id   = string
    gid      = string
  })
  default = {
    base_url = "https://docs.google.com/spreadsheets/d/e/"
    pub_id   = "2PACX-1vRzZjUzim7pcO9Q3xBOCf9_GmMEANBLVeGpPbUOX-GzfQLU6bIgEkOQsXOuuM-M9ephWi3TSSO9vpWH"
    gid      = "0"
  }
}

data "http" "get_sheet" {
  url    = format("%s%s/pub?gid=%s&output=csv", var.gsheet.base_url, var.gsheet.pub_id, var.gsheet.gid)
  method = "GET"

  request_headers = {
    Accept = "text/csv"
  }
}

output "sheet_response_body" {
  value = data.http.get_sheet.response_body
}

Expected Behavior

csv data should be output.

colA colB
123 456
789 12

Actual Behavior

a 307 redirect response html is output.

<HTML>
        <HEAD>
        <TITLE>Temporary Redirect</TITLE>
        </HEAD>
        <BODY BGCOLOR="#FFFFFF" TEXT="#000000">
        <H1>Temporary Redirect</H1>
        The document has moved <A HREF="https://doc-0s-88-sheets.googleusercontent.com/pub/n4743ueaoiib6is67cgugk097k/dd0k4os4p03bpv4iksdjr6h7gg/1683540870000/105210521312657442095/*/e@2PACX-1vRzZjUzim7pcO9Q3xBOCf9_GmMEANBLVeGpPbUOX-GzfQLU6bIgEkOQsXOuuM-M9ephWi3TSSO9vpWH?gid=0&amp;output=csv">here</A>.
        </BODY>
</HTML>

Steps to Reproduce

  1. terraform plan

How much impact is this issue causing?

High

Logs

No response

Additional Information

public link of the google sheet: sheet

With the Location parameter set (-L) the 307 redirect will success.

A curl -Li will output:

HTTP/2 307 
content-type: text/html; charset=UTF-8
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
expires: Mon, 01 Jan 1990 00:00:00 GMT
date: Thu, 04 May 2023 10:33:15 GMT
location: https://doc-0s-88-sheets.googleusercontent.com/pub/n4743ueaoiib6is67cgugk097k/8ae3sg2p0gv6016ai6lm64n7p4/1683196395000/105210521312657442095/*/e@2PACX-1vRzZjUzim7pcO9Q3xBOCf9_GmMEANBLVeGpPbUOX-GzfQLU6bIgEkOQsXOuuM-M9ephWi3TSSO9vpWH?gid=0&single=true&output=csv
p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
server: GSE
set-cookie: NID=511=llanR2tu9k6ac9<...redacted...>_QuFpih9Z5Om4; expires=Fri, 03-Nov-2023 10:33:14 GMT; path=/; domain=.google.com; HttpOnly
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
accept-ranges: none
vary: Accept-Encoding

HTTP/2 200 
content-type: text/csv
x-frame-options: ALLOW-FROM https://docs.google.com
x-robots-tag: noindex, nofollow, nosnippet
expires: Thu, 04 May 2023 10:33:18 GMT
date: Thu, 04 May 2023 10:33:18 GMT
cache-control: private, max-age=300
content-disposition: attachment; filename="test_table-data1.csv"; filename*=UTF-8''test_table%20-%20data1.csv
access-control-allow-origin: *
access-control-expose-headers: Cache-Control,Content-Disposition,Content-Encoding,Content-Length,Content-Type,Date,Expires,Server,Transfer-Encoding
content-security-policy: frame-ancestors 'self' https://docs.google.com
content-security-policy: base-uri 'self';object-src 'self';report-uri https://doc-0s-88-sheets.googleusercontent.com/spreadsheets/cspreport;script-src 'nonce-e1_RvbtZa_0Z_GdQwIpG9g' 'unsafe-inline' 'strict-dynamic' https: http: 'unsafe-eval';worker-src 'self' blob:
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
server: GSE
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
accept-ranges: none
vary: Accept-Encoding

colA,colB
123,456
789,12

Code of Conduct

  • [X] I agree to follow this project's Code of Conduct

louhnmn avatar May 08 '23 10:05 louhnmn

I actually found a workaround last week just by clicking around on the response link. So the 307 redirect response contains the redirect url in the <A HREF="...url_is_here...">...</A> tag. It will be valid for a few seconds (something like 5 seconds). So now we can just extract this url and then put a second http data-source around it and you are good to go.

so this would be the final code assuming the variables are already set up:

# first data source that will lead to the redirect response
data "http" "get_sheet" {
  url    = format("%s%s/pub?gid=%s&output=csv", var.gsheet.base_url, var.gsheet.pub_id, var.gsheet.gid)
  method = "GET"

  request_headers = {
    Accept = "text/csv"
  }
}

# a locals variable that extracts the redirect url from the first response
locals {
  sheet_redirect_url = replace(regex("<A HREF=\"(.+)\">here</A>", data.http.get_sheet_normal_unique.response_body)[0], "&amp;", "&")
}

# output just to see if the url looks good
output "sheet_extracted_redirect_url" {
  value = local.sheet_redirect_url
}

# second data source to get the actual csv (or whatever data)
data "http" "get_sheet_redirected" {
  url    = local.sheet_redirect_url
  method = "GET"

  request_headers = {
    Accept = "text/csv"
  }
}

# output to see the data
output "sheet_response_redirected" {
  value = data.http.get_sheet_redirected.response_body
}

louhnmn avatar Jul 24 '23 07:07 louhnmn