feat: BigLake REST Catalog Support: Missing x-goog-user-project Header

[masfworld]

Summary

The iceberg-go REST catalog cannot connect to Google Cloud BigLake because it lacks support for the required x-goog-user-project header in the initial /v1/config request.

Problem Description

BigLake requires the x-goog-user-project header for quota/billing attribution, but the current rest.NewCatalog() implementation fails during the initial configuration call.

Current Behavior

// This fails with BigLake
cat, err := rest.NewCatalog(ctx, "catalog", "https://biglake.googleapis.com/iceberg/v1beta/restcatalog", 
    rest.WithOAuthToken(token),
    rest.WithAdditionalProps(iceberg.Properties{
        "header.x-goog-user-project": "my-project",  // Doesn't work for initial config call
    }),
)
// Error: "biglake.googleapis.com API requires a quota project"

Expected Behavior

The catalog should successfully connect to BigLake when the proper headers are provided.

Root Cause

1. rest.NewCatalog() makes an initial call to /v1/config to fetch catalog configuration
2. This initial call uses the base HTTP client without custom headers from WithAdditionalProps
3. BigLake rejects the request due to missing x-goog-user-project header
4. The header. prefix in additional properties only applies to subsequent calls, not the config call

Evidence

Direct HTTP calls with proper headers work perfectly:

curl -H "Authorization: Bearer $TOKEN" \
     -H "x-goog-user-project: my-project" \
     "https://biglake.googleapis.com/iceberg/v1beta/restcatalog/v1/config?warehouse=gs://my-bucket"
# Returns 200 OK with catalog configuration

Proposed Solutions

Option 1: Add WithHeaders function

func WithHeaders(headers map[string]string) Option {
    // Apply headers to ALL requests including initial config
}

Option 2: Add WithHTTPClient function

func WithHTTPClient(client *http.Client) Option {
    // Allow custom HTTP client injection
}

Option 3: Fix WithAdditionalProps header handling

Ensure header. prefixed properties in WithAdditionalProps are applied to the initial config request.

Impact

This issue prevents iceberg-go from working with:

• Google Cloud BigLake
• Any REST catalog requiring custom headers for authentication/billing
• Enterprise environments with header-based routing/auth

Workaround

Currently requires bypassing rest.NewCatalog() and implementing direct HTTP calls to BigLake REST API endpoints.

Environment

• iceberg-go version: v0.3.0
• Go version: 1.21+
• Platform: Google Cloud BigLake REST Catalog

Related

This is similar to how Apache Spark's Iceberg connector handles BigLake via the spark.sql.catalog.catalog_name.header.x-goog-user-project configuration.

[rambleraptor]

I've created #579, which I'm hoping will (eventually) enable support for a Google AuthManager for handling authentication. Please take a look. I'd be curious if there's support for adding this header to an (eventual) Google AuthManager