[firestore-bigquery-export] fs-bq-schema-views fails when schema type for an existing view is changed from stringfied_map to map

[johncarter-phntm]

[REQUIRED] Step 2: Describe your configuration

• Extension name: firestore-bigquery-export
• Extension version: 0.1.13
• @firebaseextensions/fs-bq-schema-views @ 0.4.0

[REQUIRED] Step 3: Describe the problem

I'm hitting an error with firebaseextensions/fs-bq-schema-views when trying update my schema when a field is changing type from stringified_map to map.

The error is "Field friends is missing in new schema" when we change the type of "friends".

A workaround is to delete schema views when changing field types - since both schemas work as expected when creating a new view - it's only when updating from one to the other that I hit the error.

Steps to reproduce:

1. Deploy this initial view schema.json using npx @firebaseextensions/fs-bq-schema-views

{
  "fields": [
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "last_login",
      "type": "timestamp"
    },
    {
      "name": "friends",
      "type": "stringified_map"
    }
  ]
}

2. Update the schema.json as follows (based on test example), and redeploy using npx @firebaseextensions/fs-bq-schema-views

{
  "fields": [
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "last_login",
      "type": "timestamp"
    },
    {
      "fields": [
        {
          "name": "name",
          "type": "string",
          "description": "This is a nested name!"
        }
      ],
      "name": "friends",
      "type": "map",
      "description": "Maps don't need a description because a column is not created for a map. Instead, their fields are added as columns to the resulting dataset."
    }
  ]
}

Expected result

Schema view to be regenerated with the friends_name map field.

Actual result

npx @firebaseextensions/fs-bq-schema-views fails with this error:

{"code":400,"errors":[{"message":"Provided Schema does not match Table myproject:firestore_export.example_schema_example_changelog. Field friends is missing in new schema","domain":"global","reason":"invalid"}],"response":{"headers":{"alt-svc":"h3-29=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"","cache-control":"private","content-encoding":"gzip","content-type":"application/json; charset=UTF-8","date":"Thu, 22 Apr 2021 02:48:41 GMT","etag":"W3/SwUI2Vd8QsTFE+6vOFw==","server":"ESF","transfer-encoding":"chunked","vary":"Origin, X-Origin, Referer","x-content-type-options":"nosniff","x-frame-options":"SAMEORIGIN","x-xss-protection":"0"}},"message":"Provided Schema does not match Table myproject:firestore_export.example_schema_example_changelog. Field friends is missing in new schema"}
Provided Schema does not match Table myproject:firestore_export.example_schema_example_changelog. Field friends is missing in new schema

[dackers86]

Hi @johncarter-phntm

As you have correctly pointed out, the easiest way to fix this is to manually delete the tables and to re-run with the new schema.

The following document highlights other solutions including adding an additional field to the schema here https://cloud.google.com/bigquery/docs/managing-table-schemas#manually_delete_a_column.

I've investigated a code based solution that would automatically recreate the tables, however this is tricky as it is difficult to say specifically what has change in the schema. Any changes in terms of order or naming could theoretically be a reason to delete the view and unfortunately this would incur charges and therefore be potentially problematic for other users.

I'll leave this up for discussion and further investigation for if a better solution can be found.

[johncarter-phntm]

Thanks for the update. One option would be to prompt the user in this scenario (similar to how Django prompts when it hits ambiguities in makemigration?).

eg "It looks like the 'friends' field was changed from stringfied_map to map - recreate view myproject:firestore_export.example_schema_example_changelog y/n?")