🐛 [firestore-bigquery-export] Custom event handler for onSuccess not called

[myd-mannheim]

[READ] Step 1: Are you in the right place?

Issues filed here should be about bugs for a specific extension in this repository. If you have a general question, need help debugging, or fall into some other category use one of these other channels:

• For general technical questions, post a question on StackOverflow with the firebase tag.
• For general Firebase discussion, use the firebase-talk google group.
• To file a bug against the Firebase Extensions platform, or for an issue affecting multiple extensions, please reach out to Firebase support directly.

[REQUIRED] Step 2: Describe your configuration

• Extension name: [firestore-bigquery-export]
• Extension version: firebase/[email protected]
• Configuration values (redact info where appropriate): ALLOWED_EVENT_TYPES=firebase.extensions.firestore-counter.v1.onSuccess BIGQUERY_PROJECT_ID=${param:PROJECT_ID} COLLECTION_PATH=recipes DATASET_ID=recipes_dataset_export_raw DATASET_LOCATION=europe-west3 EVENTARC_CHANNEL=projects/my-xxx/locations/us-central1/channels/firebase EXCLUDE_OLD_DATA=no LOG_FAILED_EXPORTS=no MAX_DISPATCHES_PER_SECOND=100 TABLE_ID=recipes_table_export_raw TABLE_PARTITIONING=NONE TIME_PARTITIONING_FIELD_TYPE=omit USE_COLLECTION_GROUP_QUERY=no USE_NEW_SNAPSHOT_QUERY_SYNTAX=yes WILDCARD_IDS=false

[REQUIRED] Step 3: Describe the problem

I've setup firestore-bigquery-export to raise the event firebase.extensions.firestore-counter.v1.onSuccess. Whats quite odd is that in the 'Extension configuration' view of the extension instance the description of this event is 'Occurs when image resizing completes successfully. The event will contain further details about specific formats and sizes.'. Seems to be a copy/paste from the other extension ;) I implemented and deployed a custom event handler as firebase v2 function and deployed it in the same firebase project (region of the function us-central1). The actual sync of the firebase data into BigQuery works fine. However, my event handler for onSuccess is never executed and I also cannot confirm that the event is even raised (because I can't find any logs about it). In GC console unter Eventarc I see that the channel and trigger have been created with the correct with the correct destination. Trigger and channel are both in region us-central1.

Steps to reproduce:

Setup a Firebase project with a firestore collection in europe-west3 and install the firestore-bigquery-export extension with mentioned config. Deploy the following firebase function as event handler.

import eventarc = require('firebase-functions/v2/eventarc') import logger = require('firebase-functions/logger')

exports.onBigQuerySyncSuccess = eventarc.onCustomEventPublished( { eventType: 'firebase.extensions.firestore-counter.v1.onSuccess', channel: projects/${firebaseFunctionsParams.projectID}/locations/us-central1/channels/firebase, region: 'us-central1', }, (event) => { logger.info('handled firebase.extensions.firestore-counter.v1.onSuccess; received data', event) return Promise.resolve() })

Expected result

When changing a document in the collection it is syned into BigQuery and afterward the event handler onBigQuerySyncSuccess is executed.

Actual result

Sync into BigQuery works, event handler is not executed. Somehow I have the impression that the event isn't even raised.

In the GC logs there is a log entry for ext-firestore-bigquery-export-fsexportbigquery with description "Initializing extension with configuration". The "jsonPayload" does not include eventarc channel like in the extensions config data: jsonPayload: { bqProjectId: "my-xxx" clustering: null collectionPath: "recipes" databaseId: "(default)" datasetId: "recipes_dataset_export_raw" datasetLocation: "europe-west3" doBackfill: false docsPerBackfill: 200 excludeOldData: false initialized: false instanceId: "firestore-bigquery-export" location: "us-central1" logFailedExportData: false maxDispatchesPerSecond: 100 message: "Initializing extension with configuration" tableId: "recipes_table_export_raw" timePartitioning: null useCollectionGroupQuery: false useNewSnapshotQuerySyntax: true wildcardIds: false }

Thanks, Max

[myd-mannheim]

Another weird observation is that in Firebase console in the config manage screen of the extension the event are prefixed with "firebase.extensions.firestore-counter.v1" However, in the source code of the big query export extension the event type have prefix "firebase.extensions.firestore-bigquery-export.v1" (see https://github.com/firebase/extensions/blob/ecc213e1b384d4a15be629f163d22c431c85d5b2/firestore-bigquery-export/functions/src/events.ts#L8 ) In the extension.yaml (https://github.com/firebase/extensions/blob/ecc213e1b384d4a15be629f163d22c431c85d5b2/firestore-bigquery-export/extension.yaml) there is one event listed where the prefix sounds more like it actually belongs to this extension: firebase.extensions.big-query-export.v1.sync.start but it is still different from whats used according to the source code. It seems the config and whats actually happening is out of sync.

Still, even when I deploy an event handler for firebase.extensions.firestore-bigquery-export.v1.onSuccess it is not called. However, when I raise the same event on the same channel from my own firebase function then my event handler is called. I raise the event like this: adminEventarc.getEventarc().channel( 'projects/my-xxx/locations/us-central1/channels/firebase', { allowedEventTypes: 'firebase.extensions.firestore-bigquery-export.v1.onSuccess', } ).publish({ type: 'firebase.extensions.firestore-bigquery-export.v1.onSuccess', subject: 'test', data: { message: 'test-data', }, })

[myd-mannheim]

Also here https://github.com/firebase/extensions/blob/ecc213e1b384d4a15be629f163d22c431c85d5b2/firestore-bigquery-export/functions/src/events.ts#L17 I have the impression it should not be allowedEventTypes: process.env.EXT_SELECTED_EVENTS but allowedEventTypes: process.env.ALLOWED_EVENT_TYPES because ALLOWED_EVENT_TYPES is the name of the variable in the config env for the extension.

[shubhamkislay]

I'm facing the same issue. Did you get it to work?

[junaidxabd]

Having the same issue here

[cabljac]

Hi all,

Thanks for reporting this issue! We’ve received it and are reviewing it. We’ll provide updates as soon as possible.

[beenerdy]

Same issue here

[beenerdy]

Summary of events.recordSuccessEvent behavior in Firestore BigQuery Export:

• recordSuccessEvent is only called in syncBigQuery when a failed direct write to BigQuery is successfully processed via the task queue.
• For successful direct writes in fsexportbigquery, no success event is published to Eventarc, only logs.complete() is called.
• This means Eventarc listeners on the firebase channel will only see success events (e.g., firebase.extensions.big-query-export.v1.sync.success) for retried operations, not for immediate successful writes.

[myd-mannheim]

@beenerdy Thanks for the explanation and the fix, but that also leaves me a bit puzzled. Having a success event only in this one specific situation seems weird - it's also not obvious from the docs. How can I sync back the result of queries from BQ, after an update from firebase occured?

@shubhamkislay Maybe a bit late but...: My workaround was to raise the event myself in a document change event handler. The drawback was that sometimes I would sync back outdated data if the BG-sync was not completed when the event was raised.