[FR] Improve public api for working with cloud functions triggered by Firestore

[mamichels]

Improve public API for working with cloud functions triggered by Firestore

Lately I've been working on a set of cloud functions which are triggered by Firestore events (see Docs). While it does work out in the first place, we do feel quite a struggle when migrating the first functions to Typescript, which somehow feels off.

If you think that this is a feasible issue and want me to help out please let me know and point me to the favored direction.

Specifically there are 2 main issues we are encountering with the firestore client sdk.

Missing public types for triggered events

Given any triggered function (create, update, delete, write), the function should look like this.

exports.helloFirestore = (event, context) => {
  console.log('Hello Firestore!');
};

Neither of those params (event or context) types is part of this package exported types. It could be argued whether @google-cloud/firestore is the adequate module for such feature, but I can hardly imagine a better one. As @google-cloud/functions-framework is scoped on http and CloudEvent handling I don't see it fit there either.

For the event parameter this is something that might be resolved quite easily, since all required types more or less do exist within the package and are already part of the documentation. I would propose to add an exported type to the main module.

interface FirestoreEvent {
    oldValue: Document,
    updateMask: DocumentMask,
    value: Document
}

// Usage e.g.
import { FirestoreEvent } from '@google-cloud/firestore';

It could be discussed if the context parameter is also in the scope of this package. I am interested in your opinion here.

Arguably unnecessary read operations

Mostly depending on your use case it feels very awkward to work with the data that is already present within the event parameter. The official documentation also shows this:

exports.makeUpperCase = event => {
    const resource = event.value.name;
    const affectedDoc = firestore.doc(resource.split('/documents/')[1]);

    const curValue = event.value.fields.original.stringValue;
    const newValue = curValue.toUpperCase();

    if (curValue !== newValue) {
        console.log(`Replacing value: ${curValue} --> ${newValue}`);

        return affectedDoc.set({
            original: newValue,
        });
    } else {
        // Value is already upper-case
        // Don't perform a(nother) write to avoid infinite loops
        console.log('Value is already upper-case.');
    }
};

While the data is already present in the event parameter, accessing a simple value is very cumbersome. Keep in mind that this object is very shallow. A real documents will be tremendously more complex.

Furthermore, only a DocumentReference is being created for the sole purpose of updating it. If I intend to work with a simple object it would require an additional read operation, since there is no exposed api to the internal used protobuf converter.

In an ideal scenario I would expect this package to have a public accessible method to create a DocumentSnapshot by using the data of the event object.

class Firestore {
...
    convert<T>(document: Document): DocumentSnapshot<T> {...}
    // or directly from the event itself
    convert<T>(firestoreEvent: FirestoreEvent): DocumentSnapshot<T> {...}
...
}

// Usage e.g
const docSnapshot = firestore.convert(event.value);
// or directly from the event itself
const docSnapshot = firestore.convert(event);

// Data Access should be simple now
const doc = docSnapshot.data();

[inlined]

Could I interest you in Firebase's layer on top of Cloud Functions (known as Cloud Functions for Firebase)?

The firebase-functions SDK does three useful things for you:

1. It parses the input into friendlier types. This can include non-JSON types. If you use our API for example, you can work with DocumentSnapshots and Refs natively.
2. It provides annotations necessary for the firebase CLI to deploy your functions without any command-line flags (e.g. how much memory, cpu, etc to use)
3. It works with our emulator suite so that you can test your application locally

you can use the following code:

const functions = require('firebase-functions');

exports.makeUpperCase = functions.firestore.document("messages/{id}").onWrite((data, context) => {
    console.log(`Reading doc id ${context.params.id}`);

    const curValue = data.after.get("original");
    const newValue = curValue.toUpperCase();

    if (curValue !== newValue) {
        console.log(`Replacing value: ${curValue} --> ${newValue}`);
        return data.after.ref.update({ original: newValue });
    } else {
        // Value is already upper-case
        // Don't perform a(nother) write to avoid infinite loops
        // Nit: because we aren't setting anything like a timestamp, a write of the existing
        // data would have been a noop and not fired a new event anyway, so this isn't
        // strictly necessary.
        console.log('Value is already upper-case.');
    }
};

(FWIW, this will be cleaner with deep imports in the v4 SDK that is coming soon)

That will still work with gcloud functions deploy but give you better type information. For example, an onWrite or onUpdate event provides a { before: QueryDocumentSnapshot, after: QueryDocumentSnapshot } and the onCreate and onDelete events provide a QueryDocumentSnapshot.

If you decide to switch to the Firebase toolchain in addition to our SDK, this annotation will also let you call firebase emulators:start and we'll spin up an emulated version of Firestore wired up to an emulated version of GCF. And you can point your app to that local environment to get a fully functioning stack on your local machine.

[mamichels]

@inlined, thanks for the advise. It does work with some slight adjustments as you've described. For any future reader, I am leaving my observations.

I've tested it on a dummy function which i deployed via gcloud functions deploy.

import { EventContext } from 'firebase-functions';
import * as functions from 'firebase-functions';
import { QueryDocumentSnapshot } from 'firebase-functions/lib/providers/firestore';

export const testFirebaseFunction = functions.firestore
    // This path is ignored and can be replaced by an empty string. The trigger is set by the gcloud functions deploy command
    .document('{collection}/{id}')
    .onCreate(async (data: QueryDocumentSnapshot, context: EventContext) => {
    // Works like a charm
    console.log('data: ' + data);
    console.log('context: ' + context);

    // Those params will be undefined, as only params of the gcloud functions deploy trigger are being evaluated
    console.log('collection: ' + collection);
    console.log('id: ' + id);

    // Will be defined as this is part of the trigger configured with gcloud functions deploy
    console.log('document: ' + document);
});

Consider that this handler required you to return a Promise. So either work within a async function or specifically return one.

In your deployment the trigger configuration stays unchanged but you'll need to add another env var to your function. The package seems to use GCLOUD_PROJECT to implicitly set the credentials (done by firebase-admin?) to the given service account.

gcloud functions deploy test-firebase-sdk 
--region=europe-west1 
--runtime=nodejs16  
--memory=128Mi 
--trigger-event=providers/cloud.firestore/eventTypes/document.create 
--trigger-resource='projects/<MY_PROJECT>/databases/(default)/documents/ingress-event-repository-integration-test/{document}' 
--entry-point=testFirebaseFunction 
--source=./target 
--service-account=<SVC_ACCOUNT> 
--set-env-vars 'GCLOUD_PROJECT=<MY_PROJECT>';

[koistya]

How do you make it work with @google-cloud/functions-framework and @google-cloud/firestore? Something like this:

import { CloudEvent, cloudEvent } from "@google-cloud/functions-framework";
import { toDocumentEventData } from "@google/events/cloud/firestore/v1/DocumentEventData";
import protobuf, { Reader } from "protobufjs";

cloudEvent<Reader | Uint8Array>("main", async function handleEvent(event) {
  console.log(`Function triggered by event on: ${event.source}`);
  console.log(`Event type: ${event.type}`);

  console.log("Loading protos...");
  const root = await protobuf.load("data.proto");
  const DocumentEventData = root.lookupType(
    "google.events.cloud.firestore.v1.DocumentEventData",
  );

  console.log("Decoding data...");
  const data = toDocumentEventData(DocumentEventData.decode(event.data!));

  console.log("\nOld value:");
  console.log(JSON.stringify(data.oldValue, null, 2));

  console.log("\nNew value:");
  console.log(JSON.stringify(data.value, null, 2));
});

https://cloud.google.com/functions/docs/calling/cloud-firestore

[dconeybe]

@koistya Please open an issue in the https://github.com/GoogleCloudPlatform/functions-framework-nodejs repository for your specific issue.