Cloud Tasks is not supported by the Firebase Emulator Suite

[NuclearGandhi]

Environment info

firebase-tools: 11.6.0

Platform: Windows 11

Test case

Tasken from this question on StackOverflow.

I'm using the Firebase Emulator to emulate Firebase's Firestore, Functions and Authentication. By following the documentation, Enqueue functions with Cloud Tasks, I created a task queue named removeGroupCode():

exports.removeGroupCode = functions.tasks
  .taskQueue({
    .
    .
    .
  })
  .onDispatch(async (data) => {
    .
    .
    .
    }
  });

This removeGroupCode() function works fine both in production and in the local emulator. But for some reason it just doesn't get called when I'm calling it from another function in the local emulator:

exports.generateInviteCode = functions.https.onCall(async (data, context) => {
  .
  .
  .
  const queue = getFunctions(app).taskQueue("removeGroupCode");
  await queue.enqueue(
    {groupCode: groupCode},
    {scheduleDelaySeconds: 30, dispatchDeadlineSeconds: 60},
  );
  return {groupCode: groupCode};
});

Note: The above code also works fine in production, but I still would like it to work in the emulated environment for testing purposes.

Steps to reproduce

1. Setup and create a Firebase project environment for deploying Firebase Functions as shown in the documentation.
2. Create a Cloud Task function as shown here.
3. Create a Firebase function that calls the above Cloud Task function.
4. Run, and call the function from step 3 inside a local emulator.

Expected behavior

The Cloud Task function to get called in the local emulator.

Actual behavior

Function not getting called - below are the logs for the emulated environment and production environment for when createGroupCode() gets called:

Emulated:

Production (deployed):

[taeold]

@NuclearGandhi Thanks for writing up a detailed issue.

Sadly, Emulator doesn't support Cloud Tasks functions today - there isn't a "Google Cloud Task Emulator" that's hooked up to the Firebase Emulator Suite.

We also think it would be a great addition to the Emulator Suite, but we are focusing on other areas of improvement at the moment. I'm going to convert this issue into a feature request and will try to remember to post update here.

[trex-quo]

The way my team handled this was to create an internal mock for the CloudTaskClient that is used when our local flag is set. It just sends the request straight to the local emulated function instead of interacting with the Cloud Tasks client at all, which allows the function to be ran locally. It bypasses the queue functionality entirely, but that has been fine for us during local development since the rate of requests are so low and we usually just need the functions to run. Here is my implementation of the mock file if it is useful to you all.

const fetch = require("node-fetch");

class CloudTasksMock {
  constructor() {}

  queuePath(project, location, queue) {
    return `projects/${project}/locations/${location}/queues/${queue}`;
  }

  /**
   *
   * @param { import ("@google-cloud/tasks").protos.google.cloud.tasks.v2.ICreateTaskRequest } request
   */
  async createTask(request) {
    const { parent } = request;
    const httpRequest = request.task.httpRequest;

    console.log(
      `Received local queue request for parent ${parent}. Sending straight to function...`
    );

    if (httpRequest) {
      const { httpMethod, url, body } = httpRequest;

      const res = await fetch(url, {
        method: httpMethod,
        headers: {
          "content-type": "application/json",
        },
        //need to make sure to undo the base64 encoding before dispatching
        body: Buffer.from(body, "base64").toString("ascii"),
      });
      return [res];
    } else {
      throw new Error("cloudTasksMock - httpRequest is undefined");
    }
  }
}

module.exports = {
  CloudTasksMock,
};

Note that our Cloud Task processor functions are http.onRequest functions, and we have only been able to get these working properly with Cloud Task queues when the body is encoded in base64. Looking at my code it doesn't make much sense that I convert it back to ascii before sending to the local function and then perform another conversion in my local cloud function, but it works fine from what I can tell. Here is a small snippet of my Cloud Task processor function for context:

exports.cloudTaskProcessor = functions
  .runWith({
    memory: "512MB",
    timeoutSeconds: 540,
  })
  .https.onRequest(async (req, res) => {
    const reqBody = JSON.parse(
      Buffer.from(req.rawBody, "base64").toString("ascii")
    );
    ...
});

[Bastianowicz]

Thanks @trex-quo that inspired me but also led me in the wrong direction. On my local setup (for whatever reason) the port of the functions was already blocked by the request that should trigger the task in first place. This took me hours to figure out. But I have come up with a more direct solution for my case. This is a nest.js server but it will work likeways for any other environment.

I use the following as my functions factory in the dependency injection framework

                if (process.env.FUNCTIONS_EMULATOR) {
                    const logger = new Logger("Functions Factory");
                    logger.debug("Recognized Emulator environment. Stubbing Queues");
                    Object.assign(TaskQueue.prototype, {
                        enqueue: async function (data: never): Promise<void> {
                            return new Promise((resolve, reject) => {
                                const queue = main[this.functionName];
                                if (queue) {
                                    const httpBody = JSON.parse(JSON.stringify(data)); // important to experience comparable behaviour to prod env
                                    return resolve(queue.run(httpBody));
                                } else {
                                    logger.error("No such queue" + this.functionName);
                                    reject();
                                }
                            });
                        }
                    });
                }

                return getFunctions(adminApp);

It overrides (overwrites?) the original enqueue function. This new function checks if there is a function exportet with the name of the queue this task is being dispatched to. If so the data is being passed on.

This was a tough one. Hope it helps someone else. Maybe this could be a solution for @joehan as well. idk.

[moifort]

Like @Bastianowicz, this is my workaround. Put the code below just after your initializeApp(), it will display the message send to your queue.

Console

i  functions: Loaded environment variables from .env, .env.meeting-work-01, .env.local.
i  functions: Beginning execution of "meeting.task.sendCreatedAndFinishedEvents"
>  tasks:  { id: '[email protected]', status: 'started' } { scheduleTime: 2023-03-07T08:00:00.000Z }
>  tasks:  { id: '[email protected]', status: 'finished' } { scheduleTime: 2023-03-07T09:00:00.000Z }
i  functions: Finished "meeting.task.sendCreatedAndFinishedEvents" in 5.472542ms

Solution

import { initializeApp } from 'firebase-admin/app'
import { getFunctions, TaskQueue } from 'firebase-admin/functions'

const app = initializeApp()
if (process.env.FUNCTIONS_EMULATOR) {
    Object.assign(TaskQueue.prototype, {
      enqueue: (data: any, params: any) =>
        console.debug('tasks: ', data, params),
  })
}
const functions = getFunctions(app)

[trylovetom]

Any plan to support it? It's very useful.

[christhompsongoogle]

Hi @trylovetom, nothing to share at this time.

[luicfrr]

+1

[emiliobasualdo]

This was my solution in Typescript. I had to separate the task function from the actual task creation.

This is an example code for the emulator enqueuing the selectBiddingWinner task.

import {TaskQueue as FTaskQueue} from "firebase-admin/functions";

declare interface TaskQueue {
    functionName: string;
    client: unknown;
    extensionId?: string;
    enqueue(data: Record<string, string>, opts?: TaskOptions): Promise<void>
}

if (process.env.FUNCTIONS_EMULATOR) {
    FTaskQueue.prototype.enqueue = function (this: TaskQueue, data: Record<string, string>, opts?: TaskOptions) {
        logger.debug(this.functionName, data);
        return new Promise(() => {
            if (this.functionName == "locations/southamerica-east1/functions/selectBiddingWinnerTask") {
                return selectBiddingWinner(data.orderId);
            } else {
                return;
            }
        });
    };
}

This is an example of the task definition

exports.selectBiddingWinnerTask = tasks.taskQueue({
    retryConfig: {
        maxAttempts: 5,
        minBackoffSeconds: 3,
    },
    rateLimits: {
        maxConcurrentDispatches: 10,
    },
}).onDispatch(async (data: Record<string, string>) => {
    const orderId = data.orderId;
    return selectBiddingWinner(orderId)
        .then((resp) => logger.info(resp))
        .catch((e) => logger.error(`selectWinner failed for order:${orderId}`, e));
});

selectBiddingWinner is the actual function executed.

I'll be glad to receive comments/corrections that would help me generalize the code :)

[daltyboy11]

@NuclearGandhi Thanks for writing up a detailed issue.

Sadly, Emulator doesn't support Cloud Tasks functions today - there isn't a "Google Cloud Task Emulator" that's hooked up to the Firebase Emulator Suite.

We also think it would be a great addition to the Emulator Suite, but we are focusing on other areas of improvement at the moment. I'm going to convert this issue into a feature request and will try to remember to post update here.

It's frankly ridiculous that this still isn't supported after almost a year, given that the official code example enqueues a task directly from a cloud function.

There's no way to test locally a cloud task being called from a cloud function, despite it being the main use case. Do I have that right?

[Bastianowicz]

I agree that this is disappointing but since you found this thread you may workaround using some of the solutions provided. I feel your disappointment here. But let's assume that the people that provided this useful toolset may just be pretty occupied and not simply mean.

[eyal-mor]

Thank you @daltyboy11 for the example!

Extended the example to also provide a nice Queue wrapper and resolver, with states.

I'm using this in an app that is running each task with a VERY rate limited API. As such, i'm sure there are tweaks that can be done to better fit other needs.

This of course isn't the best solution in mind, as the implementation runs during the HTTP dispatch and also processes the entire Queue, defeating the purpose of said Queue.

For development sake, I believe this is a start.

// Mock TaskQueue.enqueue implementation
// Extended from: https://github.com/firebase/firebase-tools/issues/4884#issuecomment-1485075479

import { TaskQueue as FTaskQueue, TaskOptions } from "firebase-admin/functions";

declare interface TaskQueue {
  functionName: string;
  client: unknown;
  extensionId?: string;
  enqueue(data: Record<string, string>, opts?: TaskOptions): Promise<void>
}

type QueueFunc = (args: Object | null) => Promise<void>;

if (process.env.FUNCTIONS_EMULATOR === 'true') {
  process.env.FIRESTORE_EMULATOR_HOST = 'localhost:8080';

  // Local Queue to run in the emulator
  // This queue is filled when the http call is invoked.
  let queue: {func: QueueFunc, args: Object | null, state: 'pending' | 'processing' | 'done'}[] = [];

  FTaskQueue.prototype.enqueue = function (this: TaskQueue, data: Record<string, string>, opts?: TaskOptions) {
    functions.logger.debug("enqueueing", this.functionName, data, "task count", queue.length);

    return new Promise(() => {
      if (this.functionName.endsWith('updateNearbySalesAverages')) {
        queue.push({
          func: updateNearbySalesAverages,
          args: { data, opts },
          state: 'pending',
        });
        return;
      } else {
        return;
      }
    });
  };

  // Process the queue every second
  setInterval(() => {
    functions.logger.debug("processing queue", queue.length);
    if (queue.length === 0) {
      return;
    };

    // Remove the latest done task.
    // This can be improved to get all done tasks.
    const taskDone = queue.findIndex(task => task.state === 'done');
    if (taskDone >= 0) {
      queue.splice(taskDone, 1);
    }

    // If a task is in processing, we can return early.
    // This may need more work to configure based on specific needs.
    const taskProcessing = queue.findIndex(task => task.state === 'processing');

    if (taskProcessing >= 0) {
      return;
    }

    // Get the first pending task
    const taskPending = queue.findIndex(task => task.state === 'pending');

    // If there are no pending tasks, return early
    if (taskPending === -1) {
      return;
    }

    // Get the function and args for the pending task
    const {func, args} = queue[taskPending];
    // Set the task to `processing` state.
    queue[taskPending].state = 'processing';

    // Implementation dependant, my code always uses an Async function.
    func(args).then(() => {
      // When the tasks is done, set the state to `done`
      queue[taskPending].state = 'done';
    }).catch((err) => {
      // If the task errors, set the state to `done`
      // Log the error
      functions.logger.error("error processing task", err);
      queue[taskPending].state = 'done';
    });
  }, 1000);
}

[louismorgner]

+1 for making this smoothly in the local emulator. Yes, there are workarounds which can get us there, but it is a friction point for working with this easily. Surely a useful addition which I hope will be shipped at some point by the team.

[cadyfatcat]

If nothing else, better explanation of the limitation of queueing tasks in the emulator environment should be included in the documentation. Would have saved a lot of time before finding this thread.

[dzivoing]

Any news on the state of the emulator ?

[salehsed]

+1

[malcolmamal]

I love the emulator but I agree that it would also be lovely to have the task queue there as well so a +1 from me :)

[ssadel]

bump

[Maarten0110]

Would love to have this feature too!

[johnneast]

One of the things I've enjoyed the most about working with Firebase since adopting it is the developer experience of using the emulators for local developers. Having the ability to enqueue tasks natively in the emulator instead of having to use one of the work arounds shared above would be a great addition.