bullmq icon indicating copy to clipboard operation
bullmq copied to clipboard

Is it possible to share a job between different flows ?

Open guillaumeklaus opened this issue 1 year ago • 1 comments

On my current project I use bullmq (with NestJS) to queue jobs needed to generate a rather big pdf.

Each page of my pdf gathers informations from different db or api calls.

Some pages need the same first job, currently this first job is duplicated, my question is : is it possible to share this job between different flows ?

My code example :

   const firstPageGenerationFlow = await this.flowProducer.add({
      name: 'firstPageGeneration',
      queueName: 'db-api-calls',
      data: { someObject },
      children: [
        {
          name: 'mapStaticImageGeneration',
          queueName: 'db-api-calls',
          data: { someObject },
          children: [
            {
              name: 'parcelComplementaryInformations',
              queueName: 'db-api-calls',
              data: { someObject },
              children: [
                {
                  name: 'parcelRetrieval',
                  queueName: 'db-api-calls',
                  data: { someObject }
                }
              ]
            }
          ]
        }
      ]
    });
    const secondPageGenerationFlow = await this.flowProducer.add({
      name: 'secondPageGeneration',
      queueName: 'db-api-calls',
      data: { someObject },
      children: [
        {
          name: 'mapStaticImageGenerationUrbanism',
          queueName: 'db-api-calls',
          data: { someObject },
          children: [
            {
              name: 'getNearbyUrbanismPermits',
              queueName: 'db-api-calls',
              data: { someObject },
              children: [
                {
                  name: 'parcelRetrieval',
                  queueName: 'db-api-calls',
                  data: { someObject }
                }
              ]
            }
          ]
        }
      ]
    });

Both flows depend on parcelRetrieval, my goal would be to deduplicate this, any thoughts ?

guillaumeklaus avatar May 20 '24 14:05 guillaumeklaus

I think this is a case of the multiple parents feature request that some people have been requesting in the past. Unfortunatelly not supported. I am not sure if in your case you could solve this by having the job parcelRetrieval be a standard job, that as last thing it does is creating the two flows firstPageGeneration and secondPageGeneration, passing its result to both of them.

manast avatar May 23 '24 16:05 manast