Support custom background tasks in worker

[dahlia]

[!NOTE] We've decided to postpone custom background task support to Fedify 2.0.0 instead of 1.5.0 as originally planned.

This feature requires significant API changes that would be too disruptive for a minor version update. We need more time to properly design a robust worker architecture that addresses the considerations raised in this issue.

Some smaller improvements that don't require API changes will still be included in Fedify 1.5.0 or subsequent minor updates.

Please continue sharing your use cases and requirements here to help shape the implementation for 2.0.0. Thank you for your understanding.

Currently, Fedify has a built-in background worker that processes incoming and outgoing activities through a message queue. However, applications using Fedify might need their own background tasks as well. This issue proposes to extend Fedify's worker system to support custom tasks, allowing users to utilize Fedify's background worker for their own tasks.

Scope consideration

Before diving into implementation details, we should consider whether providing custom task support aligns with Fedify's core mission and scope. Fedify is primarily an ActivityPub server framework, and adding general-purpose background task functionality might be stepping outside its intended scope. Some concerns:

• It could make Fedify more complex and harder to maintain
• Users might be better served by dedicated task queue solutions (e.g., Bull, Celery)
• It might dilute Fedify's focus on ActivityPub/fediverse features

However, there are also arguments in favor of this feature:

• Many fediverse applications need background tasks that are closely tied to ActivityPub operations
• Providing integrated task support could simplify application architecture
• Users wouldn't need to set up and maintain a separate task system

This is an important consideration that should be discussed before proceeding with implementation.

Goals

• Allow users to define and register custom tasks
• Allow users to enqueue custom tasks from anywhere in their application
• Provide type safety for task data
• Support task-specific retry policies and other options
• Maintain backward compatibility

Proposed solutions

We have two potential approaches to implement this feature:

Approach 1: Extension of current design

This approach adds new APIs to the existing Federation object without major architectural changes:

// Define custom task
interface CustomTask<TData = unknown> {
  type: string;
  data: TData;
}

// Register task handler
federation.registerTaskHandler<TData>(
  type: string,
  handler: (ctx: Context<TContextData>, data: TData) => Promise<void> | void
);

// Enqueue task
context.enqueueTask<TData>(
  type: string,
  data: TData,
  options?: MessageQueueEnqueueOptions
);

Pros

• Minimal changes to existing architecture
• Easier to implement
• Maintains backward compatibility
• Quick to ship

Cons

• Keeps worker functionality tightly coupled with Federation
• Less flexibility for worker process management
• Limited scalability options

Approach 2: Separate worker architecture

This approach separates worker functionality into its own module:

// Worker configuration
interface WorkerOptions<TContextData> {
  queue: MessageQueue;
  tasks?: Record<TaskType, boolean>;
  contextData?: TContextData;
}

// Dedicated worker class
class Worker<TContextData> {
  registerTaskHandler<TData>(...);
  start(signal?: AbortSignal): Promise<void>;
}

// Task definition utility
interface TaskDefinition<TData> {
  type: string;
  validateData?: (data: unknown) => data is TData;
  retryPolicy?: RetryPolicy;
}

Pros

• Better separation of concerns
• More flexible scaling options
• Independent testing of server and worker logic
• More control over worker processes
• Selective task processing per worker

Cons

• Requires major refactoring
• More complex implementation
• Migration effort for existing users
• Takes longer to implement

Questions to consider

1. Should we move forward with this feature, given the scope concerns?
2. If yes, should we prioritize quick implementation (Approach 1) or better architecture (Approach 2)?
3. If we choose Approach 2, what's our migration strategy for existing users?
4. How do we handle task versioning and backwards compatibility?
5. What monitoring and management capabilities should we provide?

Next Steps

1. Discuss whether this feature aligns with Fedify's scope
2. If decided to proceed:
   • Gather feedback from the community on these approaches
   • Decide on the implementation approach
   • Create a detailed implementation plan
   • Write documentation for the new feature

[jfietkau]

Seems like it would be useful if it was available. Your designs look encouragingly like what I came up with on my own, except if it was written by someone who actually knows TypeScript. 🙂

In my implementation, tasks have a type (to differentiate behavior and coordinate callback handlers) and a key (to identify the target object to be worked on and to prevent duplicate entries in the queue). I don't know how universal the assumption is that preventing multiple identical tasks on the queue makes sense, but when I tried making things work with Deno's inbuilt message queue, having no easy way to check if a particular task was already on the queue made the management of recurring tasks and the bootstrapping process after restarting the application so error-prone that I decided to write my own queue instead. If Fedify had a task queue available at the time, maybe that would have worked better.

I can't claim to have an opinion on which architectural approach is more sensible.