[Feature][Api Worker] Generic Third-party System API Connector for REST-based Task Scheduling

[dill21yu]

Search before asking

• [x] I had searched in the issues and found no similar feature requirement.

Description

Propose a Generic Third-party System API Connector to enable DolphinScheduler to schedule tasks on any external system via REST APIs.

Currently, DS lacks a unified way to integrate with systems like SeaTunnel, Flink Session, or custom platforms that expose task control via RESTful interfaces. This feature would provide a configurable framework (managing endpoints for auth, task list, start, status, stop) instead of requiring custom plugins for each system, greatly expanding DS's scheduling capabilities.

Use case

As a DS administrator or user, I want to register an external system (e.g., company's customized SeaTunnel system) or other by providing its base URL, authentication, and specific API endpoints (to list, start, check status, and stop tasks), So that I can easily select and schedule that system's tasks directly within a DolphinScheduler workflow.

The expected behavior is: 1、In the DS UI, I can add a new "Third-party System" config for our SeaTunnel server's API. 2、When building a workflow, I can choose the "Generic API Task" type, select the registered "SeaTunnel" system, and see a dropdown list of available jobs fetched from its API. 3、Upon selecting a job and running the workflow, DS automatically calls the start API, polls the status API until completion, and reflects the final status (success/failure) in DS. 4、I can also manually stop the task from DS, which triggers the stop API call to the external system. This will allow us to use DS as a unified scheduler for many API-based tools without waiting for specific plugin development.

Related issues

No response

Are you willing to submit a PR?

• [x] Yes I am willing to submit a PR!

Code of Conduct

• [x] I agree to follow this project's Code of Conduct

[ruanwenjun]

Basically LGTM, I would prefer to move this into datasource.

[dill21yu]

Basically LGTM, I would prefer to move this into datasource.

@ruanwenjun Thanks for the feedback!

I agree with your suggestion to leverage the existing Datasource module to manage configurations for third-party systems. This aligns well with how DolphinScheduler already handles non-JDBC resources like SSH — treating a REST-managed external system (e.g., SeaTunnel, custom platforms ) as a "logical endpoint" with connection metadata.

Here’s an updated proposal based on your input:

🔧 Use Datasource for Third-party System Configuration

• Introduce a new datasource type: Generic API Connector (or REST API Endpoint) in the Datasource module.
• Store configuration including:
  • Base URL
  • Authentication method
  • API endpoint templates (for list, start, status check, stop) 🔄 Task Integration
• A new "Generic API Task" will reference a configured Generic API Connector.
• When selected, the UI will:
  1. Fetch available jobs from the /list endpoint (via backend proxy).
  2. Allow users to select one job.
  3. Use the mapped APIs to start, poll status, and stop the job.

This approach keeps configuration centralized and secure, while enabling flexible integration without requiring custom plugins for each system.

Could you or other maintainers please advise if there are any other modules (e.g., security, alerting, resource management) that may also need to be considered or modified to fully support this feature? I'd appreciate any additional suggestions to ensure the design is aligned with DolphinScheduler's architecture.