[DSIP-7] [API] Suggest refactor the backend api

[SbloodyS]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Description

Purpose: A user-friendly and high-performance restful API is our goal.

Refactor reason:

• Use restful style to unify API specifications
• Solve the current incomplete API document and reduce the threshold of API use
• Improve API reusability
• Reduce unnecessary data return and improve performance
• Improve the readability of the code and reduce the maintenance cost
• Improve the accuracy of error reporting

API output Standard:

1. Content-Type using application/json(segmented upload file is not applicable)
2. Users can simply understand how to call the API normally by reading the API docs
3. The input and output parameters of API should avoid using types that cannot be intuitively understood, such as JSON and MAP
4. All enumeration values in the API need to be included in the API document
5. Comply with restful specifications
6. The error information requested by API can accurately express the cause of the error
7. Rename all process to workflow as we discussed in https://github.com/apache/dolphinscheduler/issues/10257#issuecomment-1175126227
8. The service refactoring part and the controller refactoring part should be divided into two PR base on the conclusions in https://github.com/apache/dolphinscheduler/pull/10998#issuecomment-1207708303

Please reply the controller name and its function name you want to participate in and the issue you created. I will link it to this issue and assign it to you. Anyone who wants to participate is very welcome ^_^

Current API

Login page

Controller	Function	issue	status
LoginController	login	#10459
LoginController	signOut	#10459

Project manage page

Project

Controller	Function	issue	status
ProjectController	createProject	#10489	✅
ProjectController	updateProject	#10489	✅
ProjectController	queryProjectByCode	#10489	✅
ProjectController	queryProjectListPaging	#10489	✅
ProjectController	deleteProject	#10489	✅
ProjectController	queryUnauthorizedProject	#10489	✅
ProjectController	queryAuthorizedProject	#10489	✅
ProjectController	queryAuthorizedUser	#10489	✅
ProjectController	queryProjectCreatedAndAuthorizedByUser	#10489	✅
ProjectController	queryAllProjectList	#10489	✅

Workflow Monitor

Controller	Function	issue	status
DataAnalysisController	countTaskState
DataAnalysisController	countProcessInstanceState
DataAnalysisController	countDefinitionByUser
DataAnalysisController	countCommandState
DataAnalysisController	countQueueState

Workflow Relationship

Controller	Function	issue	status
WorkFlowLineageController	queryWorkFlowLineageByName
WorkFlowLineageController	queryWorkFlowLineageByCode
WorkFlowLineageController	queryWorkFlowLineage
WorkFlowLineageController	verifyTaskCanDelete

Workflow Definition

Controller	Function	issue	status
ProcessDefinitionController	createProcessDefinition	#11129
ProcessDefinitionController	copyProcessDefinition	#11129
ProcessDefinitionController	moveProcessDefinition	#11129
ProcessDefinitionController	verifyProcessDefinitionName	#11129
ProcessDefinitionController	updateProcessDefinition	#11129
ProcessDefinitionController	queryProcessDefinitionVersions	#11129
ProcessDefinitionController	switchProcessDefinitionVersion	#11129
ProcessDefinitionController	deleteProcessDefinitionVersion	#11129
ProcessDefinitionController	releaseProcessDefinition	#11129
ProcessDefinitionController	queryProcessDefinitionByCode	#11129
ProcessDefinitionController	queryProcessDefinitionByName	#11129
ProcessDefinitionController	queryProcessDefinitionList	#11129
ProcessDefinitionController	queryProcessDefinitionSimpleList	#11129
ProcessDefinitionController	queryProcessDefinitionListPaging	#11129
ProcessDefinitionController	viewTree	#11129
ProcessDefinitionController	getNodeListByDefinitionCode	#11129
ProcessDefinitionController	getNodeListMapByDefinitionCodes	#11129
ProcessDefinitionController	getProcessListByProjectCodes	#11129
ProcessDefinitionController	getTaskListByProcessDefinitionCode	#11129
ProcessDefinitionController	deleteProcessDefinitionByCode	#11129
ProcessDefinitionController	batchDeleteProcessDefinitionByCodes	#11129
ProcessDefinitionController	batchExportProcessDefinitionByCodes	#11129
ProcessDefinitionController	queryAllProcessDefinitionByProjectCode	#11129
ProcessDefinitionController	importProcessDefinition	#11129
ProcessDefinitionController	createEmptyProcessDefinition	#11129
ProcessDefinitionController	updateProcessDefinitionBasicInfo	#11129
ProcessDefinitionController	releaseWorkflowAndSchedule	#11129

Workflow Instance

Controller	Function	issue	status
ProcessInstanceController	queryProcessInstanceList	#10263
ProcessInstanceController	queryTaskListByProcessId	#10263
ProcessInstanceController	updateProcessInstance	#10263
ProcessInstanceController	queryProcessInstanceById	#10263
ProcessInstanceController	queryTopNLongestRunningProcessInstance	#10263
ProcessInstanceController	deleteProcessInstanceById	#10263
ProcessInstanceController	querySubProcessInstanceByTaskId	#10263
ProcessInstanceController	queryParentInstanceBySubId	#10263
ProcessInstanceController	viewVariables	#10263
ProcessInstanceController	viewTree	#10263
ProcessInstanceController	batchDeleteProcessInstanceByIds	#10263

Task Definition

Controller	Function	issue	status
TaskDefinitionController	createTaskDefinition
TaskDefinitionController	createTaskBindsWorkFlow
TaskDefinitionController	updateTaskDefinition
TaskDefinitionController	updateTaskWithUpstream
TaskDefinitionController	queryTaskDefinitionVersions
TaskDefinitionController	switchTaskDefinitionVersion
TaskDefinitionController	deleteTaskDefinitionVersion
TaskDefinitionController	deleteTaskDefinitionByCode
TaskDefinitionController	queryTaskDefinitionDetail
TaskDefinitionController	queryTaskDefinitionListPaging
TaskDefinitionController	genTaskCodeList
TaskDefinitionController	releaseTaskDefinition

Task Instance

Controller	Function	issue	status
TaskInstanceController	queryTaskListPaging	#11713
TaskInstanceController	forceTaskSuccess	#11713

Task Group

Controller	Function	issue	status
TaskGroupController	createTaskGroup
TaskGroupController	updateTaskGroup
TaskGroupController	queryAllTaskGroup
TaskGroupController	queryTaskGroupByStatus
TaskGroupController	queryTaskGroupByCode
TaskGroupController	closeTaskGroup
TaskGroupController	startTaskGroup
TaskGroupController	forceStart
TaskGroupController	modifyPriority
TaskGroupController	queryTasksByGroupId

Mornitor center

Controller	Function	issue	status
MonitorController	listMaster	#10458
MonitorController	listWorker	#10458
MonitorController	queryDatabaseState	#10458

Datasource center

Controller	Function	issue	status
DataSourceController	createDataSource	#10460
DataSourceController	updateDataSource	#10460
DataSourceController	queryDataSource	#10460
DataSourceController	queryDataSourceListPaging	#10460
DataSourceController	connectDataSource	#10460
DataSourceController	connectionTest	#10460
DataSourceController	deleteDataSource	#10460
DataSourceController	verifyDataSourceName	#10460
DataSourceController	unauthDatasource	#10460
DataSourceController	authedDatasource	#10460
DataSourceController	getKerberosStartupState	#10460
DataSourceController	getTables	#10460
DataSourceController	getTableColumns	#10460

Resource center

Controller	Function	issue	status
ResourcesController	createDirectory	#10983
ResourcesController	createResource	#10983
ResourcesController	updateResource	#10983
ResourcesController	queryResourceList	#10983
ResourcesController	queryResourceListPaging	#10983
ResourcesController	deleteResource	#10983
ResourcesController	verifyResourceName	#10983
ResourcesController	queryResourceJarList	#10983
ResourcesController	queryResource	#10983
ResourcesController	viewResource	#10983
ResourcesController	onlineCreateResource	#10983
ResourcesController	updateResourceContent	#10983
ResourcesController	downloadResource	#10983
ResourcesController	createUdfFunc	#10983
ResourcesController	viewUIUdfFunction	#10983
ResourcesController	updateUdfFunc	#10983
ResourcesController	queryUdfFuncListPaging	#10983
ResourcesController	queryUdfFuncList	#10983
ResourcesController	verifyUdfFuncName	#10983
ResourcesController	deleteUdfFunc	#10983
ResourcesController	authorizedFile	#10983
ResourcesController	authorizeResourceTree	#10983
ResourcesController	unauthUDFFunc	#10983
ResourcesController	authorizedUDFFunction	#10983
ResourcesController	queryResourceById	#10983
TaskGroupController	createTaskGroup
TaskGroupController	updateTaskGroup
TaskGroupController	queryAllTaskGroup
TaskGroupController	queryTaskGroupByStatus
TaskGroupController	queryTaskGroupByCode
TaskGroupController	closeTaskGroup
TaskGroupController	startTaskGroup
TaskGroupController	forceStart
TaskGroupController	queryTasksByGroupId

Dataquality center

Controller	Function	issue	status
DataQualityController	getRuleFormCreateJsonById
DataQualityController	queryRuleListPaging
DataQualityController	queryRuleList
DataQualityController	queryExecuteResultListPaging
DataQualityController	getDatasourceOptionsById

Security center

Controller	Function	issue	status
AccessTokenController	queryProcessInstanceList	#10290	✅
AccessTokenController	generateToken	#10459
AccessTokenController	queryAccessTokenList	#10459
AccessTokenController	queryAccessTokenByUser	#10459
AccessTokenController	delAccessTokenById	#10459
TenantController	createTenant	#10467
TenantController	queryTenantlistPaging	#10467
TenantController	queryTenantlist	#10467
TenantController	updateTenant	#10467
TenantController	deleteTenantById	#10467
TenantController	verifyTenantCode	#10467
UsersController	createUser
UsersController	queryUserList
UsersController	updateUser
UsersController	delUserById
UsersController	grantProject
UsersController	grantProjectByCode
UsersController	revokeProject
UsersController	grantResource
UsersController	grantUDFFunc
UsersController	grantNamespace
UsersController	grantDataSource
UsersController	getUserInfo
UsersController	listUser
UsersController	listAll
UsersController	verifyUserName
UsersController	unauthorizedUser
UsersController	authorizedUser
UsersController	registerUser
UsersController	activateUser
UsersController	batchActivateUser
AlertGroupController	createAlertgroup
AlertGroupController	list
AlertGroupController	listPaging
AlertGroupController	queryAlertGroupById
AlertGroupController	updateAlertgroup
AlertGroupController	delAlertgroupById
AlertGroupController	verifyGroupName
AlertPluginInstanceController	createAlertPluginInstance
AlertPluginInstanceController	updateAlertPluginInstance
AlertPluginInstanceController	deleteAlertPluginInstance
AlertPluginInstanceController	getAlertPluginInstance
AlertPluginInstanceController	getAlertPluginInstance
AlertPluginInstanceController	verifyGroupName
AlertPluginInstanceController	listPaging
WorkerGroupController	saveWorkerGroup	#12072
WorkerGroupController	queryAllWorkerGroupsPaging	#12072
WorkerGroupController	queryAllWorkerGroups	#12072
WorkerGroupController	deleteWorkerGroupById	#12072
WorkerGroupController	queryWorkerAddressList	#12072
QueueController	queryList	#11067	✅
QueueController	queryQueueListPaging	#11067	✅
QueueController	createQueue	#11067	✅
QueueController	updateQueue	#11067	✅
QueueController	verifyQueue	#11067	✅
EnvironmentController	createEnvironment
EnvironmentController	updateEnvironment
EnvironmentController	queryEnvironmentByCode
EnvironmentController	queryEnvironmentListPaging
EnvironmentController	deleteEnvironment
EnvironmentController	queryAllEnvironmentList
EnvironmentController	verifyEnvironment
ClusterController	createProject
ClusterController	updateCluster
ClusterController	queryClusterByCode
ClusterController	queryClusterListPaging
ClusterController	deleteCluster
ClusterController	queryAllClusterList
ClusterController	verifyCluster
K8sNamespaceController	queryProjectListPaging
K8sNamespaceController	createNamespace
K8sNamespaceController	updateNamespace
K8sNamespaceController	verifyNamespace
K8sNamespaceController	delNamespaceById
K8sNamespaceController	queryUnauthorizedNamespace
K8sNamespaceController	queryAuthorizedNamespace
K8sNamespaceController	queryAvailableNamespaceList

Scheduler

Controller	Function	issue	status
SchedulerController	ALL	#11912

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[github-actions[bot]]

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

• In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
• If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting

[zhongjiajie]

Thanks @SbloodyS and @songjianet to bring this up, and it is worth the label DSIP. It looks good overall, I think maybe we could also add some check/tests and out API style itself to make sure we are restful and make sure out API docs same as our code. I am not sure whether exists this tools or not

[SbloodyS]

1. Do you guys have any goods suggestions for the new api' path? For example using /v2 to distinguish it from the old api.

2. Currently we use {code: 0, msg: "成功", data: {}, failed: false, success: true} as our api's response structure. I suggest remove meaningless failed and success and use code to judge success or failed. That is {code: 0, msg: "成功", data: {}}.

@zhongjiajie @songjianet @caishunfeng

[caishunfeng]

1. Do you guys have any goods suggestions for the new api' path? For example using /v2 to distinguish it from the old api.
2. Currently we use {code: 0, msg: "成功", data: {}, failed: false, success: true} as our api's response structure. I suggest remove meaningless failed and success and use code to judge success or failed. That is {code: 0, msg: "成功", data: {}}.

@zhongjiajie @songjianet @caishunfeng

Argee with you. we would better to use /v2 api.
And I found the doc of resful api standard: https://dolphinscheduler.apache.org/zh-cn/development/api-standard.html

[songjianet]

1. Do you guys have any goods suggestions for the new api' path? For example using /v2 to distinguish it from the old api.
2. Currently we use {code: 0, msg: "成功", data: {}, failed: false, success: true} as our api's response structure. I suggest remove meaningless failed and success and use code to judge success or failed. That is {code: 0, msg: "成功", data: {}}.

@zhongjiajie @songjianet @caishunfeng

Argee with you. we would better to use /v2 api. And I found the doc of resful api standard: https://dolphinscheduler.apache.org/zh-cn/development/api-standard.html

This lacks the patch request method.

[SbloodyS]

I found that our restful api standard doc shows that the api include path /api/dolphinscheduler/. But in dev branch the path /api/ did not exists. Should we change the doc or use /api/v2 path? @caishunfeng @zhongjiajie

[zhongjiajie]

I found that our restful api standard doc shows that the api include path /api/dolphinscheduler/. But in dev branch the path /api/ did not exists. Should we change the doc or use /api/v2 path? @caishunfeng @zhongjiajie

I think we should correct the document, and FYI, I think one of this resource it is because our documentation are in too many places, and developer documen without version, I create an issue https://github.com/apache/dolphinscheduler/issues/9927.

[qingwli]

Count me in

[SbloodyS]

Count me in

I will make a list of all the backend api of ds for everyone to choose in a few days.

[guodongym]

Count me in @SbloodyS

[SbloodyS]

Count me in @SbloodyS

Sure.

[EricGao888]

Super coooool! Count me in plz @SbloodyS

[SbloodyS]

Some apis to be reconstructed have been added to this issue. Please reply the controller name and its function name you want to participate in and the issue you created. I will link it to this issue and assign it to you. Please feel free to take it. @liqingwang @guodongym @EricGao888

[EricGao888]

@SbloodyS May I pick the monitor page? Thx #10458

[SbloodyS]

@SbloodyS May I pick the monitor page? Thx #10458

Sure. I've assigned it to you. Looking forward to your contribution.

[qingwli]

May I pick the Security center && login part? @SbloodyS

[SbloodyS]

AccessTokenController

What you want is the whole security center or just the AccessTokenController? Accesstokencontroller is only a part of the security center.

[qingwli]

Just AccessTokenController && LoginController first.

[SbloodyS]

Just AccessTokenController && LoginController first.

Sure. Please let me know after your issue is created.

[qingwli]

#10459 Thx

[czeming]

Count me in @SbloodyS

[SbloodyS]

Count me in @SbloodyS

Sure. Which do you want to take?

[guodongym]

@SbloodyS May I pick the Datasource center? Thx #10460

[SbloodyS]

@SbloodyS May I pick the Datasource center? Thx #10460

Sure. I've assigned to you.

[tracehh]

Count me in plz. Just ProjectController first. @SbloodyS

[SbloodyS]

Count me in plz. Just ProjectController first. @SbloodyS

Sure. Looking forward to your contribution.

[WangJPLeo]

This is a good refactoring plan for me. The audit log has made an Around aspect to all non-query/statistics APIs, and customized processing based on the current API's local method name and interaction data. After the refactoring, the processing method involving the API will be more convenient and unified, so count me in please @SbloodyS

[EricGao888]

This is a good refactoring plan for me. The audit log has made an Around aspect to all non-query/statistics APIs, and customized processing based on the current API's local method name and interaction data. After the refactoring, the processing method involving the API will be more convenient and unified, so count me in please @SbloodyS

Hi @WangJPLeo May I ask whether you are working on the auditing feature currently?

[SbloodyS]

This is a good refactoring plan for me. The audit log has made an Around aspect to all non-query/statistics APIs, and customized processing based on the current API's local method name and interaction data. After the refactoring, the processing method involving the API will be more convenient and unified, so count me in please @SbloodyS

Sure. Which do you want to take?

[WangJPLeo]

This is a good refactoring plan for me. The audit log has made an Around aspect to all non-query/statistics APIs, and customized processing based on the current API's local method name and interaction data. After the refactoring, the processing method involving the API will be more convenient and unified, so count me in please @SbloodyS

Hi @WangJPLeo May I ask whether you are working on the auditing feature currently?

yes, an audit log has been implemented in the project now. @EricGao888