dolphinscheduler icon indicating copy to clipboard operation
dolphinscheduler copied to clipboard

Published 20 hours ago • verified publisher icon apache

[Improvement][Audit log] Audit log improvement design

Open qingwli opened this issue 1 year ago • 7 comments

Search before asking

  • [X] I had searched in the issues and found no similar feature requirement.

Description

1. Goal

Record user operation logs like create, update, delete, run, and stop jobs, etc. Easy to track item state.

1. Previous Design

https://github.com/apache/dolphinscheduler/issues/5822

1. Realize Method

  • Plan a: publish and subscribe mode . After user completes the modification operation, the operation behavior information is published as the form of message
  • Plan B: implementation using AOP

https://github.com/apache/dolphinscheduler/issues/5822#issuecomment-882023608

AOP can be used to get the relevant operation records in the form of log after the end of user operation, but it is not convenient to judge whether the operation is successful.

If use the publish subscribe mode , a message can be sent to the subscriber after the user's operation is successful.

If only need to record the successfully configuration modification, it will be better to use publish and subscribe mode.

Previous design choose public subscribe mode. The reason why is just want to record success operation. And AOP can support this function.

POC:

private void saveLog(ProceedingJoinPoint joinPoint, Object object) {

        Result result = (Result)object;

        // We can judge by this code. 0 means the api is successful, other codes means we can skip this request.
        System.out.println(result.getCode());
}

Publish and subscribe mode has high code intrusive, we need to add log method anywhere. So we thought AOP it's more better.

2. Schema Design

User Name Resource Type Project Name Operation Type Time
admin PROJECT NewProject CREATE 2023-12-28 10:40:23
admin USER NewUser CREATE 2023-12-28 10:40:23

2. Deficiencies

The recorded operation granularity is too rough. Not clear enough to show the level like workflow and project.

3. Classification

Project

  • Project (Create, Update, Delete)
  • Workflow (Create, Update, Delete, Import, Export, Copy, Start, Online, Offline)
  • Workflow Instance (Edit, Rerun, Stop, Kill, Pause)
  • Task (Create, Update, Delete, Move, Switch version, Delete version)
  • Task Instance (Force success)
  • Schedule(Create, Update, Delete, Online)

Resource

  • Floder (Create, Delete, Edit, Rename)
  • File (Create, Delete, ReUpload, Edit, Rename, Upload)
  • UDF Floder (Create, Delete, Edit)
  • UDF (Upload, Edit, Delete)
  • UDF Function (Create, Edit, Delete)
  • Task Group (Create, Update, Switch status)

Datasource

  • Datasource (Create, Update, Delete)

Security

  • Tenant (Create, Update, Delete)
  • User (Create, Update, Delete, Authorize)
  • Alarm Group (Create, Update, Delete)
  • Alarm Instance (Create, Update, Delete)
  • Worker Group (Create, Update, Delete)
  • Yarn Queue (Create, Update)
  • Environment (Create, Update, Delete)
  • Cluster (Create, Update, Delete)
  • K8s namespace (Create, Update)
  • Token (Create, Update, Delete)

4. Extract

We use mutili level of object like:

  • Level 1: Project.

  • Level 2: Workflow.

  • Level 3: Workflow Instance.

  • Level 1: Resource.

  • Level 2: Folder.

  • Level 3: File.

Demo:

  • Object Type
    • Project
    • Resource
    • Datasource
    • Security
  • Object Id/Enum
    • Workflow
    • Folder
  • Detail Id/Enum
    • Workflow instance
    • File

5. New Schema Design

Java Code Enum Design

    PROJECT(0, -1, "Project", true),
    RESOURCE(1,-1, "Resource", false),
    DATASOURCE(2,-1, "Datasource", true),
    SECURITY(3,-1, "Security", false),
    WORKFLOW(4,0, "Workflow", true),
    WORKFLOW_INSTANCE(5,4, "Workflow instance", true),
    WORKFLOW_INSTANCE1(6,5, "Workflow instance1", true);

    private final int code;
    private final int parentCode; // support multi level
    private final String name;
    private final boolean hasLogs; // if this object has not value, in search button, only can choose `All Sub-Levels logs`
    private int level; // project 0, workflow 1, workflow instance 2, task 2, task instance 3

6. UI design

Search field:

  • User

  • Object Type(Project, Security)

  • Scope(All Sub-Levels logs, current level logs). If Object Type hasLogs is false, only can choose All Sub-Levels logs

  • Object Name

  • Operation Type

User Name Parent Type Parent Name Object Type Object Name Operation Type Detail Time
admin PROJECT ds-test PROJECT ds-project Create v-project 2023-12-28 10:40:23
admin Security USER NewUser Delete 2023-12-28 10:40:23
admin PROJECT ds-test Workflow ds-workflow Create ds-workflow 2023-12-28 10:40:23
NewUser Workflow ds-workflow Workflow-instance Workflow-instance-1 Run 2023-12-28 10:40:23
NewUser PROJECT ds-test Workflow ds-workflow Update add new task... 2023-12-28 10:40:23

Database Design

CREATE TABLE `t_ds_audit_log` (
  `id` bigint(11) NOT NULL AUTO_INCREMENT COMMENT 'key',
  `user_id` int(11) NOT NULL COMMENT 'user id',
  `object_type` int(11) NOT NULL COMMENT 'resource type',
  `operation_id` int(11) NOT NULL COMMENT 'operation id',
  `operation_type` int(11) NOT NULL COMMENT 'operation type',
  `time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT 'create time',
  `detail` text DEFAULT NULL COMMENT 'detail',
  PRIMARY KEY (`id`) /*T![clustered_index] CLUSTERED */
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin AUTO_INCREMENT=1;

Are you willing to submit a PR?

  • [X] Yes I am willing to submit a PR!

Code of Conduct

qingwli avatar Jan 04 '24 06:01 qingwli

PTAL @caishunfeng @EricGao888 @Radeity @ruanwenjun @SbloodyS

qingwli avatar Jan 04 '24 08:01 qingwli

image Table data

qingwli avatar Feb 20 '24 07:02 qingwli

image Search data, you can multi choose object type

qingwli avatar Feb 20 '24 07:02 qingwli

image Search data about operation type

qingwli avatar Feb 20 '24 07:02 qingwli

image Search data about operation type

@qingwli if user deletes 2 process defintions in a single batch-delete operation, will there be 2 audit records in the table? or just 1 record with 2 objects

rickchengx avatar Feb 20 '24 07:02 rickchengx

image Search data about operation type

@qingwli if user deletes 2 process defintions in a single batch-delete operation, will there be 2 audit records in the table? or just 1 record with 2 objects

Will record in two lines

qingwli avatar Feb 20 '24 07:02 qingwli

image image

@rickchengx

qingwli avatar Feb 20 '24 08:02 qingwli

Compatibility, Deprecation, and Migration Plan

Compatible with current system, aspect all api method, but just create, delete, method will into record log logic, other query method will keep same. Just record log not change anything about api request.

It seems is not compatible with current system, since you need to change the schema in database.

ruanwenjun avatar Apr 01 '24 03:04 ruanwenjun

Compatibility, Deprecation, and Migration Plan

Compatible with current system, aspect all api method, but just create, delete, method will into record log logic, other query method will keep same. Just record log not change anything about api request.

It seems is not compatible with current system, since you need to change the schema in database.

Yes, I changed the schema, but this table is never used before, just defined with no data

qingwli avatar Apr 02 '24 02:04 qingwli