[Feature][seatunnel-server] Detail Design

[dijiekstra]

Search before asking

• [X] I had searched in the feature and found no similar feature requirement.

Description

Detail design

Do a simple prototype figure [archetype figure] (https://modao.cc/app/e0mCFHHqrcldsbvhrDOQyn).

Considering the time cycle, implementation cost and technical capability, this period only realizes part of the capabilities in Server&Scheduler, and only provides Api for use. (Mainly because I am not good at front-end, so welcome to have the ability to write front-end students to join)

Server

• maintenance

  • [ ] datasource
    • [ ] add
    • [ ] delete
    • [ ] update
    • [ ] list
  • [ ] auth
    • [ ] CRUD of role
  • [ ] user
    • [ ] add
    • [ ] update
    • [ ] delete
    • [ ] list

• development

  • [ ] task management
    • [ ] add
    • [ ] delete
    • [ ] update
    • [ ] list
    • [ ] execution
    • [ ] stop
    • [ ] view log

• maintenance

  • [ ] Batch instance mangement
    • [ ] exectuion panel
    • [ ] execution list
    • [ ] instance operation

The functions of the first phase are tentatively so much, and more practical functions will be added later.

Some details

• datasource
  • During configuration, can perform tests to check whether the configuration is correct
  • Jdbc-type data sources are tested through the 'JDBC API'; The remaining types are tested through the 'Java API'
• auth
  • The open source component 'Casbin' is imported to complete the permission management through the 'RBAC API' provided by it
  • Allows users or roles to be authorized
• user
  • Currently, can only manually CRUD. Integration with LDAP or others component will come soon.
• task management
  • Currently, only script mode is supported.
  • Navigation mode will be supported later (front-end support required)
• Batch Instance Management
  • Batch instances can only be maintenance in the first phase
  • Instances of the 'Streaming' type involve state consistency issues, so it's not going to be supported very quickly.

Table structure design

// todo . Supplement as soon

Scheduler

It mainly focuses on the development of general Scheduler-proxy and scheduler-engine-Embedded.

Scheduler-proxy

• [ ] submit
• [ ] delete
• [ ] suspend
• [ ] resume
• [ ] execution
• [ ] re-run
• [ ] stop
• [ ] complement data
• [ ] status query
• [ ] instance list
• [ ] temp exection
• [ ] log query

Scheduler-engine-embedded

It is mainly implemented against the interface defined in scheduler-Proxy, which is not a service but an interface implementation. Complete the execution and maintenance of tasks with 'spring-scheduling', 'Java Process' and database capabilities;

详细设计

做了个简单的原型图，参考原型图。

考虑到时间周期、实现成本、技术能力，本期只实现Server&Scheduler中的部分能力，仅提供Api的方式供使用。(主要是我不太会前端，所以欢迎有能力写前端的同学一起加入)

Server

• maintenance

  • [ ] 数据源
    • [ ] 新增数据源
    • [ ] 删除数据源
    • [ ] 修改数据源
    • [ ] 数据源列表
  • [ ] 权限管理
    • [ ] 角色列表
    • [ ] 权限列表
  • [ ] 用户管理
    • [ ] 新增用户
    • [ ] 修改用户
    • [ ] 删除用户
    • [ ] 用户列表

• development

  • [ ] 任务管理
    • [ ] 新增任务
    • [ ] 删除任务
    • [ ] 修改任务
    • [ ] 任务列表
    • [ ] 运行任务
    • [ ] 停止任务
    • [ ] 查看日志

• maintenance

  • [ ] Batch实例管理
    • [ ] 运行大盘
    • [ ] 执行流水
    • [ ] 实例操作

第一期的功能暂定就这么多，后续会加入更多实用的功能。

部分细节

• 数据源
  • 在进行配置时，可以通过进行测试来检测配置是否正确
  • JDBC类型数据源通过jdbc api进行测试；其余类型通过java api进行测试
• 权限管理
  • 引入开源组件casbin，通过其提供的rbac api完成权限管理
  • 允许对用户或角色进行授权
• 用户管理
  • 目前只支持手动进行增删改，后续将继承LDAP等组件
• 任务管理
  • 目前只支持脚本模式，提交seatunnel脚本
  • 后续将支持向导模式(需要前端支持)
• Batch实例管理
  • 第一期只支持对Batch类型的实例进行运维及管理
  • Streaming类型的实例牵扯到状态一致性的问题

表结构设计

// todo 后续补充

Scheduler

主要是通用Scheduler-proxy的开发和Scheduler-engine-embedded的开发。

Scheduler-proxy

• [ ] 提交任务
• [ ] 删除任务
• [ ] 暂停任务
• [x] 恢复任务
• [ ] 任务执行
• [ ] 任务重跑
• [ ] 任务停止
• [ ] 任务补数
• [ ] 任务状态查询
• [ ] 任务实例列表
• [ ] 任务临时执行
• [ ] 任务日志查询

Scheduler-engine-embedded

主要是针对Scheduler-proxy中定义的接口进行实现，并非是一个服务，而是接口实现；利用spring-scheduling、java process、数据库等能力完成任务的运行与管理；

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[EricJoy2048]

CC @CalvinKirs @BenJFan @ruanwenjun @ashulin

[zyd915]

太酷了 社区人才济济 一直在关注seatunnel的web项目 @dijiekstra 干得漂亮

[EricJoy2048]

太酷了 社区人才济济 一直在关注seatunnel的web项目 @dijiekstra 干得漂亮

Thanks, Are you interested in contributing to this feature？

[xuhui1231]

Looks good,At present, we are also preparing to open a second platform based on Seatunnel, and hope to communicate more together.

[zyd915]

太酷了 社区人才济济 一直在关注seatunnel的web项目 @dijiekstra 干得漂亮

Thanks, Are you interested in contributing to this feature？

我主攻后端，改了很多seatunnel后端源码增加了一些新特性支持 抽时间整理贡献pr给官方库 前端的不是我所擅长的方向

[xuhui1231]

这就尴尬了,,,我也是平台开发方向,,,前端没咋接触过,偶尔画画原型图...如果有需要的话我们可以一起做

------------------ 原始邮件 ------------------ 发件人: "apache/incubator-seatunnel" @.>; 发送时间: 2022年6月20日(星期一) 中午1:54 @.>; @.@.>; 主题: Re: [apache/incubator-seatunnel] [Feature][seatunnel-server] Detail Design (Issue #1969)

太酷了 社区人才济济 一直在关注seatunnel的web项目 @dijiekstra 干得漂亮

Thanks, Are you interested in contributing to this feature？

我主攻后端，改了很多seatunnel后端源码增加了一些新特性支持 抽时间整理贡献pr给官方库 前端的不是我所擅长的方向

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[zyd915]

这就尴尬了,,,我也是平台开发方向,,,前端没咋接触过,偶尔画画原型图...如果有需要的话我们可以一起做 … ------------------ 原始邮件 ------------------ 发件人: "apache/incubator-seatunnel" @.>; 发送时间: 2022年6月20日(星期一) 中午1:54 @.>; @.@.>; 主题: Re: [apache/incubator-seatunnel] [Feature][seatunnel-server] Detail Design (Issue #1969) 太酷了 社区人才济济 一直在关注seatunnel的web项目 @dijiekstra 干得漂亮 Thanks, Are you interested in contributing to this feature？ 我主攻后端，改了很多seatunnel后端源码增加了一些新特性支持 抽时间整理贡献pr给官方库 前端的不是我所擅长的方向 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

好的

[xiaochaohit]

Looks great, looking forward to the feature launch