trafficstars

Purpose of the pull request

close https://github.com/apache/dolphinscheduler/issues/10495

Brief change log

In DB, I created a table named t_ds_relation_resources_task containing an auto-incremented id, fullName and type, which can be found in dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql . A new field named resource_ids_new is added to the table Task Definition, TaskDefinitionLog to record ids in table t_ds_relation_resources_task. To test the resource center, we need to execute table creation and field update SQL, which can be found in the dolphinscheduler-dao/src/main/resources/sql/upgrade/3.1.0_schema/mysql/dolphinscheduler_ddl.sql.

Most of the changes are in dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ResourcesService.java, dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ResourcesController.java, dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/HadoopUtils.java and dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java. The original Ids of resources files were all replaced with their unique path (fullName). In the parameter, notice fileName is different from fullName. fullName is in the format of "bucketName/TenantCode/resources(udfs)/fileName" fileName is just the last part of the fullName. resTenantCode is the tenant code of the resource. It is used for the permission check where the current login user (ordinary user) can not read resource files if they are not the correct tenant. the scenario is possible because tenantCode is part of the URI.

In the frontEnd. To access resources with the right permission. all URL path has a query pattern of "fullName" and "tenantCode" E.g. prefix=dolphinscheduler-test/tenant1/ds/resources/testDDD1/&tenantCode=tenant1/ds In this example. fullName and tenantCode are the fullName and resTenantCode mentioned above.

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

Sep 21 '22 01:09 EricPyZhou

:warning: This PR do not change database DDL synchronize.

Oct 12 '22 15:10 mergeable[bot]

Codecov Report

Merging #12076 (625a00b) into dev (233bbdb) will decrease coverage by 0.70%. The diff coverage is 14.17%.

:exclamation: Current head 625a00b differs from pull request most recent head 79fcf11. Consider uploading reports for the commit 79fcf11 to get more accurate results

@@             Coverage Diff              @@
##                dev   #12076      +/-   ##
============================================
- Coverage     39.61%   38.91%   -0.71%     
+ Complexity     4186     4152      -34     
============================================
  Files          1038     1039       +1     
  Lines         38843    39225     +382     
  Branches       4449     4477      +28     
============================================
- Hits          15389    15263     -126     
- Misses        21711    22219     +508     
  Partials       1743     1743

Impacted Files	Coverage Δ
...scheduler/api/dto/resources/ResourceComponent.java	`25.00% <0.00%> (-6.25%)`	:arrow_down:
...heduler/api/service/impl/ResourcesServiceImpl.java	`37.73% <ø> (-14.23%)`	:arrow_down:
...inscheduler/api/service/impl/UsersServiceImpl.java	`70.51% <0.00%> (ø)`
...che/dolphinscheduler/dao/entity/ResourcesTask.java	`0.00% <0.00%> (ø)`
...uler/dao/utils/ResourceProcessDefinitionUtils.java	`40.00% <0.00%> (-37.78%)`	:arrow_down:
...r/server/master/runner/task/BaseTaskProcessor.java	`16.41% <0.00%> (+0.60%)`	:arrow_up:
...lphinscheduler/service/storage/StorageOperate.java	`100.00% <ø> (ø)`
...hinscheduler/service/storage/impl/OssOperator.java	`32.80% <0.00%> (-1.37%)`	:arrow_down:
...dolphinscheduler/service/storage/impl/S3Utils.java	`0.00% <0.00%> (ø)`
...server/worker/utils/TaskExecutionCheckerUtils.java	`8.00% <0.00%> (ø)`
... and 34 more