flyteorg
[FlyteCTL] enable kubectl-like configuration from multiple files
Tracking issue
https://github.com/flyteorg/flyte/issues/6237
Why are the changes needed?
Currently, configuring over multiple projects / domains can be painful as multiple commands need to be used for each setting.
What changes were proposed in this pull request?
A kubectl like config method is proposed, which enable providing multiple config simultaneously through flytectl update --attrFile config.yml. config.yml include configs from different subcommands specified by kind keyword (see example below). Multiple config file is also supported here by using flytectl update --attrFile config.yml --attrFile config1.yml.
---
kind: project
name: protein-design
id: protein-design
---
kind: workflow-execution-config
domain: development
project: protein-design
attributes:
projectQuotaCpu: "1000"
projectQuotaMemory: 10Ti
workflow_execution_config:
security_context:
run_as: [email protected]
How was this patch tested?
Unit test
Labels
Please add one or more of the following labels to categorize your PR:
- added: For new features.
- changed: For changes in existing functionality.
- deprecated: For soon-to-be-removed features.
- removed: For features being removed.
- fixed: For any bug fixed.
- security: In case of vulnerabilities
This is important to improve the readability of release notes.
Setup process
Screenshots
Check all the applicable boxes
- [x] I updated the documentation accordingly.
- [x] All new and existing tests passed.
- [x] All commits are signed-off.
Related PRs
Docs link
https://flyte--6292.org.readthedocs.build/en/6292/api/flytectl/gen/flytectl_update.html
Summary by Bito
This PR enhances flytectl's workflow execution by adding kubectl-like configuration support for multiple attribute files individually or via a directory. It modernizes file operations by replacing deprecated packages, refactors the configuration structure, and improves error handling. The changes include documentation updates reflecting these new capabilities and extensive updates to the monodocs-environment.lock.yaml file with dependency version changes.Unit tests added: True
Estimated effort to review (1-5, lower is better): 5
Code Review Agent Run #38bf67
Actionable Suggestions - 1
-
flytectl/cmd/config/subcommand/workflowexecutionconfig/attrupdateconfig_flags.go - 1
- Type mismatch between flag and struct definition · Line 53-53
Review Details
-
Files reviewed - 5 · Commit Range:
b8a0b09..2f38c33- flytectl/cmd/config/subcommand/workflowexecutionconfig/attrupdateconfig_flags.go
- flytectl/cmd/config/subcommand/workflowexecutionconfig/update_config.go
- flytectl/cmd/testutils/test_utils.go
- flytectl/cmd/update/matchable_workflow_execution_config.go
- flytectl/cmd/update/matchable_workflow_execution_config_test.go
-
Files skipped - 0
-
Tools
- Golangci-lint (Linter) - ✖︎ Failed
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
AI Code Review powered by 
Codecov Report
Attention: Patch coverage is 50.34965% with 71 lines in your changes missing coverage. Please review.
Project coverage is 58.44%. Comparing base (
ff7e79f) to head (65fe6b2). Report is 27 commits behind head on master.
Additional details and impacted files
@@ Coverage Diff @@
## master #6292 +/- ##
===========================================
+ Coverage 33.82% 58.44% +24.62%
===========================================
Files 1329 938 -391
Lines 147808 71200 -76608
===========================================
- Hits 49992 41614 -8378
+ Misses 92974 26418 -66556
+ Partials 4842 3168 -1674
| Flag | Coverage Δ | |
|---|---|---|
| unittests-datacatalog | 59.03% <ø> (+11.02%) |
:arrow_up: |
| unittests-flyteadmin | 56.30% <ø> (+6.20%) |
:arrow_up: |
| unittests-flytecopilot | 30.99% <ø> (ø) |
|
| unittests-flytectl | 64.10% <50.34%> (+6.00%) |
:arrow_up: |
| unittests-flyteidl | 76.12% <ø> (+69.33%) |
:arrow_up: |
| unittests-flyteplugins | 61.00% <ø> (+11.98%) |
:arrow_up: |
| unittests-flytepropeller | 54.82% <ø> (+18.29%) |
:arrow_up: |
| unittests-flytestdlib | 64.04% <ø> (+13.66%) |
:arrow_up: |
Flags with carried forward coverage won't be shown. Click here to find out more.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
🚀 New features to boost your workflow:
- ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
Changelist by Bito
This pull request implements the following key changes.
| Key Change | Files Impacted |
|---|---|
| New Feature - Enhanced Multi-File Configuration for Update Commands |
- - |
| Bug Fix - Refined Attribute File Validation in Update Commands |
- - - - - - |
| Feature Improvement - Improved Configuration Defaults and Flag Parsing |
- - - - - - - |
| Testing - Extended Test Coverage for New and Updated Features |
- - - |
| Documentation - Updated Documentation for Enhanced Configuration |
- - - - - |
| Other Improvements - Dependency Updates in Environment Lock File |
- |
Code Review Agent Run #7699d3
Actionable Suggestions - 0
Review Details
-
Files reviewed - 2 · Commit Range:
2f38c33..f43ae50- flytectl/cmd/config/subcommand/workflowexecutionconfig/attrupdateconfig_flags.go
- flytectl/cmd/config/subcommand/workflowexecutionconfig/attrupdateconfig_flags_test.go
-
Files skipped - 0
-
Tools
- Golangci-lint (Linter) - ✖︎ Failed
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
AI Code Review powered by
Code Review Agent Run #988f93
Actionable Suggestions - 1
-
flytectl/cmd/config/attrupdateconfig_flags.go - 1
- Possible pointer dereferencing issue in elemValueOrNil · Line 22-22
Review Details
-
Files reviewed - 10 · Commit Range:
f43ae50..62def7b- flytectl/cmd/config/attrupdateconfig_flags.go
- flytectl/cmd/config/attrupdateconfig_flags_test.go
- flytectl/cmd/config/subcommand/workflowexecutionconfig/attrupdateconfig_flags.go
- flytectl/cmd/config/subcommand/workflowexecutionconfig/attrupdateconfig_flags_test.go
- flytectl/cmd/config/subcommand/workflowexecutionconfig/update_config.go
- flytectl/cmd/config/update_config.go
- flytectl/cmd/update/interfaces/updater.go
- flytectl/cmd/update/matchable_workflow_execution_config.go
- flytectl/cmd/update/matchable_workflow_execution_config_test.go
- flytectl/cmd/update/update.go
-
Files skipped - 0
-
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
AI Code Review powered by
Code Review Agent Run #ea5294
Actionable Suggestions - 1
-
flytectl/cmd/update/matchable_cluster_resource_attribute.go - 1
- Consider more explicit struct emptiness check · Line 67-68
Filtered by Review Rules
Bito filtered these suggestions based on rules created automatically for your feedback. Manage rules.
-
flytectl/cmd/update/update.go - 1
- Consider flag existence validation · Line 153-157
Review Details
-
Files reviewed - 13 · Commit Range:
62def7b..f4d4a28- flytectl/cmd/config/subcommand/clusterresourceattribute/file_config.go
- flytectl/cmd/config/subcommand/executionclusterlabel/file_config.go
- flytectl/cmd/config/subcommand/executionqueueattribute/file_config.go
- flytectl/cmd/config/subcommand/plugin_override/file_config.go
- flytectl/cmd/config/subcommand/taskresourceattribute/file_config.go
- flytectl/cmd/config/subcommand/workflowexecutionconfig/file_config.go
- flytectl/cmd/update/matchable_cluster_resource_attribute.go
- flytectl/cmd/update/matchable_execution_cluster_label.go
- flytectl/cmd/update/matchable_execution_queue_attribute.go
- flytectl/cmd/update/matchable_plugin_override.go
- flytectl/cmd/update/matchable_task_resource_attribute.go
- flytectl/cmd/update/matchable_workflow_execution_config.go
- flytectl/cmd/update/update.go
-
Files skipped - 0
-
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
Bito Usage Guide
Commands
Type the following command in the pull request comment and save the comment.
/review- Manually triggers a full AI review.
Refer to the documentation for additional commands.
Configuration
This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].
Documentation & Help
AI Code Review powered by
Code Review Agent Run #c58d0b
Actionable Suggestions - 0
Review Details
-
Files reviewed - 7 · Commit Range:
f4d4a28..b3a5500- flytectl/cmd/update/interfaces/updater.go
- flytectl/cmd/update/matchable_execution_cluster_label.go
- flytectl/cmd/update/matchable_execution_queue_attribute.go
- flytectl/cmd/update/matchable_plugin_override.go
- flytectl/cmd/update/matchable_task_resource_attribute.go
- flytectl/cmd/update/matchable_workflow_execution_config.go
- flytectl/cmd/update/update.go
-
Files skipped - 0
-
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
Bito Usage Guide
Commands
Type the following command in the pull request comment and save the comment.
/review- Manually triggers a full AI review.
Refer to the documentation for additional commands.
Configuration
This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].
Documentation & Help
AI Code Review powered by
Code Review Agent Run #85577c
Actionable Suggestions - 0
Review Details
-
Files reviewed - 2 · Commit Range:
b3a5500..1f4a819- flytectl/cmd/update/update.go
- flytectl/cmd/update/update_test.go
-
Files skipped - 0
-
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
Bito Usage Guide
Commands
Type the following command in the pull request comment and save the comment.
/review- Manually triggers a full AI review.
Refer to the documentation for additional commands.
Configuration
This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].
Documentation & Help
AI Code Review powered by
Code Review Agent Run #d139b4
Actionable Suggestions - 0
Review Details
-
Files reviewed - 2 · Commit Range:
1f4a819..a48b410- flytectl/cmd/update/update.go
- flytectl/cmd/update/update_test.go
-
Files skipped - 0
-
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
Bito Usage Guide
Commands
Type the following command in the pull request comment and save the comment.
/review- Manually triggers a full AI review.
Refer to the documentation for additional commands.
Configuration
This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].
Documentation & Help
AI Code Review powered by
Code Review Agent Run #66e219
Actionable Suggestions - 19
-
monodocs-environment.lock.yaml - 19
- Possible unintended dependency replacement · Line 2444-2444
- Potential security risk in requests downgrade · Line 4050-4050
- Consider keeping urllib3 dependency · Line 4459-4461
- Consider keeping both dependencies · Line 5007-5007
- Inconsistent protobuf version constraints between platforms · Line 6461-6461
- Dependency removed but still used · Line 6685-6685
- Consider keeping required dependencies · Line 6803-6803
- Missing dependency in platform configuration · Line 8132-8132
- Consider keeping stack_data dependency · Line 8771-8771
- Consider keeping both dependencies instead of replacing · Line 9288-9288
- Consider keeping uri-template dependency · Line 9448-9448
- Consider adding instead of replacing dependency · Line 9559-9559
- Consider keeping both dependencies · Line 10281-10281
- Version mismatch in package URL · Line 15425-15425
- Possible unintended dependency replacement · Line 24314-24314
- Consider keeping both dependencies instead of replacing · Line 15975-15975
- Possible unintended dependency replacement · Line 16503-16503
- Consider Python version compatibility implications · Line 16702-16704
- Missing dependencies in pyspark package URL update · Line 19701-19701
Additional Suggestions - 10
-
monodocs-environment.lock.yaml - 10
- Consider security implications of relaxed dependencies · Line 4001-4002
- Consider aligning HDF5 version requirements · Line 7855-7855
- Consider impact of pydantic version constraint · Line 14872-14872
- Consider consistent dependency version requirements · Line 15422-15422
- Possible inconsistent package naming convention · Line 16782-16783
- Consider protobuf version constraint implications · Line 18476-18476
- Consider adding Python version constraint · Line 20992-20992
- Consider updating fsspec version consistently · Line 28494-28494
- Consider keeping consistent dependency requirements · Line 8646-8647
- Consider dependency version compatibility implications · Line 20644-20644
Review Details
-
Files reviewed - 7 · Commit Range:
a48b410..65fe6b2- flytectl/cmd/update/update.go
- flytectl/docs/source/gen/flytectl_demo_reload.rst
- flytectl/docs/source/gen/flytectl_demo_start.rst
- flytectl/docs/source/gen/flytectl_sandbox_start.rst
- flytectl/docs/source/gen/flytectl_update.rst
- flytectl/docs/source/gen/flytectl_update_workflow-execution-config.rst
- monodocs-environment.lock.yaml
-
Files skipped - 0
-
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
Bito Usage Guide
Commands
Type the following command in the pull request comment and save the comment.
/review- Manually triggers a full AI review.
Refer to the documentation for additional commands.
Configuration
This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].
Documentation & Help
AI Code Review powered by
What's the next steps for this? I'm in the process of building out our Flyte management infrastructure and would really love this feature.
@machichima do you still want to land this? If so I can take a look and we can merge this.
@machichima do you still want to land this? If so I can take a look and we can merge this.
Yes please! Please let me know if anything needs to be improved