[Proposal][Feature] Add variable management module

[muchunjin]

Search before asking

• [X] I had searched in the feature and found no similar feature requirement.

Background

In the actual production environment, Flink jobs are generally complex, and usually require multiple external components. For example, Flink jobs consume data from Kafka, then connect external components such as HBase or Redis to obtain additional business information, and then write it to the downstream external components. There are the following problems:

1. The connection information of external components, such as IP, port and user password, needs to be configured in the application args and transferred to the Flink job, so that the connection information of external components is distributed in multiple applications. Once the connection information of external components changes, many application args parameters need to be modified, which will lead to high operation and maintenance costs.
2. There are many people in the team developing real-time computing jobs. There is no uniform specification for the connection information of the external components passed to the job, resulting in different parameter names of the same component. This is difficult to count which external components are dependent.
3. In production practice, there are usually multiple sets of environments, such as test environment and production environment. It is not intuitive to judge whether a component belongs to a test environment or a production environment simply through IP and ports. Even if it can be judged, there are some omissions. In this way, the connection information online to the production environment may be external components of the test environment, or components of the production environment are inadvertently configured during testing, leading to production failures.

Proposal

1. Create a unified module to manage variables. Users can define their own variables in this module. Variable information includes variable code (passed to Flink program as parameter name), variable value, variable name, variable description, and which applications depend on
2. In the same Team module, the variable code or variable name is unique

• By using placeholders, such as adding -- kafka {mykafka} to the args of the application, mykafka is the variable code of a variable
• Search by variable code or variable name. After a variable is selected, the parameter transfer method is displayed

you can check the thread

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[wolfboys]

This is a great feature, very useful, priority will be given to the plan, you are welcome to contribute this feature.

[muchunjin]

Here I summarize the functions to be completed in version 1.2.4.

1.The table structure of the variable contains the following main contents.

• variable code is placeholder, For example the variable code is "collect.kafka.cluster", it will be referenced as ${collect.kafka.cluster}, it is unique within a team.
• variable name is removed, after discussion, it is not necessarily retained.
• variable value is the value of a specific variable, such as ip and port.
• description is a detailed introduction.
• team is the team the variable belongs to.

2.Basic functions of variable management.

• Add and modify variables: need to check whether the variable code is unique under the current team.
• Deleting a variable: It is necessary to check whether it has been referenced by the Application under the team to which the variable belongs. If there is metadata management in the future, it is also necessary to check whether it has been referenced by the metadata. If it is referenced, it will prompt that the variable is referenced and cannot be deleted.

3.References in Flink sql and program args.

• In the application, the variables are uniformly referenced in the form of placeholders ${xxx}, such as in Flink sql, program args.
• For a better experience, it is necessary to provide the function of following the search prompt when entering variable placeholders. For example, when entering "${", the variable list is displayed, and when "${kafka" is entered, the display contains variables for kafka strings, in "variable code" or "description".
• When the focus is lost, it is judged whether there is a variable placeholder, and if so, a verification button is displayed to prompt the user to perform verification. When the user clicks the verification button, the replaced content of the placeholder is displayed, and only the user clicks the verification button to click the submit button.
• If the user enters "okir23${j2Rts}pEt", this placeholder "${j2Rts}" will not be replaced if "j2Rts" is not a variable we defined.
• If I copy some sql from a old application, and the sql includes some placeholders, these placeholders will take effect in the new application.

4.Future plan

• Display the application or metadata list of a referenced variable in the variable basic management module.
• Improve user experience.

[1996fanrui]

Hi @macksonmu , thanks for your great work~