logging-flume
logging-flume copied to clipboard
apache
[FLUME-3095]add capacity unit support in configuration file
FLUME-3095 By the way I want to ask, how to modify the User Guide Documentation
Now only the capacity type of configuration with the unit, because the capacity type of the configuration unit is unified by default, can be compatible with the previous use without units, but the time type of the configuration unit is not uniform, there are milliseconds and seconds, if added Unit, then it will not be compatible with the previous use without units, do not know what advice you have? In addition, I need to update the User Guide Documentation, where is the update?
First of all thank you for your patch, @iaiengxia, I think it tackles an issue which might be often painful for our users. But I'd suggest to use a more robust solution, e.g. config by typesafehub. It has getDuration() and getBytes() methods. If we used the already existing library we wouldn't need to have our own implementation, which, even if we had tests for it, would be very error prone.
What do you think?
For the time unit: the above mentioned getDuration() interprets the numbers without any unit as milliseconds, so for cases where Flume expects it to be in seconds we should first try to parse it as number (or explicitly check whether there is no unit set) and if it fails then use the getDuration() and convert the returned milliseconds to seconds.
The user guide can be found here: https://github.com/apache/flume/blob/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst To render the user guide to html to check your changes use the following commands:
export LC_ALL=C.UTF-8 # Required to build the javadocs on some platforms and in some locales
mvn clean install -Psite -DskipTests
The compiled file will be placed in the target/site directory.
To be honest, before I write the tool class, I did not find a library that could meet my needs. Thank you for increasing my knowledge @adenes . I read the source code of config.jar and read the development documentation and did some basic Of the test, now listed under the advantages and disadvantages I know: Advantages of using config.jar: The solution is more mature and does not need to test the parsing parameters part of the code
Disadvantages of using config.jar: There is no separate way to directly parse a string with a unit, you must reload the configuration file, the change is relatively large, not fully compatible with the original program Can only parse a number and a unit of the parameters, such as 1g500m this with a number of units can not be resolved (do not know if there is such a demand)
Advantages of using custom tool classes to parse: You can parse parameters with multiple units Easy to use, the change is very small (read the time through a static method can be processed), and has been achieved
Disadvantages of using custom tool classes to parse: Requires complete testing
What do you think?
For the time unit, I suggest that some of the recommendations, if I provided in the Context class getDuration () method used to resolve the time, if no unit does not need to parse the direct return number, there are units I will analyze the actual unit , Internally stored in milliseconds, but what number of units do I return?
For example, the default unit in milliseconds, timeout = 3000, I will return 3000, timeout = 3s (internal resolution into 3000ms), I will return 3000 (return the number of units in milliseconds).But in the default unit for the case of seconds, timeout = 3, I will return 3, timeout = 3s, How much should I return , if according to the above logic I will still return 3000, but the user expects to return the unit for seconds, How do I know the user's expectations unit?
Perhaps you can use the getDuration (String key, TimeUnit unit) method to let the user tell me what unit it needs to return value, but this will lead to another problem, if the default unit is seconds, timeout = 3, I can not know 3 real units and convert them into units that the user expects, because no unit can not simply return the number, and need to be converted into the unit desired by the user.
It seems a bit difficult to analyze the time unit, because I need to meet the needs of existing components for time parameter resolution. I can not provide a common way to meet all the needs of the present situation. Based on the current situation, my suggestion is , No unit number directly return the number, there are units according to the actual unit analysis, and unified return in milliseconds as a unit of the number, if the user needs seconds, then their own conversion (really, the user needs for the second unit is not the beginning support time units, resulting in too long settings to read up is not convenient)...
