[STORM-3404]KafkaOffsetLagUtil cant pull the offset correctly

[wxy929629]

when use SASL_PLAIN kafka JAAS auth, missing sasl.mechanism will lead to KafkaOffsetLagUtil cant pull the offset correctly kafka version 2.2.0

Apache JIRA url: https://issues.apache.org/jira/browse/STORM-3404

[wxy929629]

@srdo I have already change the target to the master branch.

[srdo]

@wxy929629 Thanks. Could you fix the checkstyle issues that have been introduced? The build output has the following lines

[WARN] /home/travis/build/apache/storm/external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java:147:86: WhitespaceAround: '{' is not preceded with whitespace. [WhitespaceAround]

[WARN] /home/travis/build/apache/storm/external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java:149: 'if rcurly' has incorrect indentation level 14, expected level should be 16. [Indentation]

I.e. there's some indentation that needs to be fixed in the lines you added.

[wxy929629]

@wxy929629 Thanks. Could you fix the checkstyle issues that have been introduced? The build output has the following lines

[WARN] /home/travis/build/apache/storm/external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java:147:86: WhitespaceAround: '{' is not preceded with whitespace. [WhitespaceAround]

[WARN] /home/travis/build/apache/storm/external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java:149: 'if rcurly' has incorrect indentation level 14, expected level should be 16. [Indentation]

I.e. there's some indentation that needs to be fixed in the lines you added.

@srdo

Ok i have done this

[Ethanlm]

@wxy929629 Thanks for contributing!

Can you please help me understand this problem better?

Do you mean when Kafka broker is set up with SASL_PLAIN, missing sasl.mechanism will lead to failures (by default, sasl.mechanism is GSSAPI)? If in that case, does it make sense to provide an option for users to choose sasl.mechanism, just like what we have for security.protocol ? https://github.com/apache/storm/blob/master/external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java#L118

Can you please explain more on what happened, by providing stack trace and logs if possible? Also the jira indicates it is tested on storm 1.2.2. Can you provide the logs and stack trace from the testing from storm master branch? Although I understand this is mostly about kafka client setup, not very related to storm version, but it is still nice to see since this PR is based on storm master branch. It will help reviewers and other storm users to understand this better.

Also did you test this with Kafka-1.x and make sure it still works?

Thanks

[wxy929629]

@wxy929629 Thanks for contributing!

Can you please help me understand this problem better?

Do you mean when Kafka broker is set up with SASL_PLAIN, missing sasl.mechanism will lead to failures (by default, sasl.mechanism is GSSAPI)? If in that case, does it make sense to provide an option for users to choose sasl.mechanism, just like what we have for security.protocol ? https://github.com/apache/storm/blob/master/external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java#L118

Can you please explain more on what happened, by providing stack trace and logs if possible? Also the jira indicates it is tested on storm 1.2.2. Can you provide the logs and stack trace from the testing from storm master branch? Although I understand this is mostly about kafka client setup, not very related to storm version, but it is still nice to see since this PR is based on storm master branch. It will help reviewers and other storm users to understand this better.

Also did you test this with Kafka-1.x and make sure it still works?

Thanks

Yes, when i set the SASL_PLAIN to connect to the kafka, i can not pull the lag info correctly. So i need to add a property sasl.mechanism=PLAIN, i provide an option for users to choose sasl.mechanism to this PR, please review it

Because my test environment is not ready any more, so i perhaps can't provide the log and trace for what happend.But i think the failures is easy to reappearance on any kafka brokers which setup SASL_PLAIN