Updated hadoop version from 2.6.0 to 3.1.0

[abossert]

As discussed on the dev mail list, I have upgraded Hadoop from 2.6.0 to 3.1.0. The build is successful, but no other functionality or incorporation of new Hadoop 3.1 features yet...

[sanjaypujare]

Both the Jenkins and Travis builds/tests have failed. Could you check the failures and see if they are related to these changes?

[abossert]

Both the Jenkins and Travis builds/tests have failed. Could you check the failures and see if they are related to these changes?

Yeah, I was just looking into that. I reverted all my changes, so I have a vanilla Master branch...the build fails there too with the tests:

 <failure message="Version number pattern does not match: Unknown" type="java.lang.AssertionError">java.lang.AssertionError: Version number pattern does not match: Unknown
	at com.datatorrent.stram.util.VersionInfoTest.testMavenProperties(VersionInfoTest.java:64)
</failure>

[sanjaypujare]

If this is the same failure you see in your PR CI tests then that's good news. Not sure why we are getting "Unknown" from the VersionInfo object but it might be worthwhile debugging and fixing that in a separate PR.

[vrozov]

@abossert Please open JIRA and follow "Opening Pull Requests" guidelines (see http://apex.apache.org/contributing.html)

[sanjaypujare]

@abossert just checking. Hopefully you are not stuck on anything and making progress on this PR.

[abossert]

Sorry, have been bed ridden for a day with a nasty flu and swamped with work...I am still not entirely sure where to look for the version error that needs to be fixed (prevents initial compile and test from running prior to any Hadoop version change)...digging into it today, but would appreciate any thoughts you might have?

Sent from my iPhone

On Jan 23, 2019, at 23:53, sanjaypujare [email protected] wrote:

@abossert just checking. Hopefully you are not stuck on anything and making progress on this PR.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[sanjaypujare]

@tweise added that test (com.datatorrent.stram.util.VersionInfoTest.testMavenProperties). Any ideas?

[abossert]

OK, so in com.datatorrent.stram.util.VersionInfo on line 61, it is not getting populated and is not producing any error either, but rather staying with the default values of Unknown:

Enumeration<URL> resources = classInJar.getClassLoader().getResources("META-INF/maven/" + groupId + "/" + artifactId + "/pom.properties");

On Thu, Jan 24, 2019 at 12:11 PM sanjaypujare [email protected] wrote:

@tweise https://github.com/tweise added that test (com.datatorrent.stram.util.VersionInfoTest.testMavenProperties). Any ideas?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache/apex-core/pull/607#issuecomment-457278052, or mute the thread https://github.com/notifications/unsubscribe-auth/AYuy1DZUdFYG0uWU61iTlmSwXZOjWbdyks5vGek5gaJpZM4aCUt- .

--

M. Aaron Bossert (202) 594-1703 Punch Cyber Analytics Group

[tweise]

@abossert looks like a different error than what is shown in the CI. How are you running the tests?

Try mvn clean install if that's not what you are doing already.

[abossert]

I am at a loss. Tried completely restarting and all seems fine now. Got the initial build to work pre-upgrade to hadoop 3.1.0:

[DEBUG] Using catalog /Users/mbossert/.m2/archetype-catalog.xml [DEBUG] Reading the catalog /Users/mbossert/.m2/archetype-catalog.xml [INFO]

[INFO] Reactor Summary: [INFO] [INFO] Apache Apex ........................................ SUCCESS [ 2.744 s] [INFO] Apache Apex API .................................... SUCCESS [ 8.776 s] [INFO] Apache Apex Common Library ......................... SUCCESS [ 6.235 s] [INFO] Apache Apex Buffer Server .......................... SUCCESS [ 4.226 s] [INFO] Apache Apex Stream Processing Engine ............... SUCCESS [08:27 min] [INFO] Apache Apex Application Maven Archetype ............ SUCCESS [ 1.977 s] [INFO] Apache Apex App Configuration Maven Archetype ...... SUCCESS [ 0.613 s] [INFO]

[INFO] BUILD SUCCESS [INFO]

[INFO] Total time: 08:52 min [INFO] Finished at: 2019-01-24T16:16:35-05:00 [INFO] Final Memory: 100M/1516M [INFO]

Process finished with exit code 0

On Thu, Jan 24, 2019 at 2:57 PM Thomas Weise [email protected] wrote:

@abossert https://github.com/abossert looks like a different error than what is shown in the CI. How are you running the tests?

Try mvn clean install if that's not what you are doing already.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache/apex-core/pull/607#issuecomment-457334953, or mute the thread https://github.com/notifications/unsubscribe-auth/AYuy1EmSUgjp__Qycyq77RnAEEB3SHDuks5vGhAegaJpZM4aCUt- .

--

M. Aaron Bossert (202) 594-1703 Punch Cyber Analytics Group

[abossert]

Another question...I have been able to validate that all of the constant values that were causing build errors have been replaced with the appropriate replacements from Hadoop 2.6.1 -> 3.1.0, however the two following ones just show up as deprecated with no clue about a replacement. I found a reference to them in Hadoop 2.5 source code...but not sure where that would leave us...can anyone chime in on background for how these variables are used in Apex?

@Deprecated /** Only used by HttpServer. / public static final String HADOOP_SSL_ENABLED_KEY = "hadoop.ssl.enabled"; @Deprecated /* Only used by HttpServer. */ public static final boolean HADOOP_SSL_ENABLED_DEFAULT = false;

On Thu, Jan 24, 2019 at 4:18 PM Aaron Bossert [email protected] wrote:

I am at a loss. Tried completely restarting and all seems fine now. Got the initial build to work pre-upgrade to hadoop 3.1.0:

[DEBUG] Using catalog /Users/mbossert/.m2/archetype-catalog.xml [DEBUG] Reading the catalog /Users/mbossert/.m2/archetype-catalog.xml [INFO]

[INFO] Reactor Summary: [INFO] [INFO] Apache Apex ........................................ SUCCESS [ 2.744 s] [INFO] Apache Apex API .................................... SUCCESS [ 8.776 s] [INFO] Apache Apex Common Library ......................... SUCCESS [ 6.235 s] [INFO] Apache Apex Buffer Server .......................... SUCCESS [ 4.226 s] [INFO] Apache Apex Stream Processing Engine ............... SUCCESS [08:27 min] [INFO] Apache Apex Application Maven Archetype ............ SUCCESS [ 1.977 s] [INFO] Apache Apex App Configuration Maven Archetype ...... SUCCESS [ 0.613 s] [INFO]

[INFO] BUILD SUCCESS [INFO]

[INFO] Total time: 08:52 min [INFO] Finished at: 2019-01-24T16:16:35-05:00 [INFO] Final Memory: 100M/1516M [INFO]

Process finished with exit code 0

On Thu, Jan 24, 2019 at 2:57 PM Thomas Weise [email protected] wrote:

@abossert https://github.com/abossert looks like a different error than what is shown in the CI. How are you running the tests?

Try mvn clean install if that's not what you are doing already.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache/apex-core/pull/607#issuecomment-457334953, or mute the thread https://github.com/notifications/unsubscribe-auth/AYuy1EmSUgjp__Qycyq77RnAEEB3SHDuks5vGhAegaJpZM4aCUt- .

--

M. Aaron Bossert (202) 594-1703 Punch Cyber Analytics Group

--

M. Aaron Bossert (202) 594-1703 Punch Cyber Analytics Group

[abossert]

nevermind...I found the replacement...but not sure how you guys would like to tackle it...any thoughts?

ParameterValueNotes dfs.http.policy HTTP_ONLY or HTTPS_ONLY or HTTP_AND_HTTPS HTTPS_ONLY turns off http access. This option takes precedence over the deprecated configuration dfs.https.enable and hadoop.ssl.enabled. If using SASL to authenticate data transfer protocol instead of running DataNode as root and using privileged ports, then this property must be set to HTTPS_ONLY to guarantee authentication of HTTP servers. (See dfs.data.transfer.protection .)

On Thu, Jan 24, 2019 at 5:21 PM Aaron Bossert [email protected] wrote:

Another question...I have been able to validate that all of the constant values that were causing build errors have been replaced with the appropriate replacements from Hadoop 2.6.1 -> 3.1.0, however the two following ones just show up as deprecated with no clue about a replacement. I found a reference to them in Hadoop 2.5 source code...but not sure where that would leave us...can anyone chime in on background for how these variables are used in Apex?

@Deprecated /** Only used by HttpServer. / public static final String HADOOP_SSL_ENABLED_KEY = "hadoop.ssl.enabled"; @Deprecated /* Only used by HttpServer. */ public static final boolean HADOOP_SSL_ENABLED_DEFAULT = false;

On Thu, Jan 24, 2019 at 4:18 PM Aaron Bossert [email protected] wrote:

I am at a loss. Tried completely restarting and all seems fine now. Got the initial build to work pre-upgrade to hadoop 3.1.0:

[DEBUG] Using catalog /Users/mbossert/.m2/archetype-catalog.xml [DEBUG] Reading the catalog /Users/mbossert/.m2/archetype-catalog.xml [INFO]

[INFO] Reactor Summary: [INFO] [INFO] Apache Apex ........................................ SUCCESS [ 2.744 s] [INFO] Apache Apex API .................................... SUCCESS [ 8.776 s] [INFO] Apache Apex Common Library ......................... SUCCESS [ 6.235 s] [INFO] Apache Apex Buffer Server .......................... SUCCESS [ 4.226 s] [INFO] Apache Apex Stream Processing Engine ............... SUCCESS [08:27 min] [INFO] Apache Apex Application Maven Archetype ............ SUCCESS [ 1.977 s] [INFO] Apache Apex App Configuration Maven Archetype ...... SUCCESS [ 0.613 s] [INFO]

[INFO] BUILD SUCCESS [INFO]

[INFO] Total time: 08:52 min [INFO] Finished at: 2019-01-24T16:16:35-05:00 [INFO] Final Memory: 100M/1516M [INFO]

Process finished with exit code 0

On Thu, Jan 24, 2019 at 2:57 PM Thomas Weise [email protected] wrote:

@abossert https://github.com/abossert looks like a different error than what is shown in the CI. How are you running the tests?

Try mvn clean install if that's not what you are doing already.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache/apex-core/pull/607#issuecomment-457334953, or mute the thread https://github.com/notifications/unsubscribe-auth/AYuy1EmSUgjp__Qycyq77RnAEEB3SHDuks5vGhAegaJpZM4aCUt- .

--

M. Aaron Bossert (202) 594-1703 Punch Cyber Analytics Group

--

M. Aaron Bossert (202) 594-1703 Punch Cyber Analytics Group

--

M. Aaron Bossert (202) 594-1703 Punch Cyber Analytics Group

[sanjaypujare]

My preference: just make the change to honor dfs.http.policy only and not worry about dfs.https.enable and hadoop.ssl.enabled . We haven't heard about any existing Apex users using SSL.

[abossert]

Given that the new one is ternary (https, http, and either) and the old one is Boolean, how would suggest to handle it? True == HTTPS, false == HTTP, and then err towards true in case of either?

Sent from my iPhone

On Jan 24, 2019, at 19:16, sanjaypujare [email protected] wrote:

My preference: just make the change to honor dfs.http.policy only and not worry about dfs.https.enable and hadoop.ssl.enabled . We haven't heard about any existing Apex users using SSL.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[sanjaypujare]

Good question. As per https://hadoop.apache.org/docs/r3.1.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml, HTTP_AND_HTTPS means service is provided both on http and https. We use this setting for both outbound and inbound (apex web-service) and enhancing our inbound to support HTTP_AND_HTTPS will be some work which is not really needed at this time. I am okay to treat HTTP_AND_HTTPS as HTTPS_ONLY.

[abossert]

OK, so I am past this part...on to the next error:

Running com.datatorrent.stram.StramMiniClusterTest Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.003 sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.003 sec <<< ERROR! java.lang.NoClassDefFoundError: org/apache/hadoop/net/ServerSocketUtil at com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:139) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.net.ServerSocketUtil at com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:139)

Does this make sense? When I look at org.apache.hadoop.net in the Javadoc, I can't find it even in the 2.6.0 version, which compiled just fine...

On Thu, Jan 24, 2019 at 11:07 PM sanjaypujare [email protected] wrote:

Good question. As per https://hadoop.apache.org/docs/r3.1.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml, HTTP_AND_HTTPS means service is provided both on http and https. We use this setting for both outbound and inbound (apex web-service) and enhancing our inbound to support HTTP_AND_HTTPS will be some work which is not really needed at this time. I am okay to treat HTTP_AND_HTTPS as HTTPS_ONLY.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache/apex-core/pull/607#issuecomment-457448856, or mute the thread https://github.com/notifications/unsubscribe-auth/AYuy1FQexz08SfhioRtZ4o95VbIHluAKks5vGoMNgaJpZM4aCUt- .

--

M. Aaron Bossert (202) 594-1703 Punch Cyber Analytics Group

[abossert]

Sorry, by "it", I meant the ServerSocketUtil class

On Fri, Jan 25, 2019 at 1:58 AM Aaron Bossert [email protected] wrote:

OK, so I am past this part...on to the next error:

Running com.datatorrent.stram.StramMiniClusterTest Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.003 sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.003 sec <<< ERROR! java.lang.NoClassDefFoundError: org/apache/hadoop/net/ServerSocketUtil at com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:139) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.net.ServerSocketUtil at com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:139)

Does this make sense? When I look at org.apache.hadoop.net in the Javadoc, I can't find it even in the 2.6.0 version, which compiled just fine...

On Thu, Jan 24, 2019 at 11:07 PM sanjaypujare [email protected] wrote:

Good question. As per https://hadoop.apache.org/docs/r3.1.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml, HTTP_AND_HTTPS means service is provided both on http and https. We use this setting for both outbound and inbound (apex web-service) and enhancing our inbound to support HTTP_AND_HTTPS will be some work which is not really needed at this time. I am okay to treat HTTP_AND_HTTPS as HTTPS_ONLY.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache/apex-core/pull/607#issuecomment-457448856, or mute the thread https://github.com/notifications/unsubscribe-auth/AYuy1FQexz08SfhioRtZ4o95VbIHluAKks5vGoMNgaJpZM4aCUt- .

--

M. Aaron Bossert (202) 594-1703 Punch Cyber Analytics Group

--

M. Aaron Bossert (202) 594-1703 Punch Cyber Analytics Group

[abossert]

Running into another build/test issue...The testing has been running for hours and seems to be stuck in an endless loop of the following...Does this make sense to anyone?

2019-01-25 20:35:44,796 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:35:45,803 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:35:46,811 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:35:47,819 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:35:48,830 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:35:49,837 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:35:50,846 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:35:51,852 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:35:52,860 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:35:53,864 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:35:54,875 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:35:55,883 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:35:56,890 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:35:57,900 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:35:58,907 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:35:59,917 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:00,925 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:01,930 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:02,933 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:03,942 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:04,952 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:05,960 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:06,966 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:07,968 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:08,976 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:09,985 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:10,994 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:12,003 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:13,013 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:14,020 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:15,030 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:16,041 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:17,051 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:18,061 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:19,070 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:20,076 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:21,080 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:22,084 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:23,093 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:24,100 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:25,103 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:26,111 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:27,119 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:28,121 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:29,127 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:30,136 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:31,145 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:32,149 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:33,158 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:34,167 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:35,177 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:36,186 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:37,188 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:38,190 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:39,195 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:40,197 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:41,201 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:42,203 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:43,207 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:44,211 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:45,216 [master] WARN stram.StreamingContainerManager updateRecoveryCheckpoints - Marking operator PTOperator[id=2,name=o1,state=ACTIVE] blocked committed window ffffffffffffffff, recovery window ffffffffffffffff, current time 1548466605215, last window id change time 1548466544796, window processing timeout millis 60000 2019-01-25 20:36:45,216 [master] INFO stram.StreamingContainerManager updateCheckpoints - Blocked operator PTOperator[id=2,name=o1,state=ACTIVE] container PTContainer[id=2(container-166),state=ACTIVE] time 60419ms 2019-01-25 20:36:45,216 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:46,217 [master] INFO engine.StreamingContainer processHeartbeatResponse - Received shutdown request type ABORT 2019-01-25 20:36:46,217 [master] INFO stram.StramLocalCluster run - Container container-166 restart. 2019-01-25 20:36:46,217 [master] INFO stram.StreamingContainerManager scheduleContainerRestart - Initiating recovery for container-166@localhost 2019-01-25 20:36:46,217 [master] WARN stram.StreamingContainerManager updateRecoveryCheckpoints - Marking operator PTOperator[id=2,name=o1,state=ACTIVE] blocked committed window ffffffffffffffff, recovery window ffffffffffffffff, current time 1548466606217, last window id change time 1548466544796, window processing timeout millis 60000 2019-01-25 20:36:46,217 [master] INFO stram.StreamingContainerManager scheduleContainerRestart - Affected operators [PTOperator[id=2,name=o1,state=ACTIVE]] 2019-01-25 20:36:46,219 [master] INFO stram.StreamingContainerManager assignContainer - Removing container agent container-166 2019-01-25 20:36:46,219 [container-167] INFO stram.StramLocalCluster run - Started container container-167 2019-01-25 20:36:46,219 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:46,220 [container-167] INFO stram.StramLocalCluster log - container-167 msg: [container-167] Entering heartbeat loop.. 2019-01-25 20:36:47,161 [container-166] ERROR stram.StreamingContainerManager processHeartbeat - Unknown container container-166 2019-01-25 20:36:47,161 [container-166] INFO engine.StreamingContainer processHeartbeatResponse - Received shutdown request type ABORT 2019-01-25 20:36:47,161 [container-166] INFO stram.StramLocalCluster log - container-166 msg: [container-166] Exiting heartbeat loop.. 2019-01-25 20:36:47,164 [container-166] INFO stram.StramLocalCluster run - Container container-166 terminating. 2019-01-25 20:36:47,224 [master] WARN stram.StreamingContainerManager calculateEndWindowStats - Some operators are behind for more than 1000 windows! Trimming the end window stats map 2019-01-25 20:36:47,225 [container-167] INFO engine.StreamingContainer processHeartbeatResponse - Deploy request: [OperatorDeployInfo[id=2,name=o1,type=GENERIC,checkpoint={ffffffffffffffff, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=inport1,streamId=TestTuples,sourceNodeId=1,sourcePortName=outport,locality=,partitionMask=0,partitionKeys=]],outputs=[]]] 2019-01-25 20:36:47,225 [container-167] INFO engine.StreamingContainer deployNodes - Restoring operator 2 to checkpoint ffffffffffffffff stateless=false.

On Fri, Jan 25, 2019 at 2:01 AM Aaron Bossert [email protected] wrote:

Sorry, by "it", I meant the ServerSocketUtil class

On Fri, Jan 25, 2019 at 1:58 AM Aaron Bossert [email protected] wrote:

OK, so I am past this part...on to the next error:

Running com.datatorrent.stram.StramMiniClusterTest Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.003 sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.003 sec <<< ERROR! java.lang.NoClassDefFoundError: org/apache/hadoop/net/ServerSocketUtil at com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:139) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.net.ServerSocketUtil at com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:139)

Does this make sense? When I look at org.apache.hadoop.net in the Javadoc, I can't find it even in the 2.6.0 version, which compiled just fine...

On Thu, Jan 24, 2019 at 11:07 PM sanjaypujare [email protected] wrote:

Good question. As per https://hadoop.apache.org/docs/r3.1.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml, HTTP_AND_HTTPS means service is provided both on http and https. We use this setting for both outbound and inbound (apex web-service) and enhancing our inbound to support HTTP_AND_HTTPS will be some work which is not really needed at this time. I am okay to treat HTTP_AND_HTTPS as HTTPS_ONLY.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache/apex-core/pull/607#issuecomment-457448856, or mute the thread https://github.com/notifications/unsubscribe-auth/AYuy1FQexz08SfhioRtZ4o95VbIHluAKks5vGoMNgaJpZM4aCUt- .

--

M. Aaron Bossert (202) 594-1703 Punch Cyber Analytics Group

--

M. Aaron Bossert (202) 594-1703 Punch Cyber Analytics Group

--

M. Aaron Bossert (202) 594-1703 Punch Cyber Analytics Group

[tweise]

@abossert thanks for the contribution. Seeing no activity on the PR since Jan, what's the plan forward?

[abossert]

Thomas,

It’s still on my radar...but have been sidetracked with a large project at work. I’m circling back around to this later this week.

Aaron

Sent from my iPhone

On Apr 13, 2019, at 15:52, Thomas Weise [email protected] wrote:

@abossert thanks for the contribution. Seeing no activity on the PR since Jan, what's the plan forward?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[vrozov]

@abossert any update? If not, I'll close the PR, please re-open when there is an update.