tutorials
tutorials copied to clipboard
streamsets
the tutorial compilation issue
Hi, I git cloned datacollector 1.2.2.0, datacollector-api 1.2.2.0, can you tell me which version of datacollector-plugin-api I should get? it has no 1.2.2.0.
While compiling the datacollector: mvn clean install -DskipTests, the container failed. the log:
[INFO] Building StreamSets Data Collector Container 1.2.2.0 [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ streamsets-datacollector-container --- [INFO] Deleting C:\Users\g706872\git\datacollector\container\target [INFO] [INFO] --- hadoop-maven-plugins:2.5.0:version-info (version-info) @ streamsets-datacollector-container --- [WARNING] [svn, info] failed with error code 1 [INFO] SCM: GIT [INFO] Computed MD5: 62ac5d1ee141cfb7921c48f46b28db [INFO] [INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ streamsets-datacollector-container --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 9 resources [INFO] Copying 1 resource [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ streamsets-datacollector-container --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 364 source files to C:\Users\g706872\git\datacollector\container\target\classes [INFO] ------------------------------------------------------------- [WARNING] COMPILATION WARNING : [INFO] ------------------------------------------------------------- [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/memory/MultipleMonitorLocker.java:[24,16] sun.misc.Unsafe is internal proprietary API and may be removed in a future release [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/memory/MultipleMonitorLocker.java:[24,16] sun.misc.Unsafe is internal proprietary API and may be removed in a future release [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/memory/MultipleMonitorLocker.java:[24,16] sun.misc.Unsafe is internal proprietary API and may be removed in a future release [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/memory/MultipleMonitorLocker.java:[24,16] sun.misc.Unsafe is internal proprietary API and may be removed in a future release [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/memory/MultipleMonitorLocker.java:[33,24] sun.misc.Unsafe is internal proprietary API and may be removed in a future release [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/memory/MultipleMonitorLocker.java:[35,5] sun.misc.Unsafe is internal proprietary API and may be removed in a future release [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/memory/MultipleMonitorLocker.java:[37,27] sun.misc.Unsafe is internal proprietary API and may be removed in a future release [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/memory/MultipleMonitorLocker.java:[39,17] sun.misc.Unsafe is internal proprietary API and may be removed in a future release [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/execution/runner/cluster/ClusterRunner.java: Some input files use or override a deprecated API. [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/execution/runner/cluster/ClusterRunner.java: Recompile with -Xlint:deprecation for details. [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/config/StageDefinition.java: Some input files use unchecked or unsafe operations. [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/config/StageDefinition.java: Recompile with -Xlint:unchecked for details. [INFO] 12 warnings [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/definition/ConfigDefinitionExtractor.java:[391,12] an enum switch case label must be the unqualified name of an enumeration constant [INFO] 1 error [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------
Is there a specific reason you're trying to use such an old version? We are currently on 3.13.0.
Hi, Mr. Pat,
I picked the 1.2.2.0 because it is the version used in this tutorial I am trying to follow. I am not sure the difference of building new version and the old version. and I am not sure the steps/code in the tutorial work with new version of streamsets datacollector. If you have equivalent tutorial on how to create custom java destination, that will be awesome. Can you share with me if you have?
if i want to try the new version, which version number do you recommend? and all three modules are with the same version number?
Thanks,
Wei
On Wed, Mar 4, 2020 at 2:14 PM Pat Patterson [email protected] wrote:
Is there a specific reason you're trying to use such an old version? We are currently on 3.13.0.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/streamsets/tutorials/issues/118?email_source=notifications&email_token=AOXQL4H2RPMD4GL3IRDS2TLRF2SC7A5CNFSM4LBQXRV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENZWNGA#issuecomment-594765464, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOXQL4AG6KNE34ZJDRDI25LRF2SC7ANCNFSM4LBQXRVQ .
Check the version of StreamSets Data Collector that you are using. It needs to be the same as that. The same instructions should still work.
Hi, Mr. Pat,
I got all three project from master and I am trying to build following the order. I am still having issue with building datacollector. I managed to solve as many as I can and I am stuck at this:
(very similar warning about the same line 201, column 19 for many
dependencies)
[WARNING]
'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)'
must be unique: com.streamsets:streamsets-datacollector-kafka_multiso
urce-protolib:test-jar -> duplicate declaration of version
${project.version} @
com.streamsets:streamsets-datacollector-root-lib:3.15.0-SNAPSHOT,
C:\Users\g706872
git\datacollector\root-lib\pom.xml, line 201, column 19
[WARNING]
[WARNING] Some problems were encountered while building the effective model
for
com.streamsets:streamsets-datacollector-mongodb_3-lib:jar:3.15.0-SNAPSHOT
[WARNING]
'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)'
must be unique: com.streamsets:streamsets-datacollector-kafka_multiso
urce-protolib:test-jar -> duplicate declaration of version
${project.version} @
com.streamsets:streamsets-datacollector-root-lib:3.15.0-SNAPSHOT,
C:\Users\g706872
git\datacollector\root-lib\pom.xml, line 201, column 19
[WARNING]
[WARNING] Some problems were encountered while building the effective model
for
com.streamsets:streamsets-datacollector-databricks-ml_2-lib:jar:3.15.0-SNAPSHOT
[WARNING]
'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)'
must be unique: com.streamsets:streamsets-datacollector-kafka_multiso
urce-protolib:test-jar -> duplicate declaration of version
${project.version} @
com.streamsets:streamsets-datacollector-root-lib:3.15.0-SNAPSHOT,
C:\Users\g706872
git\datacollector\root-lib\pom.xml, line 201, column 19
[WARNING]
[WARNING] It is highly recommended to fix these problems because they
threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support
building such malformed projects.
[WARNING]
[INFO]
[INFO] Reactor Build Order: [INFO] [INFO] StreamSets Data Collector Jython 2.7.0 Lib [INFO] StreamSets Data Collector MapR 6.1.0 Lib [INFO] StreamSets Data Collector MapR 6.1.0 MEP 6 Lib [INFO] Streamsets Data Collector MapR Spark 2.1 MEP 3.0 Lib [INFO] StreamSets Data Collector MongoDB 3.0.2 Lib [INFO] StreamSets Data Collector Databricks ML 2 Lib [INFO] [INFO]
[INFO] Building StreamSets Data Collector Jython 2.7.0 Lib 3.15.0-SNAPSHOT [INFO]
[INFO]
[INFO] Reactor Summary: [INFO] [INFO] StreamSets Data Collector Jython 2.7.0 Lib ......... FAILURE [ 1.247 s] [INFO] StreamSets Data Collector MapR 6.1.0 Lib ........... SKIPPED [INFO] StreamSets Data Collector MapR 6.1.0 MEP 6 Lib ..... SKIPPED [INFO] Streamsets Data Collector MapR Spark 2.1 MEP 3.0 Lib SKIPPED [INFO] StreamSets Data Collector MongoDB 3.0.2 Lib ........ SKIPPED [INFO] StreamSets Data Collector Databricks ML 2 Lib ...... SKIPPED [INFO]
[INFO] BUILD FAILURE [INFO]
[INFO] Total time: 12.922 s [INFO] Finished at: 2020-03-06T14:06:40-05:00 [INFO] Final Memory: 175M/917M [INFO]
[ERROR] Failed to execute goal on project streamsets-datacollector-jython_2_7-lib: Could not resolve dependencies for project com.streamsets:streamsets-datacollect or-jython_2_7-lib:jar:3.15.0-SNAPSHOT: Failure to find com.streamsets:streamsets-datacollector-jython-protolib:jar:tests:3.15.0-SNAPSHOT in http://packages.conflue nt.io/maven/ was cached in the local repository, resolution will not be reattempted until the update interval of confluent has elapsed or updates are forced -> [He lp 1]
While I was trying to build streamsets-datacollector-jython-protolib:jar:tests:3.15.0-SNAPSHOT separately, I got:
[ERROR] C:\users\g706872\git\datacollector\jython-protolib\src\test\java\com\streamsets\pipeline\stage\processor\scripting\TestJythonAbstractScriptingProcessor.jav a:[19,43] package com.streamsets.datacollector.runner does not exist [ERROR] C:\users\g706872\git\datacollector\jython-protolib\src\test\java\com\streamsets\pipeline\stage\origin\scripting\TestJythonAbstractScriptingSource.java:[19, 43] package com.streamsets.datacollector.runner does not exist
Then, I tried to rebuild the container, since I know the com.streamsets.datacollector.runner is in the container project. the container was build successfully, but still the jython-protolib test jar build failed for the same reason.
Can you help me on this compilation issue? or is it enough to create my own java custom destination if my custom destination has nothing to do with jython, MarR, MongoDB, databricks?
Thanks,
Wei
On Wed, Mar 4, 2020 at 2:53 PM Pat Patterson [email protected] wrote:
Check the version of StreamSets Data Collector that you are using. It needs to be the same as that. The same instructions should still work.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/streamsets/tutorials/issues/118?email_source=notifications&email_token=AOXQL4CGBJPYHKLM66SKAELRF2WSPA5CNFSM4LBQXRV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENZ5XUA#issuecomment-594795472, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOXQL4EBOXZ7OZPHPUBKCUDRF2WSPANCNFSM4LBQXRVQ .
Hi Wei - I'll investigate the build failure. Yes - try creating your custom destination now. I don't think those specific issues will be a problem.