gemini
gemini copied to clipboard
src-d
Adds usage report
I've created this PR for #219. This is not supposed to be merged but in this way, you can comment and also answer some questions in this report. After the review, I think that we can then write the document needed for #219.
Most probably I missed things that are needed before considering to Gemini a product, but @smacker you will surely be able to integrate them given you knowledge of the project 👍. I approached this task as somebody who doesn't know anything about Gemini.
I see ./sbt test failed for you. But were you able to run gemini in development mode?
@smacker what do you mean by development mode exactly?
I've just tried running standalone and some exceptions occured (this doens't happen when inside the container):
✔ se7entyse7en in ~/Projects/src-d/gemini (usage-report) $ SPARK_HOME=~/Downloads/spark-2.2.3-bin-hadoop2.7 ./hash --replace src/test/resources/siva
Using spark-submit from /Users/se7entyse7en/Downloads/spark-2.2.3-bin-hadoop2.7
Running Hashing as Apache Spark job, master: local[*]
Hashing 5 repositories in: 'src/test/resources/siva' ()
file:/Users/se7entyse7en/Projects/src-d/gemini/src/test/resources/siva/duplicate-funcs/27f7db976994baf808b205c1ca95ba961cebf59d.siva
file:/Users/se7entyse7en/Projects/src-d/gemini/src/test/resources/siva/duplicate-files/f281ab6f2e0e38dcc3af05360667d8f530c00103.siva
file:/Users/se7entyse7en/Projects/src-d/gemini/src/test/resources/siva/duplicate-files/9279be3cf07fb3cca4fc964b27acea57e0af461b.siva
file:/Users/se7entyse7en/Projects/src-d/gemini/src/test/resources/siva/unique-files/5fb38a5744b2496ff8484e57b46a754433ede457.siva
file:/Users/se7entyse7en/Projects/src-d/gemini/src/test/resources/siva/2636f3c62f1a407b2996da6e3fe6fdc5d1ccd764.siva
WARN 18:13:32 tech.sourced.gemini.Gemini (Gemini.scala:37) - Getting repositories at src/test/resources/siva in siva format
WARN 18:13:35 tech.sourced.gemini.Gemini (Gemini.scala:46) - Hashing
WARN 18:13:35 tech.sourced.gemini.Hash (Hash.scala:125) - Listing files
WARN 18:13:36 tech.sourced.gemini.Hash (Hash.scala:141) - Extracting UASTs
WARN 18:13:37 tech.sourced.gemini.Hash (Hash.scala:162) - Extracting features
[Stage 2:================> (58 + 4) / 200]ERROR 18:13:46 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:================> (59 + 4) / 200]ERROR 18:13:46 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:================> (60 + 4) / 200]ERROR 18:13:47 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=================> (61 + 4) / 200]ERROR 18:13:47 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=================> (62 + 4) / 200]ERROR 18:13:47 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:47 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:47 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:47 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=================> (64 + 4) / 200]ERROR 18:13:47 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==================> (66 + 4) / 200]ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===================> (68 + 4) / 200]ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===================> (71 + 4) / 200]ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
WARN 18:13:48 org.apache.spark.internal.Logging$class (Logging.scala:66) - FATAL hashes/chaos_machine.py: transform failed: received 9 errors:
check: key "col_offset": variable "pos_start_exists" redeclared (true vs false)
check: key "col_offset": variable "pos_start_exists" redeclared (true vs false)
check: key "col_offset": variable "pos_start_exists" redeclared (true vs false)
check: key "col_offset": variable "pos_start_exists" redeclared (true vs false)
check: key "col_offset": variable "pos_start_exists" redeclared (true vs false)
check: key "col_offset": variable "pos_start_exists" redeclared (true vs false)
check: key "col_offset": variable "pos_start_exists" redeclared (true vs false)
check: key "col_offset": variable "pos_start_exists" redeclared (true vs false)
check: key "col_offset": variable "pos_start_exists" redeclared (true vs false)
ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:====================> (74 + 4) / 200]ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=====================> (77 + 4) / 200]ERROR 18:13:49 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:49 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=====================> (78 + 4) / 200]ERROR 18:13:49 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:======================> (79 + 4) / 200]ERROR 18:13:49 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:49 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:49 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=======================> (83 + 4) / 200]ERROR 18:13:50 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=======================> (84 + 4) / 200]ERROR 18:13:50 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:50 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:50 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:50 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:========================> (87 + 4) / 200]ERROR 18:13:50 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:50 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=========================> (92 + 4) / 200]ERROR 18:13:51 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:51 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:51 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:51 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==========================> (94 + 4) / 200]ERROR 18:13:51 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==========================> (95 + 4) / 200]ERROR 18:13:51 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==========================> (96 + 4) / 200]ERROR 18:13:51 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:51 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:51 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===========================> (98 + 4) / 200]ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===========================> (100 + 4) / 200]ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:============================> (102 + 4) / 200]ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:============================> (105 + 4) / 200]ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=============================> (106 + 4) / 200]ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=============================> (107 + 4) / 200]ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=============================> (108 + 4) / 200]ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==============================> (111 + 4) / 200]ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===============================> (113 + 4) / 200]ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===============================> (116 + 4) / 200]ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:================================> (118 + 4) / 200]ERROR 18:13:54 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:================================> (119 + 4) / 200]ERROR 18:13:54 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:54 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=================================> (120 + 4) / 200]ERROR 18:13:54 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:54 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=================================> (122 + 4) / 200]ERROR 18:13:54 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=================================> (123 + 4) / 200]ERROR 18:13:55 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==================================> (124 + 4) / 200]ERROR 18:13:55 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:55 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==================================> (125 + 4) / 200]ERROR 18:13:55 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==================================> (126 + 4) / 200]ERROR 18:13:55 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:55 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:55 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:55 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===================================> (130 + 4) / 200]ERROR 18:13:55 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:====================================> (132 + 4) / 200]ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:====================================> (134 + 4) / 200]ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=====================================> (137 + 4) / 200]ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:======================================> (139 + 4) / 200]ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:======================================> (141 + 4) / 200]ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:========================================> (147 + 4) / 200]ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=========================================> (150 + 4) / 200]ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=========================================> (151 + 4) / 200]ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==========================================> (153 + 4) / 200]ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===========================================> (158 + 4) / 200]ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===========================================> (159 + 4) / 200]ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:============================================> (160 + 4) / 200]ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:============================================> (162 + 4) / 200]ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:============================================> (163 + 4) / 200]ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==============================================> (168 + 4) / 200]ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==============================================> (169 + 4) / 200]ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==============================================> (170 + 4) / 200]ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===============================================> (172 + 4) / 200]ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===============================================> (173 + 4) / 200] WARN 18:13:59 org.apache.spark.internal.Logging$class (Logging.scala:66) - FATAL sorts/external-sort.py: syntax error: Errors produced trying to get an AST for both Python versions
------ Python2 errors:
[b'Traceback (most recent call last):\n File "<string>", line 1, in <module>\n File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast2dict.py", line 19, in ast2dict\n return visitor.parse()\n File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast2dict.py", line 45, in parse\n tree = ast.parse(self.codestr, mode=\'exec\')\n File "/usr/lib/python2.7/ast.py", line 37, in parse\n return compile(source, filename, mode, PyCF_ONLY_AST)\n File "<unknown>", line 161\n main()\n ^\nIndentationError: expected an indented block\n']
------ Python3 errors:
['Traceback (most recent call last):\n File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast_checks.py", line 53, in check_ast\n current_ast = ast2dict(code)\n File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast2dict.py", line 19, in ast2dict\n return visitor.parse()\n File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast2dict.py", line 45, in parse\n tree = ast.parse(self.codestr, mode=\'exec\')\n File "/usr/local/lib/python3.6/ast.py", line 35, in parse\n return compile(source, filename, mode, PyCF_ONLY_AST)\n File "<unknown>", line 161\n main()\n ^\nIndentationError: expected an indented block\n']
ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===============================================> (174 + 4) / 200]ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:================================================> (176 + 4) / 200]ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=================================================> (179 + 4) / 200]ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=================================================> (180 + 4) / 200]ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=================================================> (181 + 4) / 200]ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==================================================> (185 + 4) / 200]ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===================================================> (188 + 4) / 200]ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:====================================================> (190 + 4) / 200]ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=====================================================> (194 + 4) / 200] WARN 18:14:01 org.apache.spark.internal.Logging$class (Logging.scala:66) - FATAL File_Transfer_Protocol/ftp_send_receive.py: syntax error: Errors produced trying to get an AST for both Python versions
------ Python2 errors:
[b'Traceback (most recent call last):\n File "<string>", line 1, in <module>\n File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast2dict.py", line 19, in ast2dict\n return visitor.parse()\n File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast2dict.py", line 45, in parse\n tree = ast.parse(self.codestr, mode=\'exec\')\n File "/usr/lib/python2.7/ast.py", line 37, in parse\n return compile(source, filename, mode, PyCF_ONLY_AST)\n File "<unknown>", line 1\n """\n ^\nIndentationError: unexpected indent\n']
------ Python3 errors:
['Traceback (most recent call last):\n File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast_checks.py", line 53, in check_ast\n current_ast = ast2dict(code)\n File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast2dict.py", line 19, in ast2dict\n return visitor.parse()\n File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast2dict.py", line 45, in parse\n tree = ast.parse(self.codestr, mode=\'exec\')\n File "/usr/local/lib/python3.6/ast.py", line 35, in parse\n return compile(source, filename, mode, PyCF_ONLY_AST)\n File "<unknown>", line 1\n """\n ^\nIndentationError: unexpected indent\n']
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:02 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:02 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:02 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:======================================================>(199 + 1) / 200]ERROR 18:14:02 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:02 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
WARN 18:14:02 tech.sourced.gemini.Hash (Hash.scala:52) - Feature Extraction exceptions
WARN 18:14:02 tech.sourced.gemini.Hash (Hash.scala:55) - Processed: 0, skipped: 169
WARN 18:14:02 tech.sourced.gemini.Hash$$anonfun$report$2 (Hash.scala:56) - StatusRuntimeException -> 169
WARN 18:14:02 tech.sourced.gemini.Hash (Hash.scala:183) - creating document frequencies
WARN 18:14:05 tech.sourced.gemini.Hash (Hash.scala:240) - hashing features
WARN 18:14:05 tech.sourced.gemini.Gemini (Gemini.scala:49) - Saving hashes to DB
WARN 18:14:05 tech.sourced.gemini.Hash (Hash.scala:290) - save meta to DB
WARN 18:14:06 tech.sourced.gemini.Hash (Hash.scala:270) - save document frequencies to DB
WARN 18:14:06 tech.sourced.gemini.Hash (Hash.scala:309) - save hashtables to DB
Done
what do you mean by development mode exactly
something like
./sbt assemblyPackageDependency
./sbt assembly
./hash -k local src/test/resources/siva
./report -k local
./query -k local <options>
according to your output most probably you did not.
./report & ./query most probably returned you nothing because ./hash failed.
I actually ran gemini locally, but with all the deps on containers:
d481f61dd2d5 bblfsh/bblfshd:v2.12.0-drivers "bblfshd -ctl-addres…" About a minute ago Up About a minute 0.0.0.0:9432->9432/tcp srcd-cli-bblfshd
8fb09de3a4b8 scylladb/scylla:2.0.0 "/docker-entrypoint.…" 4 days ago Up About a minute 7000-7001/tcp, 9160/tcp, 9180/tcp, 10000/tcp, 0.0.0.0:9042->9042/tcp gemini_scylla_1
5146ff34d557 gemini_featurext "python ./server.py" 4 days ago Up About a minute gemini_featurext_1
1b91ec55d391 bblfsh/bblfshd:v2.11.0-drivers "bblfshd" 4 days ago Up About a minute gemini_bblfshd_1
Do you perhaps mean running everything on the host?
I mean according to your logs:
WARN 18:14:02 tech.sourced.gemini.Hash (Hash.scala:55) - Processed: 0, skipped: 169
gemini didn't process anything and didn't find any similarities