stellar-random-walk
stellar-random-walk copied to clipboard
data61
test case to reproduce "Wrong Transports: 1" message
input graph: 1 3 1 4 1 5 1 6 2 3 2 4 2 5 2 6
spark version: 2.1.0 the parms as flow: { w2vIter: 10, w2vLr: 0.025, w2vPartitions: 1, w2vDim: 128, w2vWindow: 10, walkLength: 20, numWalks: 10, p: 1.0, q: 1.0, weighted: true, directed: false, input: /user/kgalg/node2vec/graph/karate.edgelist, output: /user/kgalg/node2vec/randomwalk, rddPartitions: 200, singleOutput: true, partitioned: true, cmd: randomwalk }
running log: randomwalk.log
Q1: why does "Wrong Transports: 1" often appear in log file? Q2: what triggers the "ERROR org.apache.spark.internal.Logging$class.logError" message in the output? Q3: Are these two issues related to spark version? spark2.1.0 does work?
@shps We notice the same thing: If we run the program in local mode, the result is correct. However, when you run the application in standalone mode, the following line:
https://github.com/data61/stellar-random-walk/blob/master/randomwalk/src/main/scala/au/csiro/data61/randomwalk/algorithm/VCutRandomWalk.scala#L112
results in empty GraphMap object outside the mapPartitionWithIndex method. Besides, manipulating an object inside each partition seems very strange. Is it correct?
