Run-tests.sh will fail if run DF in transformer

[allwefantasy]

In transformer's fit function I have following code:

         word2vec = Word2Vec(vectorSize=self.getEmbeddingSize(), minCount=1, inputCol=self.getInputCol(),
                            outputCol="word_embedding")

        vectorsDf = word2vec.fit(
            dataset.select(f.split(self.getInputCol(), "\\s+").alias(self.getInputCol()))).getVectors()
       
        # vectorsDf._sc._jsc    is None

When you run the run-tests.sh script, vectorsDf._sc._jsc will be None and you can not do some actions eg. collect,take .

I run this in IDE or spark-submit , It is just fine.

[thunterdb]

@allwefantasy can you open a pull request (which will fail obviously) so as to have a reproducible use case?