nlu
nlu copied to clipboard
JohnSnowLabs
spark nlu load error
Am trying to explore NLU models first and then the NLU Healthcare models. nlu.load('emotion') step is failing. Attached the logs.
OS – Linux RHEL Pyspark – version 3.0.1 Command used for install - python3 -m pip install nlu pyspark==3.0.1 --trusted-host pypi.org --trusted-host files.pythonhosted.org I have created a python venv and install the NLU per above command.
I also tried reinstalling with below command: python3 -m pip install --upgrade nlu streamlit pyspark==3.0.2
Code below:
import nlu
pp=nlu.load('emotion')
classifierdl_use_emotion download started this may take some time.
Approximate size to download 21.3 MB
[ / ]
An error occurred while calling z:com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.downloadModel.
: java.lang.NoClassDefFoundError: org/tensorflow/Tensor
at com.johnsnowlabs.ml.tensorflow.TensorflowWrapper$.read(TensorflowWrapper.scala:397)
at com.johnsnowlabs.ml.tensorflow.ReadTensorflowModel.readTensorflowModel(TensorflowSerializeModel.scala:145)
at com.johnsnowlabs.ml.tensorflow.ReadTensorflowModel.readTensorflowModel$(TensorflowSerializeModel.scala:120)
at com.johnsnowlabs.nlp.annotators.classifier.dl.ClassifierDLModel$.readTensorflowModel(ClassifierDLModel.scala:291)
at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadClassifierDLTensorflowModel.readTensorflow(ClassifierDLModel.scala:278)
at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadClassifierDLTensorflowModel.readTensorflow$(ClassifierDLModel.scala:276)
at com.johnsnowlabs.nlp.annotators.classifier.dl.ClassifierDLModel$.readTensorflow(ClassifierDLModel.scala:291)
at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadClassifierDLTensorflowModel.$anonfun$$init$$1(ClassifierDLModel.scala:285)
at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadClassifierDLTensorflowModel.$anonfun$$init$$1$adapted(ClassifierDLModel.scala:285)
at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.$anonfun$onRead$1(ParamsAndFeaturesReadable.scala:47)
at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.$anonfun$onRead$1$adapted(ParamsAndFeaturesReadable.scala:46)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.onRead(ParamsAndFeaturesReadable.scala:46)
at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.$anonfun$read$1(ParamsAndFeaturesReadable.scala:57)
at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.$anonfun$read$1$adapted(ParamsAndFeaturesReadable.scala:57)
at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:35)
at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:24)
at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadModel(ResourceDownloader.scala:333)
at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadModel(ResourceDownloader.scala:327)
at com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader$.downloadModel(ResourceDownloader.scala:456)
at com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.downloadModel(ResourceDownloader.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.tensorflow.Tensor
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 34 more
[OK!]
EXCEPTION: Could not resolve singular Component for type=emotion and nlp_ref=classifierdl_use_emotion and nlu_ref=emotion and lang =en
Traceback (most recent call last):
File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/nlu/pipe/component_resolution.py", line 852, in construct_component_from_identifier
is_licensed=is_licensed)
File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/nlu/components/classifier.py", line 69, in init
else : self.model = ClassifierDl.get_pretrained_model(nlp_ref, language)
File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/nlu/components/classifiers/classifier_dl/classifier_dl.py", line 11, in get_pretrained_model
return ClassifierDLModel.pretrained(name,language,bucket)
File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/sparknlp/annotator.py", line 8063, in pretrained
return ResourceDownloader.downloadModel(ClassifierDLModel, name, lang, remote_loc)
File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/sparknlp/pretrained.py", line 62, in downloadModel
raise e
File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/sparknlp/pretrained.py", line 59, in downloadModel
j_obj = _internal._DownloadModel(reader.name, name, language, remote_loc, j_dwn).apply()
File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/sparknlp/internal.py", line 214, in init
name, language, remote_loc)
File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/sparknlp/internal.py", line 165, in init
self._java_obj = self.new_java_obj(java_obj, *args)
File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/sparknlp/internal.py", line 175, in new_java_obj
return self._new_java_obj(java_class, *args)
File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/pyspark/ml/wrapper.py", line 69, in _new_java_obj
return java_obj(*java_args)
File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/py4j/java_gateway.py", line 1305, in call
answer, self.gateway_client, self.target_id, self.name)
File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/pyspark/sql/utils.py", line 128, in deco
return f(*a, **kw)
File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/py4j/protocol.py", line 328, in get_return_value
format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling z:com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.downloadModel.
: java.lang.NoClassDefFoundError: org/tensorflow/Tensor
at com.johnsnowlabs.ml.tensorflow.TensorflowWrapper$.read(TensorflowWrapper.scala:397)
at com.johnsnowlabs.ml.tensorflow.ReadTensorflowModel.readTensorflowModel(TensorflowSerializeModel.scala:145)
at com.johnsnowlabs.ml.tensorflow.ReadTensorflowModel.readTensorflowModel$(TensorflowSerializeModel.scala:120)
at com.johnsnowlabs.nlp.annotators.classifier.dl.ClassifierDLModel$.readTensorflowModel(ClassifierDLModel.scala:291)
at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadClassifierDLTensorflowModel.readTensorflow(ClassifierDLModel.scala:278)
at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadClassifierDLTensorflowModel.readTensorflow$(ClassifierDLModel.scala:276)
at com.johnsnowlabs.nlp.annotators.classifier.dl.ClassifierDLModel$.readTensorflow(ClassifierDLModel.scala:291)
at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadClassifierDLTensorflowModel.$anonfun$$init$$1(ClassifierDLModel.scala:285)
at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadClassifierDLTensorflowModel.$anonfun$$init$$1$adapted(ClassifierDLModel.scala:285)
at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.$anonfun$onRead$1(ParamsAndFeaturesReadable.scala:47)
at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.$anonfun$onRead$1$adapted(ParamsAndFeaturesReadable.scala:46)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.onRead(ParamsAndFeaturesReadable.scala:46)
at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.$anonfun$read$1(ParamsAndFeaturesReadable.scala:57)
at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.$anonfun$read$1$adapted(ParamsAndFeaturesReadable.scala:57)
at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:35)
at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:24)
at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadModel(ResourceDownloader.scala:333)
at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadModel(ResourceDownloader.scala:327)
at com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader$.downloadModel(ResourceDownloader.scala:456)
at com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.downloadModel(ResourceDownloader.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.tensorflow.Tensor
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 34 more
ValueError Traceback (most recent call last) /apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/nlu/init.py in load(request, path, verbose, gpu, streamlit_caching) 341 if nlu_ref == '': continue --> 342 nlu_component = nlu_ref_to_component(nlu_ref, authenticated=is_authenticated) 343 # if we get a list of components, then the NLU reference is a pipeline, we do not need to check order
/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/nlu/pipe/component_resolution.py in nlu_ref_to_component(nlu_reference, detect_lang, authenticated, is_recursive_call) 322 authenticated=authenticated, --> 323 is_recursive_call=is_recursive_call) 324 if resolved_component is None:
/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/nlu/pipe/component_resolution.py in resolve_component_from_parsed_query_data(lang, component_type, dataset, component_embeddings, nlu_ref, trainable, path, authenticated, is_recursive_call) 467 if constructed_component is None: --> 468 raise ValueError(f'EXCEPTION : Could not create NLU component for nlp_ref={nlp_ref} and nlu_ref={nlu_ref}') 469 else:
ValueError: EXCEPTION : Could not create NLU component for nlp_ref=classifierdl_use_emotion and nlu_ref=emotion
During handling of the above exception, another exception occurred:
Exception Traceback (most recent call last)
/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/nlu/init.py in load(request, path, verbose, gpu, streamlit_caching) 360 print(e[1]) 361 raise Exception( --> 362 "Something went wrong during loading and fitting the pipe. Check the other prints for more information and also verbose mode. Did you use a correct model reference?") 363 364
Exception: Something went wrong during loading and fitting the pipe. Check the other prints for more information and also verbose mode. Did you use a correct model reference?
I get the same error on Windows 10 despite following instructions step by step.
