elit icon indicating copy to clipboard operation
elit copied to clipboard
verified publisher icon emorynlp

ELIT installation problem

Open chan-hub opened this issue 3 years ago • 8 comments

Can't install ELIT properly - ran into 'illegal instruction(core dump)' upon a test command.

Here's the result of installation script:

chan@cscope37: pip3 install elit --pre Collecting elit Using cached https://files.pythonhosted.org/packages/13/2b/a83c0d2578fa2426ae8f58c4dcf387688e6434752b2cd3cca223c5a9a140/elit-0.1.35-py3- none-any.whl Collecting mxnet>=1.4.1 (from elit) Using cached https://files.pythonhosted.org/packages/81/f5/d79b5b40735086ff1100c680703e0f3efc830fa455e268e9e96f3c857e93/mxnet-1.6.0-py2. py3-none-any.whl Collecting numpy==1.14.5 (from elit) Using cached https://files.pythonhosted.org/packages/68/1e/116ad560de97694e2d0c1843a7a0075cc9f49e922454d32f49a80eb6f1f2/numpy-1.14.5-cp3 6-cp36m-manylinux1_x86_64.whl Collecting gluonnlp==0.6.0 (from elit) Collecting marisa-trie==0.7.5 (from elit) Collecting tqdm==4.26.0 (from elit) Using cached https://files.pythonhosted.org/packages/79/43/19c9fee28110cd47f73e6bc596394337fe9f3e5825b4de402bbf30b3beb5/tqdm-4.26.0-py2. py3-none-any.whl Collecting argparse==1.4.0 (from elit) Using cached https://files.pythonhosted.org/packages/f2/94/3af39d34be01a24a6e65433d19e107099374224905f1e0cc6bbe1fd22a2f/argparse-1.4.0-p y2.py3-none-any.whl Collecting gensim==3.6.0 (from elit) Using cached https://files.pythonhosted.org/packages/27/a4/d10c0acc8528d838cda5eede0ee9c784caa598dbf40bd0911ff8d067a7eb/gensim-3.6.0-cp3 6-cp36m-manylinux1_x86_64.whl Collecting graphviz<0.9.0,>=0.8.1 (from mxnet>=1.4.1->elit) Using cached https://files.pythonhosted.org/packages/53/39/4ab213673844e0c004bed8a0781a0721a3f6bb23eb8854ee75c236428892/graphviz-0.8.4-p y2.py3-none-any.whl Collecting requests<3,>=2.20.0 (from mxnet>=1.4.1->elit) Using cached https://files.pythonhosted.org/packages/2d/61/08076519c80041bc0ffa1a8af0cbd3bf3e2b62af10435d269a9d0f40564d/requests-2.27.1- py2.py3-none-any.whl Collecting scipy>=0.18.1 (from gensim==3.6.0->elit) Using cached https://files.pythonhosted.org/packages/c8/89/63171228d5ced148f5ced50305c89e8576ffc695a90b58fe5bb602b910c2/scipy-1.5.4-cp36 -cp36m-manylinux1_x86_64.whl Collecting smart-open>=1.2.1 (from gensim==3.6.0->elit) Using cached https://files.pythonhosted.org/packages/09/db/fab79b619923e26cecc5fb460c80f71f99666fe19182d5bb600ec4d6ff10/smart_open-6.0.0 -py3-none-any.whl Collecting six>=1.5.0 (from gensim==3.6.0->elit) Using cached https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.p y3-none-any.whl Collecting urllib3<1.27,>=1.21.1 (from requests<3,>=2.20.0->mxnet>=1.4.1->elit) Using cached https://files.pythonhosted.org/packages/ec/03/062e6444ce4baf1eac17a6a0ebfe36bb1ad05e1df0e20b110de59c278498/urllib3-1.26.9-p y2.py3-none-any.whl Collecting certifi>=2017.4.17 (from requests<3,>=2.20.0->mxnet>=1.4.1->elit) Using cached https://files.pythonhosted.org/packages/11/dd/e015f3780f42dd9af62cf0107b44ea1298926627ecd70c17b0e484e95bcd/certifi-2022.5.1 8.1-py3-none-any.whl Collecting charset-normalizer~=2.0.0; python_version >= "3" (from requests<3,>=2.20.0->mxnet>=1.4.1->elit) Using cached https://files.pythonhosted.org/packages/06/b3/24afc8868eba069a7f03650ac750a778862dc34941a4bebeb58706715726/charset_normaliz er-2.0.12-py3-none-any.whl Collecting idna<4,>=2.5; python_version >= "3" (from requests<3,>=2.20.0->mxnet>=1.4.1->elit) Using cached https://files.pythonhosted.org/packages/04/a2/d918dcd22354d8958fe113e1a3630137e0fc8b44859ade3063982eacd2a4/idna-3.3-py3-non e-any.whl Installing collected packages: graphviz, urllib3, certifi, charset-normalizer, idna, requests, numpy, mxnet, gluonnlp, marisa-trie, tqdm, argparse, scipy, smart-open, six, gensim, elit Successfully installed argparse-1.4.0 certifi-2022.5.18.1 charset-normalizer-2.0.12 elit-0.1.35 gensim-3.6.0 gluonnlp-0.6.0 graphviz-0.8.4 idna-3.3 marisa-trie-0.7.5 mxnet-1.6.0 numpy-1.14.5 requests-2.27.1 scipy-1.5.4 six-1.16.0 smart-open-6.0.0 tqdm-4.26.0 urllib3-1.26.9

And here's the test command ran into core dump: chan@cscope37:~/ELIT/elit-main$ elit dep_parser parse Illegal instruction (core dumped)

Omitting the 'dep_parser' arg, the command showed help contents: chan@cscope37:~/ELIT/elit-main$ elit parse Unrecognized command usage: elit []

    commands:
        token_tagger: use token tagger
        pos_tagger: run pos tagger
        ner_tagger: run ner tagger
        dep_parser: run syntactic dependency parser
        sdp_parser: run semantic dependency parser
        install: Install model
        download: download pre-trained model or word embedding

positional arguments: command command to run

optional arguments: -h, --help show this help message and exit

chan-hub avatar Jun 03 '22 23:06 chan-hub

Hi, seems that you have an legacy elit installed. The latest version doesn't print commands you gave. What you called is not what you installed just now. To confirm, run

$ elit --help

which should show ELIT-2.0.0-alpha.8 but your system probably would show 1.x.

You can either uninstall the previous version or install elit in a clean new conda env or a virtual env or use colab instead. See https://colab.research.google.com/drive/1vEatxoXQbfLC4Hpy1lLG_DFOIfguPcUE?usp=sharing#scrollTo=yaHSawqjHObr

hankcs avatar Jun 04 '22 09:06 hankcs

Hi, This is the very first ELIT installation - there's no old version of ELIT around. And the command 'elit --help' doesn't show any version info, but the standard help info:

chan@cscope37:~/ELIT/elit-main/docs$ elit --help usage: elit [] commands: token_tagger: use token tagger pos_tagger: run pos tagger ner_tagger: run ner tagger dep_parser: run syntactic dependency parser sdp_parser: run semantic dependency parser install: Install model download: download pre-trained model or word embedding positional arguments: command command to run optional arguments: -h, --help show this help message and exit

The issue may be related to a broken python runtime - originally, this ubuntu 18.04 server comes w/ python 2.7.17 (as default python) and python 3.6. Since ELIT requires python >= 3.7, python 3.8 was installed.

chan@cscope37:~/ELIT/elit-main$ python3.8 -V Python 3.8.5

Default python is still 2.7 however: chan@cscope37:~/ELIT/elit-main$ python -V Python 2.7.17

chan-hub avatar Jun 05 '22 01:06 chan-hub

Well, then the pip on your system is of Python2. ELIT2.x only supports Python3 so pip downloaded the legacy version. To install 2.x, you need to run pip3 install instead. I'd always suggest to use a conda env which gives you the access to MKL/CUDA accerlation.

hankcs avatar Jun 05 '22 01:06 hankcs

The installation log shows that pip3 was used instead of pip, but the end result is 'elit-0.1.35'. The pip3 version is 3.6, however, that might be the reason grabbing older version ? Tried re-installing w/ pip3.8 but then ran into http connection problem. Will try later to see how it goes. Thanks for quick respond.

chan@cscope37: pip3 install elit --pre Collecting elit Using cached https://files.pythonhosted.org/packages/13/2b/a83c0d2578fa2426ae8f58c4dcf387688e6434752b2cd3cca223c5a9a140/elit-0.1.35-py3-none-any.whl Collecting mxnet>=1.4.1 (from elit) Using cached https://files.pythonhosted.org/packages/81/f5/d79b5b40735086ff1100c680703e0f3efc830fa455e268e9e96f3c857e93/mxnet-1.6.0-py2.py3-none-any.whl ... Installing collected packages: graphviz, urllib3, certifi, charset-normalizer, idna, requests, numpy, mxnet, gluonnlp, marisa-trie, tqdm, argparse, scipy, smart-open, six, gensim, elit Successfully installed argparse-1.4.0 certifi-2022.5.18.1 charset-normalizer-2.0.12 elit-0.1.35 gensim-3.6.0 gluonnlp-0.6.0 graphviz-0.8.4 idna-3.3 marisa-trie-0.7.5 mxnet-1.6.0 numpy-1.14.5 requests-2.27.1 scipy-1.5.4 six-1.16.0 smart-open-6.0.0 tqdm-4.26.0 urllib3-1.26.9

chan-hub avatar Jun 05 '22 07:06 chan-hub

After installing python3.7 w/ its pip, still ran into an error - looks like python exception:

chan@cscope6:~/project/ELIT/elit-main$ pip install elit --pre Defaulting to user installation because normal site-packages is not writeable Collecting elit Downloading elit-2.0.0a8-py3-none-any.whl (485 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 485.8/485.8 kB 2.0 MB/s eta 0:00:00 Collecting toposort==1.5 Downloading toposort-1.5-py2.py3-none-any.whl (7.6 kB) Collecting unofficial-stog==0.0.21 Downloading unofficial_stog-0.0.21.tar.gz (220 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 220.9/220.9 kB 1.8 MB/s eta 0:00:00 Preparing metadata (setup.py) ... done Collecting penman==1.2.1 Downloading Penman-1.2.1-py3-none-any.whl (43 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.3/43.3 kB 625.8 kB/s eta 0:00:00 Collecting uvicorn==0.13.1 Downloading uvicorn-0.13.1-py3-none-any.whl (45 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.5/45.5 kB 711.5 kB/s eta 0:00:00 Collecting fastapi==0.65.2 Downloading fastapi-0.65.2-py3-none-any.whl (51 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.2/51.2 kB 866.7 kB/s eta 0:00:00 Collecting transformers==4.6.1 Downloading transformers-4.6.1-py3-none-any.whl (2.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 2.4 MB/s eta 0:00:00 Collecting torch<1.9.0,>=1.6.0 Downloading torch-1.8.1-cp37-cp37m-manylinux1_x86_64.whl (804.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 804.1/804.1 MB 484.4 kB/s eta 0:00:00 Collecting termcolor==1.1.0 Downloading termcolor-1.1.0.tar.gz (3.9 kB) Preparing metadata (setup.py) ... done Collecting sentencepiece==0.1.95 Downloading sentencepiece-0.1.95-cp37-cp37m-manylinux2014_x86_64.whl (1.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 2.5 MB/s eta 0:00:00 Collecting pynvml==8.0.4 Downloading pynvml-8.0.4-py3-none-any.whl (36 kB) Collecting scipy==1.5.4 Downloading scipy-1.5.4-cp37-cp37m-manylinux1_x86_64.whl (25.9 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 25.9/25.9 MB 3.7 MB/s eta 0:00:00 Collecting phrasetree==0.0.4 Downloading phrasetree-0.0.4.tar.gz (43 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.0/44.0 kB 755.8 kB/s eta 0:00:00 Preparing metadata (setup.py) ... done Collecting editdistance==0.5.3 Downloading editdistance-0.5.3-cp37-cp37m-manylinux1_x86_64.whl (179 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 179.2/179.2 kB 778.1 kB/s eta 0:00:00 Collecting alnlp==1.0.0rc27 Downloading alnlp-1.0.0rc27.tar.gz (130 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 130.8/130.8 kB 1.7 MB/s eta 0:00:00 Preparing metadata (setup.py) ... done Collecting pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2 Downloading pydantic-1.9.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.1 MB) ━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/11.1 MB 672.6 kB/s eta 0:00:12 ERROR: Exception: Traceback (most recent call last): File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 435, in _error_catcher yield File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 516, in read data = self._fp.read(amt) if not fp_closed else b"" File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 90, in read data = self.__fp.read(amt) File "/usr/lib/python3.7/http/client.py", line 461, in read n = self.readinto(b) File "/usr/lib/python3.7/http/client.py", line 505, in readinto n = self.fp.readinto(b) File "/usr/lib/python3.7/socket.py", line 589, in readinto return self._sock.recv_into(b) File "/usr/lib/python3.7/ssl.py", line 1071, in recv_into return self.read(nbytes, buffer) File "/usr/lib/python3.7/ssl.py", line 929, in read return self._sslobj.read(len, buffer) socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper status = run_func(*args) File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper return func(self, options, args) File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 342, in run reqs, check_supported_wheels=not options.target_dir File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve collected.requirements, max_rounds=try_to_avoid_resolution_too_deep File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve state = resolution.resolve(requirements, max_rounds=max_rounds) File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 373, in resolve failure_causes = self._attempt_to_pin_criterion(name) File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion criteria = self._get_updated_criteria(candidate) File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 204, in _get_updated_criteria self._add_to_criteria(criteria, requirement, parent=candidate) File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria if not criterion.candidates: File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in bool return bool(self._sequence) File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in bool return any(self) File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in return (c for c in iterator if id(c) not in self._incompatible_ids) File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built candidate = func() File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 220, in _make_candidate_from_link version=version, File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 297, in init version=version, File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 161, in init self.dist = self._prepare() File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 230, in _prepare dist = self._prepare_distribution() File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 302, in _prepare_distribution return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True) File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 428, in prepare_linked_requirement return self._prepare_linked_requirement(req, parallel_builds) File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 479, in _prepare_linked_requirement hashes, File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 159, in unpack_url hashes=hashes, File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 96, in get_http_url from_path, content_type = download(link, temp_dir.path) File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/network/download.py", line 146, in call for chunk in chunks: File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/cli/progress_bars.py", line 53, in _rich_progress_bar for chunk in iterable: File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/network/utils.py", line 87, in response_chunks decode_content=False, File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 573, in stream data = self.read(amt=amt, decode_content=decode_content) File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 538, in read raise IncompleteRead(self._fp_bytes_read, self.length_remaining) File "/usr/lib/python3.7/contextlib.py", line 130, in exit self.gen.throw(type, value, traceback) File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 440, in _error_catcher raise ReadTimeoutError(self._pool, None, "Read timed out.") pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

Could you help on this ?

Thanks !

chan-hub avatar Jun 10 '22 23:06 chan-hub

Hi, the new error is due to your network failed to reach files.pythonhosted.org, which is a public site not maintained by us. I'd recommend you to either retry, or to use a pip mirror/VPN, depending on your geo location.

hankcs avatar Jun 11 '22 00:06 hankcs

Hi, Yes, the server LAN gateway wasn't stable - caused downloading errors. Migrating to another server w/ solid WAN connection solved the network problem - successfully installed & tested: Downloading: 43%|██████████████████████████████████████▌ | 587k/1.36M [00:00<00:00, 1.96MB/s]Downloading: 72%|███████████████████████████████████████████████████████████████▊ | 972k/1.36M [00:00<00:00, 2.62MB/s]Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████| 1.36M/1.36M [00:00<00:00, 2.36MB/s] Building model

Emory NLP is a research lab in Atlanta. [W NNPACK.cpp:80] Could not initialize NNPACK! Reason: Unsupported hardware. { "lem": [ ["emory", "nlp", "be", "a", "research", "lab", "in", "atlanta", "."] ], "pos": [ ["NNP", "NNP", "VBZ", "DT", "NN", "NN", "IN", "NNP", "."] ], "ner": [ [["ORG", 0, 2, "Emory NLP"], ["GPE", 7, 8, "Atlanta"]] ], "srl": [ [[["ARG1", 0, 2, "Emory NLP"], ["PRED", 2, 3, "is"], ["ARG2", 3, 8, "a research lab in Atlanta"]]] ],

Two more questions:

  1. Can the '[W NNPACK.cpp:80] Could not initialize NNPACK! Reason: Unsupported hardware.' msg be ignored ? Look like a C preprocessor complains about a macro line - the server is Ubuntu 18.04 running on Intel CPU

  2. How the ELIT parsing result be represented in node list form, where each node captures the form, lemma, PoS, dependency label, etc.

Thanks so much for your help !

chan-hub avatar Jun 13 '22 00:06 chan-hub

  1. This is raised by pytorch. Maybe your hardware is just too old. You can build from source to get rid of it: https://discuss.pytorch.org/t/bug-w-nnpack-cpp-80-could-not-initialize-nnpack-reason-unsupported-hardware/107518
  2. Format per each annotation can be found in links in the README.md.

hankcs avatar Jun 13 '22 00:06 hankcs