MLWIC2 Classify having trouble finding python

I'm trying to get this package to work with just the example images and example code, but I get an error about not being able to find python even though it's installed in the exact location I've specified. Here's the code and the error:

classify(path_prefix = "/home/pmcmahon/Documents/MLWIC2_Package_Images", data_info = "/home/pmcmahon/Documents/image_labels.csv",
model_dir = "/home/pmcmahon/Documents/MLWIC2_helper_files", python_loc = "/anaconda3/bin/", save_predictions = "model_predictions.txt", make_output = TRUE, output_name = "MLWIC2_output.csv", num_cores = 4 )

Your data_info file exists: /home/pmcmahon/Documents/image_labels.csv. Your `path_prefix exists: /home/pmcmahon/Documents/MLWIC2_Package_Images. You are not using a Windows computer. sh: 1: /anaconda3/bin/python: not found The classify function did not run properly. Warning message: In system(eval_py) : error in running command

Aug 28 '20 18:08 whilton9

I don't understand why it's not finding python. Can you open your terminal and type /anaconda3/bin/python? This will open python. Then type print(2+2). See what happens

Aug 28 '20 18:08 mikeyEcology

If you're able to run python from the command line like this, then let's try adding print_cmd=TRUE to your classify call, so it should look like this:

classify(path_prefix = "/home/pmcmahon/Documents/MLWIC2_Package_Images",
data_info = "/home/pmcmahon/Documents/image_labels.csv",
model_dir = "/home/pmcmahon/Documents/MLWIC2_helper_files",
python_loc = "/anaconda3/bin/",
save_predictions = "model_predictions.txt",
make_output = TRUE,
output_name = "MLWIC2_output.csv",
num_cores = 4,
print_cmd=TRUE # this is the changed line
)

This will print the command that is being passed to python.

Then in your terminal window, exit python by typing exit(), navigate to your MLWIC2_helper_files folder in the terminal (by typing cd /home/pmcmahon/Documents/MLWIC2_helper_files, and open Python again by typing python. Then copy the output from the R console and paste it into python in your terminal.

Aug 28 '20 19:08 mikeyEcology

I was able to open python and typing print(2+2) yielded an answer of 4, so it appears to be working. I then did everything you suggested in the last comment. However, when I remove print_cmd=TRUE, I still get the message saying that the classify function did not run properly.

Sep 02 '20 18:09 whilton9

Yes-it's not running properly because it's printing something instead of running, but it should have printed a system call that will look like python run.py eval --num_threads .... Do you see this?

Sep 02 '20 19:09 mikeyEcology

Yes it printed this: "/anaconda3/bin/python run.py eval --num_threads 4 --architecture resnet --depth 18 --log_dir species_model --snapshot_prefix species_model --path_prefix /home/pmcmahon/Documents/MLWIC2_Package_Images --batch_size 128 --val_info /home/pmcmahon/Documents/image_labels.csv --delimiter , --save_predictions /home/pmcmahon/Documents/MLWIC2_helper_files/model_predictions.txt --top_n 5 --num_gpus 2 --num_classes 1000\n"

Sep 02 '20 19:09 whilton9

Ok. so copy that and paste it into your python, which you can get to by typing /anaconda3/bin/python into a terminal window.

Sep 02 '20 20:09 mikeyEcology

I did that, but nothing has changed.

Sep 02 '20 20:09 whilton9

Is there an error? Otherwise it will be running in the background. Does this file exist /home/pmcmahon/Documents/MLWIC2_helper_files/model_predictions.txt? I didn't mean to close this issue, keep posting here and I think it will reopen

Sep 02 '20 20:09 mikeyEcology

I wasn't sure if I should remove print_cmd=TRUE or not. If I leave it in the command, this is the output:

Your data_info file exists: /home/pmcmahon/Documents/image_labels.csv. Your `path_prefix exists: /home/pmcmahon/Documents/MLWIC2_Package_Images. You are not using a Windows computer. [1] "/anaconda3/bin/python run.py eval --num_threads 4 --architecture resnet --depth 18 --log_dir species_model --snapshot_prefix species_model --path_prefix /home/pmcmahon/Documents/MLWIC2_Package_Images --batch_size 128 --val_info /home/pmcmahon/Documents/image_labels.csv --delimiter , --save_predictions /home/pmcmahon/Documents/MLWIC2_helper_files/model_predictions.txt --top_n 5 --num_gpus 2 --num_classes 1000\n"

If I remove it, this is the output:

Your data_info file exists: /home/pmcmahon/Documents/image_labels.csv. Your `path_prefix exists: /home/pmcmahon/Documents/MLWIC2_Package_Images. You are not using a Windows computer. sh: 1: /anaconda3/bin/python: not found The classify function did not run properly. Warning message: In system(eval_py) : error in running command

So there was an error and that file doesn't exist.

Sep 02 '20 21:09 whilton9

What happens if in the terminal you navigate to /anaconda3/bin/ and then type python run.py eval --num_threads 4 --architecture resnet --depth 18 --log_dir species_model --snapshot_prefix species_model --path_prefix /home/pmcmahon/Documents/MLWIC2_Package_Images --batch_size 128 --val_info /home/pmcmahon/Documents/image_labels.csv --delimiter , --save_predictions /home/pmcmahon/Documents/MLWIC2_helper_files/model_predictions.txt --top_n 5 --num_gpus 2 --num_classes 1000

Sep 03 '20 17:09 mikeyEcology

It output this: (null): can't open file 'run.py': [Errno 2] No such file or directory

Sep 08 '20 19:09 whilton9

Sorry. In type cd /home/pmcmahon/Documents/MLWIC2_helper_files, then type python run.py eval --num_threads 4 --architecture resnet --depth 18 --log_dir species_model --snapshot_prefix species_model --path_prefix /home/pmcmahon/Documents/MLWIC2_Package_Images --batch_size 128 --val_info /home/pmcmahon/Documents/image_labels.csv --delimiter , --save_predictions /home/pmcmahon/Documents/MLWIC2_helper_files/model_predictions.txt --top_n 5 --num_gpus 2 --num_classes 1000

Sep 08 '20 20:09 mikeyEcology

Ok I changed to that directory and typed the command, but got the same error message

Sep 08 '20 20:09 whilton9

what is in your MLWIC2_helper_files folder? Can you type ls in the terminal while in this folder?

Sep 08 '20 20:09 mikeyEcology

data_info_train.csv __MACOSX MLWIC2_helper_files MLWIC2_test_tf.py

Sep 09 '20 15:09 whilton9

Ok. Something went wrong when you downloaded the helper files folder. Can you try downloading it from the readme and unzipping it. It should contain:

ConfMatrixID.py pycache/ architectures/ empty_animal/ run.py utils.py arch.py checkpoint data_loader.py species_model/

Sep 09 '20 16:09 mikeyEcology

Ok I downloaded it again and it contains all those things. However, I'm getting the same errors when running the classify command

Sep 09 '20 17:09 whilton9

Ok. Try this again: in terminal type cd /home/pmcmahon/Documents/MLWIC2_helper_files, then type python run.py eval --num_threads 4 --architecture resnet --depth 18 --log_dir species_model --snapshot_prefix species_model --path_prefix /home/pmcmahon/Documents/MLWIC2_Package_Images --batch_size 128 --val_info /home/pmcmahon/Documents/image_labels.csv --delimiter , --save_predictions /home/pmcmahon/Documents/MLWIC2_helper_files/model_predictions.txt --top_n 5 --num_gpus 2 --num_classes 1000

Sep 09 '20 18:09 mikeyEcology

Here's the output after doing that:

Traceback (most recent call last): File "run.py", line 21, in import tensorflow as tf ModuleNotFoundError: No module named 'tensorflow'

Sep 09 '20 19:09 whilton9

This means that tensorflow is not installed properly. I recently updated the setup function so that it should install tensorflow properly on your machine. You can update MLWIC2 and run setup again, or you can do it by yourself in conda. In terminal type conda activate r-reticulate then pip install tensorflow

Sep 09 '20 19:09 mikeyEcology

I ran setup again and got this output when I ran classify:

Your data_info file exists: /home/pmcmahon/Documents/image_labels.csv. Your `path_prefix exists: /home/pmcmahon/Documents/MLWIC2_Package_Images. You are not using a Windows computer. sh: 1: /anaconda3/bin/python: not found The classify function did not run properly. Warning message: In system(eval_py) : error in running command

I then ran those commands in the terminal and got the same output when I tried running classify again

Sep 09 '20 19:09 whilton9

What was your output when you ran it in the terminal?

Sep 09 '20 20:09 mikeyEcology

Collecting tensorflow Downloading tensorflow-2.3.0-cp38-cp38-manylinux2010_x86_64.whl (320.5 MB) |████████████████████████████████| 320.5 MB 732 bytes/s Collecting tensorboard<3,>=2.3.0 Downloading tensorboard-2.3.0-py3-none-any.whl (6.8 MB) |████████████████████████████████| 6.8 MB 7.2 MB/s Collecting h5py<2.11.0,>=2.10.0 Downloading h5py-2.10.0-cp38-cp38-manylinux1_x86_64.whl (2.9 MB) |████████████████████████████████| 2.9 MB 6.7 MB/s Collecting grpcio>=1.8.6 Downloading grpcio-1.32.0-cp38-cp38-manylinux2014_x86_64.whl (3.8 MB) |████████████████████████████████| 3.8 MB 9.4 MB/s Collecting wrapt>=1.11.1 Downloading wrapt-1.12.1.tar.gz (27 kB) Collecting gast==0.3.3 Downloading gast-0.3.3-py2.py3-none-any.whl (9.7 kB) Requirement already satisfied: wheel>=0.26 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.8/site-packages (from tensorflow) (0.35.1) Collecting numpy<1.19.0,>=1.16.0 Downloading numpy-1.18.5-cp38-cp38-manylinux1_x86_64.whl (20.6 MB) |████████████████████████████████| 20.6 MB 7.7 MB/s Collecting astunparse==1.6.3 Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB) Collecting google-pasta>=0.1.8 Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB) |████████████████████████████████| 57 kB 3.5 MB/s Collecting tensorflow-estimator<2.4.0,>=2.3.0 Downloading tensorflow_estimator-2.3.0-py2.py3-none-any.whl (459 kB) |████████████████████████████████| 459 kB 20.3 MB/s Collecting absl-py>=0.7.0 Downloading absl_py-0.10.0-py3-none-any.whl (127 kB) |████████████████████████████████| 127 kB 7.2 MB/s Collecting six>=1.12.0 Downloading six-1.15.0-py2.py3-none-any.whl (10 kB) Collecting scipy==1.4.1 Downloading scipy-1.4.1-cp38-cp38-manylinux1_x86_64.whl (26.0 MB) |████████████████████████████████| 26.0 MB 5.6 MB/s Collecting termcolor>=1.1.0 Using cached termcolor-1.1.0.tar.gz (3.9 kB) Collecting protobuf>=3.9.2 Downloading protobuf-3.13.0-cp38-cp38-manylinux1_x86_64.whl (1.3 MB) |████████████████████████████████| 1.3 MB 9.1 MB/s Collecting keras-preprocessing<1.2,>=1.1.1 Downloading Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB) |████████████████████████████████| 42 kB 1.1 MB/s Collecting opt-einsum>=2.3.2 Downloading opt_einsum-3.3.0-py3-none-any.whl (65 kB) |████████████████████████████████| 65 kB 2.9 MB/s Collecting tensorboard-plugin-wit>=1.6.0 Downloading tensorboard_plugin_wit-1.7.0-py3-none-any.whl (779 kB) |████████████████████████████████| 779 kB 12.8 MB/s Collecting google-auth-oauthlib<0.5,>=0.4.1 Downloading google_auth_oauthlib-0.4.1-py2.py3-none-any.whl (18 kB) Collecting requests<3,>=2.21.0 Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB) |████████████████████████████████| 61 kB 410 kB/s Collecting werkzeug>=0.11.15 Downloading Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB) |████████████████████████████████| 298 kB 10.6 MB/s Requirement already satisfied: setuptools>=41.0.0 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.8/site-packages (from tensorboard<3,>=2.3.0->tensorflow) (49.6.0.post20200814) Collecting google-auth<2,>=1.6.3 Downloading google_auth-1.21.1-py2.py3-none-any.whl (93 kB) |████████████████████████████████| 93 kB 600 kB/s Collecting markdown>=2.6.8 Downloading Markdown-3.2.2-py3-none-any.whl (88 kB) |████████████████████████████████| 88 kB 2.9 MB/s Collecting requests-oauthlib>=0.7.0 Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB) Collecting chardet<4,>=3.0.2 Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB) |████████████████████████████████| 133 kB 10.5 MB/s Requirement already satisfied: certifi>=2017.4.17 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow) (2020.6.20) Collecting idna<3,>=2.5 Downloading idna-2.10-py2.py3-none-any.whl (58 kB) |████████████████████████████████| 58 kB 4.3 MB/s Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB) |████████████████████████████████| 127 kB 12.9 MB/s Collecting pyasn1-modules>=0.2.1 Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB) |████████████████████████████████| 155 kB 6.2 MB/s Collecting cachetools<5.0,>=2.0.0 Downloading cachetools-4.1.1-py3-none-any.whl (10 kB) Collecting rsa<5,>=3.1.4; python_version >= "3.5" Downloading rsa-4.6-py3-none-any.whl (47 kB) |████████████████████████████████| 47 kB 4.1 MB/s Collecting oauthlib>=3.0.0 Downloading oauthlib-3.1.0-py2.py3-none-any.whl (147 kB) |████████████████████████████████| 147 kB 20.8 MB/s Collecting pyasn1<0.5.0,>=0.4.6 Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB) |████████████████████████████████| 77 kB 4.3 MB/s Building wheels for collected packages: wrapt, termcolor Building wheel for wrapt (setup.py) ... done Created wheel for wrapt: filename=wrapt-1.12.1-cp38-cp38-linux_x86_64.whl size=78072 sha256=573b8e05638979fb768a30d9d098636a87d20beb1f90328fb72489b6260a61fb Stored in directory: /home/pmcmahon/.cache/pip/wheels/5f/fd/9e/b6cf5890494cb8ef0b5eaff72e5d55a70fb56316007d6dfe73 Building wheel for termcolor (setup.py) ... done Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4830 sha256=409e8ae29881363c2d813e7e199e890585d6e3522dd9eb09885aa7ae704a7d81 Stored in directory: /home/pmcmahon/.cache/pip/wheels/a0/16/9c/5473df82468f958445479c59e784896fa24f4a5fc024b0f501 Successfully built wrapt termcolor Installing collected packages: six, absl-py, tensorboard-plugin-wit, numpy, oauthlib, chardet, idna, urllib3, requests, requests-oauthlib, pyasn1, pyasn1-modules, cachetools, rsa, google-auth, google-auth-oauthlib, werkzeug, protobuf, markdown, grpcio, tensorboard, h5py, wrapt, gast, astunparse, google-pasta, tensorflow-estimator, scipy, termcolor, keras-preprocessing, opt-einsum, tensorflow Successfully installed absl-py-0.10.0 astunparse-1.6.3 cachetools-4.1.1 chardet-3.0.4 gast-0.3.3 google-auth-1.21.1 google-auth-oauthlib-0.4.1 google-pasta-0.2.0 grpcio-1.32.0 h5py-2.10.0 idna-2.10 keras-preprocessing-1.1.2 markdown-3.2.2 numpy-1.18.5 oauthlib-3.1.0 opt-einsum-3.3.0 protobuf-3.13.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 requests-2.24.0 requests-oauthlib-1.3.0 rsa-4.6 scipy-1.4.1 six-1.15.0 tensorboard-2.3.0 tensorboard-plugin-wit-1.7.0 tensorflow-2.3.0 tensorflow-estimator-2.3.0 termcolor-1.1.0 urllib3-1.25.10 werkzeug-1.0.1 wrapt-1.12.1

Sep 09 '20 20:09 whilton9

Shoot, I forgot you need the right version of tensorflow. In conda run pip uninstall tensorflow and if necessary, keep running it until you get the error "Cannot uninstall requirement tensorflow, not installed" (This is a very finicky package). Then run pip install tensorflow==1.14

Sep 09 '20 20:09 mikeyEcology

After running the uninstall command twice, I got this: WARNING: Skipping tensorflow as it is not installed.

But when I ran the install command I got: ERROR: Could not find a version that satisfies the requirement tensorflow==1.14 (from versions: 2.2.0rc1, 2.2.0rc2, 2.2.0rc3, 2.2.0rc4, 2.2.0, 2.3.0rc0, 2.3.0rc1, 2.3.0rc2, 2.3.0) ERROR: No matching distribution found for tensorflow==1.14

Sep 09 '20 20:09 whilton9

try pip install tensorflow==1.14.0?

Sep 09 '20 20:09 mikeyEcology

Nah, I got the same error

Sep 09 '20 21:09 whilton9

The problem is you're using the wrong python version. You can change it in conda by using conda install python==3.7

Sep 09 '20 21:09 mikeyEcology

Ok I installed python3.7 and tensorflow1.14.0, but classify gave me the same error

Sep 09 '20 21:09 whilton9

Did you try running it from the terminal?

Sep 10 '20 15:09 mikeyEcology

MLWIC2 MLWIC2 copied to clipboard

Classify having trouble finding python

MLWIC2
MLWIC2 copied to clipboard