Issue running demos on Ubuntu 14

[olivermueller]

Hi there,

I get the following error message on all demos:

Importing a file into MALLET: [data/demo/20newsgroups/corpus/corpus.txt] --> [data/demo/20newsgroups/model-mallet/corpus.mallet] Traceback (most recent call last): File "bin/train_mallet.py", line 42, in main() File "bin/train_mallet.py", line 39, in main TrainMallet( args.corpus_path, args.model_path, args.token_regex, args.topics, args.iters, args.quiet, args.overwrite ) File "bin/train_mallet.py", line 25, in TrainMallet BuildLDA( corpus_filename, model_path, tokenRegex = token_regex, numTopics = num_topics, numIters = num_iters ) File "/home/oliver/Desktop/termite-data-server-master/bin/modellers/MalletLDA.py", line 31, in init importer.ImportFileOrFolder( tokenRegex ) File "/home/oliver/Desktop/termite-data-server-master/bin/modellers/MalletLDA.py", line 76, in ImportFileOrFolder self.Shell( command ) File "/home/oliver/Desktop/termite-data-server-master/bin/modellers/MalletLDA.py", line 44, in Shell p = subprocess.Popen( command, stdout = subprocess.PIPE, stderr = subprocess.STDOUT ) File "/usr/lib/python2.7/subprocess.py", line 710, in init errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory

Best, Oliver

[jcchuang]

The file corpus.txt should have been automatically extracted from corpus.db when you download the dataset.

Run the following command to generate the file: bin/export_corpus.py data/demo/20newsgroups/corpus data/demo/20newsgroups/corpus/corpus.txt

You might want to remove the following folder, so that you have a clean start when you train an LDA in MALLET. rm -rf data/demo/20newsgroups/model-mallet

Then, try running the following again. ./demo.py 20newsgroups mallet

[olivermueller]

The file corpus.txt is being created. Here is a longer extract of the error message:

Copying [data/demo/infovis/corpus/corpus.db] --> [apps/temp_20140503_190729_997992_3269/data/corpus.db] Copying [data/demo/infovis/corpus/corpus.txt] --> [apps/temp_20140503_190729_997992_3269/data/corpus.txt] Extracting [data/demo/infovis/corpus/corpus.txt] --> [apps/temp_20140503_190729_997992_3269/data/sentences.txt] An error occured while creating app: infovis_mallet [apps/infovis_mallet] Traceback (most recent call last): File "bin/read_mallet.py", line 87, in main() File "bin/read_mallet.py", line 84, in main ImportMalletLDA( args.app_name, args.model_path, args.corpus_path, args.database_path, args.quiet, args.overwrite ) File "bin/read_mallet.py", line 50, in ImportMalletLDA SplitSentences( corpus_filename, app_sentences_filename ) File "/home/oliver/termite-data-server-master/bin/apps/SplitSentences.py", line 14, in init self.Shell( command ) File "/home/oliver/termite-data-server-master/bin/apps/SplitSentences.py", line 17, in Shell p = subprocess.Popen( command, stdout = subprocess.PIPE, stderr = subprocess.STDOUT ) File "/usr/lib/python2.7/subprocess.py", line 710, in init errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory

The error occurs for mallet and gensim.

[jcchuang]

There must be other issues that cause these files to be missing. Regenerating these files doesn't fix the root problem. Could you remove the data/demo/infovis and apps/infovis_mallet folders, and run "./demo.py infovis mallet"? What is the full console output?

[olivermueller]

Here is the full console output:

oliver@ubuntu:~$ cd termite-data-server-master/ oliver@ubuntu:~/termite-data-server-master$ sudo python demo.py 20newsgroups mallet [sudo] password for oliver:

Build a topic model (mallet) using a demo dataset (20newsgroups) database = data/demo/20newsgroups/corpus corpus = data/demo/20newsgroups/corpus model = data/demo/20newsgroups/model-mallet app = 20newsgroups_mallet

Setting up the 20newsgroups dataset... Creating folder 'data/demo/20newsgroups'... Downloading... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 38.1M 100 38.1M 0 0 279k 0 0:02:19 0:02:19 --:--:-- 291k Uncompressing... Extracting corpus.txt from corpus.db... Exporting database [data/demo/20newsgroups/corpus/corpus.db] to file [data/demo/20newsgroups/corpus/corpus.txt] Corpus available: data/demo/20newsgroups/corpus Available: tools/mallet-2.0.7 Available: tools/mallet-2.0.7 Available: tools/corenlp-3.3.1

Training an LDA topic model using MALLET... corpus = data/demo/20newsgroups/corpus/corpus.txt model = data/demo/20newsgroups/model-mallet token_regex = \w{3,} topics = 20 iters = 1000

Importing a file into MALLET: [data/demo/20newsgroups/corpus/corpus.txt] --> [data/demo/20newsgroups/model-mallet/corpus.mallet] Traceback (most recent call last): File "bin/train_mallet.py", line 42, in main() File "bin/train_mallet.py", line 39, in main TrainMallet( args.corpus_path, args.model_path, args.token_regex, args.topics, args.iters, args.quiet, args.overwrite ) File "bin/train_mallet.py", line 25, in TrainMallet BuildLDA( corpus_filename, model_path, tokenRegex = token_regex, numTopics = num_topics, numIters = num_iters ) File "/home/oliver/termite-data-server-master/bin/modellers/MalletLDA.py", line 31, in init importer.ImportFileOrFolder( tokenRegex ) File "/home/oliver/termite-data-server-master/bin/modellers/MalletLDA.py", line 76, in ImportFileOrFolder self.Shell( command ) File "/home/oliver/termite-data-server-master/bin/modellers/MalletLDA.py", line 44, in Shell p = subprocess.Popen( command, stdout = subprocess.PIPE, stderr = subprocess.STDOUT ) File "/usr/lib/python2.7/subprocess.py", line 710, in init errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory

Import a MALLET LDA topic model as a web2py application... app_name = 20newsgroups_mallet app_path = apps/20newsgroups_mallet model_path = data/demo/20newsgroups/model-mallet corpus_filename = data/demo/20newsgroups/corpus/corpus.txt database_filename = data/demo/20newsgroups/corpus/corpus.db

Creating app: 20newsgroups_mallet [apps/temp_20140518_102959_056298_2262] Creating folder: [apps/temp_20140518_102959_056298_2262/data] Creating folder: [apps/temp_20140518_102959_056298_2262/databases] Linking folder: [apps/temp_20140518_102959_056298_2262/models] Linking folder: [apps/temp_20140518_102959_056298_2262/views] Linking folder: [apps/temp_20140518_102959_056298_2262/controllers] Linking folder: [apps/temp_20140518_102959_056298_2262/static] Linking folder: [apps/temp_20140518_102959_056298_2262/modules] Creating file: [apps/temp_20140518_102959_056298_2262/init.py] Copying [data/demo/20newsgroups/corpus/corpus.db] --> [apps/temp_20140518_102959_056298_2262/data/corpus.db] Copying [data/demo/20newsgroups/corpus/corpus.txt] --> [apps/temp_20140518_102959_056298_2262/data/corpus.txt] Extracting [data/demo/20newsgroups/corpus/corpus.txt] --> [apps/temp_20140518_102959_056298_2262/data/sentences.txt] An error occured while creating app: 20newsgroups_mallet [apps/20newsgroups_mallet] Traceback (most recent call last): File "bin/read_mallet.py", line 87, in main() File "bin/read_mallet.py", line 84, in main ImportMalletLDA( args.app_name, args.model_path, args.corpus_path, args.database_path, args.quiet, args.overwrite ) File "bin/read_mallet.py", line 50, in ImportMalletLDA SplitSentences( corpus_filename, app_sentences_filename ) File "/home/oliver/termite-data-server-master/bin/apps/SplitSentences.py", line 14, in init self.Shell( command ) File "/home/oliver/termite-data-server-master/bin/apps/SplitSentences.py", line 17, in Shell p = subprocess.Popen( command, stdout = subprocess.PIPE, stderr = subprocess.STDOUT ) File "/usr/lib/python2.7/subprocess.py", line 710, in init errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory oliver@ubuntu:~/termite-data-server-master$

[jsbarry]

Ran into some of the same problems here.

1. make sure you have curl installed in your distro (i.e. sudo apt-get install curl). If you don't then the line

curl --insecure --location http://homes.cs.washington.edu/~jcchuang/termite-datasets/$DEMO.zip > $DOWNLOAD_PATH/$DEMO.zip

located in fetch_dataset.sh cannot execute

2. make sure the mallet, CoreNLP and gensim are well downloaded in the utils/tools. CoreNLP is quite large so it may take time.
3. You can always check that the demo.zip can actually open and you can see the contents. Part of the problems I encountered were exactly that...the db wasn't being unzipped so the .txt file couldn't be created