Nathan Ellis Rasmussen comments

Results 17 comments of


                                            Nathan Ellis Rasmussen

How to start training from a checkpoint or snapshot? [question] [documentation]

Hey, me-from-the-past. I haven't done this yet, but we know some things now. For one thing, we'll have to call `run_with_continue()` somehow. If the previous run went to completion, I...

How to start training from a checkpoint or snapshot? [question] [documentation]

> If you can get a message to us-even-further-in-the-past, tell him not to screw around with the "Simple" API. Again, for emphasis. Granted that the division of labor between model...

tokenize_and_cache cooks up wacky paths

> the typical workflow for jiant And there's the crux of ... about half of the things I'm queueing up (and trying to desnark a little) to file as issues....

Evaluation script example

`--ZZsrc` causes your run configuration to include the contents of `${EXP_DIR}/models/${MODEL_TYPE}/config.json`, including a `model_path` argument. Your command-line `model_path` can only override it if you say `--ZZoverrides model_path` along with all...

Evaluation script example

I wish I knew some good guidelines to point you to. I am struggling too. The authors of Jiant have left many assumptions undocumented. When you say "do the evaluation,"...

Evaluation script example

Checking whether you've tokenized ----------------------------- Make sure you have tokenized your test inputs. If you don't remember, you can check the cache directory. If you followed the quick start guide...

Evaluation script example

Training and testing together -------------- [verified] If you use `configurator.py SingleTaskConfigurator`, add the `--do_test` flag to its arguments. [unverified] If you use `configurator.py SimpleAPIMultiTaskConfigurator`, add `--test_task_name_list ${TASK}` (or `$TASK1,$TASK2,$TASK3`). (Yes,...

Evaluation script example

Incidental discoveries and accidental complexity when testing my already trained model ------------- Things that ought to be documented somewhere. My command line: ``` python jiant/jiant/proj/main/runscript.py run \ --ZZsrc ${EXP_DIR}/models/roberta-base/config.json \...

Evaluation script example

Extracting model's predictions part 1: For a Glue or SuperGlue task -------------------------------------- Calling the runscript with `--write_test_preds` leaves an opaque binary `test_preds.p` in the output dir. `benchmark_submission_formatter.py` knows how to...

Evaluation script example

Extracting model's predictions part 2: Any other task ------------ The get-it-done way: Put the following in `dump_preds.py`. ``` import sys import torch all_preds = torch.load(sys.argv[1]) for task, pred_dict in all_preds.items():...