Lev McKinney

Test examples sometimes timeout

8

## The current behavior It appears that `test_run_example_notebooks` is sometimes timing out when executing during CI checks. ## Expected behavior The tests should not be timing out. This should be...

RewardNet refactor

7

# RewardNet Refactor This issue proposes shrinking the `RewardNet` class to make it easier to extend and maintain. The RewardNet class acts as the base class for all pytorch reward...

There are some differences between stable-baselines' `VecNormilize` and imitation's `RunningNorm/NormilizedRewardFunction` that might cause performance regressions. The VecNormilization in normalizes based on an estimate of rewards so far in the episode....

Implemented the ability to train rewards in preference comparison against multiple policies

6

## Description This pull request adds the `MixtureOfTrajectoryGenerators` class which allows preference comparison to be trained on data from multiple sources. This includes the ability to train against multiple agents...

HF Trainer TrainingArguments can't be used with `default_factory`

1

**Describe the bug** I've really been loving using simple-parsing in my projects. It looks like you are trying to maintain compatibility with hugging faces dataclass #172. One use case I've...

Support for gemma architecture models

2

We also need to bump some dependencies for this since gemma does not exist in the version of transformers we were requiring.

Drop Python 3.9 support too while we're at it (enabling `|` for union types) _Originally posted by @norabelrose in https://github.com/AlignmentResearch/tuned-lens/issues/125#issuecomment-1968239228_

Currently, if you try to create a prediction trajectory from a model and lens loaded in `bfloat16` error. ``` 294 traj_log_probs.append( --> 295 logits.log_softmax(dim=-1).squeeze().detach().cpu().numpy() 296 ) 298 # Add model...

Prediction depth

2

In the paper there is a nice visualization of prediction depth. Prediction depth is defined in the paper is the first layer where the most likely token is equal to...

This feature will be removed in #63. It would be nice to reimplement this so that we can not only see which tokens have a high probability at each layer...