hyperformer

hyperformer copied to clipboard

Adding "Principled Weight Initialization for Hypernetworks"

6

Hey, Do you think it is possible to add the initialization from https://openreview.net/forum?id=H1lma24tPB to this model? Thanks, Ohad

OhadRubin

@rabeehk It seems all these embeddings are initialized from a pytorch default gassian normal distribution with N(0, 1).

jianghaojun

Off the shelf generation from trained hyperformer++

6

``` from hyperformer.adapters import AdapterController, AutoAdapterConfig from hyperformer.third_party.models import T5Config, T5ForConditionalGeneration from transformers import AutoTokenizer, set_seed import os os.environ["CUDA_VISIBLE_DEVICES"]="0" set_seed(42) config = T5Config.from_pretrained('t5-3b',cache_dir="/local/nlpswordfish/tuhin/") tokenizer = AutoTokenizer.from_pretrained('t5-3b',cache_dir="/local/nlpswordfish/tuhin/") adapter_config = AutoAdapterConfig.get('meta-adapter') #####################...

tuhinjubcse