true_poetry
true_poetry copied to clipboard
summerstay
Using existing pertained models
Sorry for the perhaps obvious question. How can I use my own fine-tuned GPT-2 model, or Gwern's poetry model with this? What needs to be changed in the code? Works well with model = GPT2LMHeadModel.from_pretrained("gpt2-xl")
but how do I define a different model to use?
That's terrific that you were able to run it with the gpt-xl model! You're the first person I know of who has used my program besides me.
In the same directory where true_poetry.py is, I have a folder called "poetry." In that folder sits Gwern's pytorch_model.bin and associated small files. I downloaded all the files in a zipped tar file using this command: rsync --verbose rsync:// 78.46.86.149:873/biggan/2019-12-13-gwern-gpt-2-1.5b-poetry-model-500522.tar.xz ./ That download also includes a version for use with tensorflow, instead of pytorch. You can discard that.
As for fine-tuning your own model, Gwern's post about how he did it https://www.gwern.net/GPT-2 goes into enough detail that I was able to fine tune the 345M size model. But since gpt2-xl is so much better anyway, and it is impossible to train without special hardware, you are probably going to get better results by experimenting with finding just the right prompt and sticking with one of these two models than by training your own model.
I have now put this information in the README.
Thanks so much. Probably just me being really inexperienced, but the tar file only seems to have the TF files, I don't see PyTorch_model.bin or other smaller files as mentioned, and your new /poetry folder doesn't seem to have them either. I tried converting the TF to PyTorch, but it wouldn't work for the 1.5B model. Idk what im doing wrong. This project is awesome btw :)
Got it to work with this:
%env OPENAI_GPT2_CHECKPOINT_PATH=/content/true_poetry/gwern
!transformers-cli convert --model_type gpt2 --tf_checkpoint model-500522 --config /content/true_poetry/gwern/config.json --pytorch_dump_output /
This one was neat:
Find peace in all you do and rest to hear, And in your heart the peace of all your thoughts, The peace that comes from self and nature given; That, in its increase and tender flow Like sunlight, smiles itself in all around; That dies as in its youth the flower; that in you resides That makes you holy in their sight; that teaches truth and loves To seek the in, and freely takes that find to give; The peace that comes to men from God above, And is the spring again
Thank you, I will add this to the directions.
On Tue, May 12, 2020 at 3:10 PM ktynski [email protected] wrote:
Got it to work with this:
%env OPENAI_GPT2_CHECKPOINT_PATH=/content/true_poetry/gwern
!transformers-cli convert --model_type gpt2 --tf_checkpoint model-500522 --config /content/true_poetry/gwern/config.json --pytorch_dump_output /
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/summerstay/true_poetry/issues/1#issuecomment-627536496, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA3NRIT54I7XA4RFZL5ILJTRRGNIVANCNFSM4M6FZSFQ .
I love it! Thanks for sharing. And thanks for your other comment too: it already helped someone else with the same problem.
On Tue, May 12, 2020 at 3:27 PM ktynski [email protected] wrote:
This one was neat:
Find peace in all you do and rest to hear, And in your heart the peace of all your thoughts, The peace that comes from self and nature given; That, in its increase and tender flow Like sunlight, smiles itself in all around; That dies as in its youth the flower; that in you resides That makes you holy in their sight; that teaches truth and loves To seek the in, and freely takes that find to give; The peace that comes to men from God above, And is the spring again
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/summerstay/true_poetry/issues/1#issuecomment-627544873, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA3NRIW5HFTLFI5A5DXXO6TRRGPIHANCNFSM4M6FZSFQ .
Glad to help! I'm going to try and fine-tune Gwern's poetry model with song lyrics, or rap lyrics. You can train the 774M on Colab pretty easily. Wish I could train the 1.5. I've fine-tuned 774M on lyrics, and it does pretty well, but almost never rhymes, which I feel like is what makes the difference between people being like "oh ok" and "holy shit." Yours is the first attempt I've seen that does a good job, I think it's a clever approach.
