minGPT icon indicating copy to clipboard operation
minGPT copied to clipboard

[WIP] codestyle and Catalyst example

Open Scitator opened this issue 5 years ago • 1 comments

Dear @karpathy , I am Sergey, author of Catalyst - high level framework for deep learning research and development on top of PyTorch.

Recently, my friends told me about "high-level frameworks overselling" going here, so... I decided to join :joy:

Long story short, this PR

  • fixes codestyle a bit (flake, black, etc, etc) - we just use our catalyst-codestyle
  • introduces minGPT_coef to be able to train tinyGPT on tinyshakespeare (it works on CPU-based mac easily!)
  • rewrites your example a bit to fit the codestyle and make it more readable - I just remove magic config from everywhere
  • adds Catalyst-based example as an alternative solution for "Pure PyTorch" or "Custom Trainer" variants - you should have a choice, is't it?)

Generally speaking I was just trying to introduce Catalyst support for repo with minimal number of code changes.

You can check tiny experiment results in the notebook. Time-per-epoch and final loss looks close enough (but from my perspective it still looks like Catalyst is 1 second-per-epoch faster and have a bit lower. loss).

PS. Maybe, we will create a comparison tutorial with this example.

Scitator avatar Aug 20 '20 21:08 Scitator

My default is to dislike monolithic (Keras-like) frameworks and I also dislike forced coding styles that expand out code into billion lines of code and whitespace. I don't find it more readable at all.

That said I understand this is a strong matter of preference. I will not be merging this PR into this repo but please definitely feel free to adapt any of this code in whatever way you like, build on it, include it into Catalyst examples, etc.

Thank you!

karpathy avatar Aug 21 '20 02:08 karpathy