[WIP] codestyle and Catalyst example
Dear @karpathy , I am Sergey, author of Catalyst - high level framework for deep learning research and development on top of PyTorch.
Recently, my friends told me about "high-level frameworks overselling" going here, so... I decided to join :joy:
Long story short, this PR
- fixes codestyle a bit (flake, black, etc, etc) - we just use our catalyst-codestyle
- introduces
minGPT_coefto be able to train tinyGPT on tinyshakespeare (it works on CPU-based mac easily!) - rewrites your example a bit to fit the codestyle and make it more readable - I just remove magic config from everywhere
- adds Catalyst-based example as an alternative solution for "Pure PyTorch" or "Custom Trainer" variants - you should have a choice, is't it?)
Generally speaking I was just trying to introduce Catalyst support for repo with minimal number of code changes.
You can check tiny experiment results in the notebook. Time-per-epoch and final loss looks close enough (but from my perspective it still looks like Catalyst is 1 second-per-epoch faster and have a bit lower. loss).
PS. Maybe, we will create a comparison tutorial with this example.
My default is to dislike monolithic (Keras-like) frameworks and I also dislike forced coding styles that expand out code into billion lines of code and whitespace. I don't find it more readable at all.
That said I understand this is a strong matter of preference. I will not be merging this PR into this repo but please definitely feel free to adapt any of this code in whatever way you like, build on it, include it into Catalyst examples, etc.
Thank you!