- Starter AI
- Posts
- Deep Dive: Building GPT from scratch - part 8
Hello and welcome back to this deep dive series on Starter AI!
Today, we’re finally jumping into the Generative Pretrained Transformer (GPT) to Gen AI infinite Shakespear on demand, re-implementing nanoGPT. You’ll be glad to see all the new skills from the previous 7 parts come into play now!
Due to popular demand, we’re breaking this lecture into two, one-hour sessions. Enjoy!
The roadmap
The goal of this series is to implement a GPT from scratch, and to understand everything needed to do that. We’re following Andrej’s Zero To Hero videos. Catch up here:
Neural Networks & Backpropagation part 1 - 2024/02/09
Neural Networks & Backpropagation part 2 - 2024/02/16
Generative language model - bigrams - 2024/02/23
Generative language model - MLP - 2024/03/01
Generative language model - activations & gradients - 2024/03/08
Generative language model - backpropagation - 2024/03/15
Generative language model - WaveNet - 2024/03/22
To follow along, subscribe to the newsletter at starterai.dev. You can also follow me on LinkedIn.
GPT - groundwork for tiny Shakespeare
Today’s lecture is called “Let's build GPT: from scratch, in code, spelled out.”, and we’re tackling the first hour.
We’re building a small GPT that we’ll train on Shakespeare’s works (a toy dataset under 1mb, so about a million characters), that will end up re-implementing nanoGPT, which in turn is a rewrite of minGPT.
NanoGPT reproduces GPT-2 124M on OpenWebText (the actual OpenAI training set is not open, so this is best-effort reproduction of the dataset), and also allows you to load the official OpenAI’s weights. All that in about 300 lines of PyTorch for the model, and another 300 lines for the boilerplate training loop. Easily one of the coolest repos on github at the moment.