The what
I'm gonna follow CMU courses, cuz most of them have at least their lecture slides available publicly. For now, I plan to do these in sequence:
- 15-281: AI
- 10-315: Intro to ML
- 11-785: Intro to DL
- 10-714: DL sys
- 11-868: LLM sys
- 11-777: Multimodal ML
- 10-625: Convex Optimization (maybe)
Update: Other great classes and resources below!!!
That being said, I'm not just gonna look at the lecture slides/videos/readings, I'll very likely look outside too. The courses are more like a structured index of topics that are probably helpful to know about. The most important thing would be to implement everything I learn, cuz if I cant implement it then I probably don't understand it. Here's an overview of each class and what they cover (as I understand them):
15-281 AI
The broad topics as seen on the schedule are:
- Agents, search spaces & problems
- Uninformed and Informed search
- Adversarial Search
- Classical Planning
- Constraint Satisfaction Problems (sounds like linear programming?)
- Convex Optimization
- Integer Optimization
- Bayesian Networks
- Markov Chains, Hidden Markov Models
- Markov Decision Processes
- Reinforcement Learning
- Game Theory
10-315 Intro to ML
Oh boy. I've "officially" taken this class (I took 281 too but dropped it right at the end, and I dont remember much from it). But as always, I wasn't able to learn as much as I'd liked to have from "taking" the class, so I'll do it again myself. This was just last sem so I'm a lot more familiar with the topics:
- KNN
- Decision Trees
- Linear Regression
- Kernel Regression
- Bayes Optimal Classifier
- MLE/MAP
- Logistic Regression
- Ensemble Methods
- MLPs
- CNNs
- PCA, clustering
- Gaussian Mixture Models
- SVM
- Learning Theory
11-785 Intro to DL
I've only heard stories of suffering from people who took this class. Here's what I think the class covers from looking at the lecture topics:
- ANN training (backpropagation, SGD, momentum, normalization, dropout, etc)
- CNNs
- RNNs and LSTMs
- LMs, transformers
- VAEs
- Diffusion
- GANs
- Graph neural nets
10-714 DLsys
This was a class I came across,, I think on youtube? (sometimes the recs r good). I thought this would be a perfect class for me to learn about DL, cuz I love systems and it has both "deep learning" and "systems" in its title.
- Backprop, autograd
- NN library implementations/abstractions
- Hardware accelerators, GPUs
- CNNs & their implementation
- RNNs & their implementation
- Transformers & implementation
- Distributed training (brief)
- GANs & implementation
- Model Deployment
11-868 LLM sys
This one seems to be more focused on LLMs (well, that's obvious from the title).
- GPU programming, autograd, DL framework design
- Transformers, Tokenization, LLaMA, GPT3
- Acceleration of transformers on GPUs
- Distributed training
- Serving (Triton, LightLLM)
- Quantization & compression (GPTQ), LoRA & QLoRA
- ZeRO
- Ocra
- PagedAttention
- Jax
- MoE, FlashAttention
- Speculative decoding, RAG
11-777 MMML
"Multimodal ML". This shud act as a good segue into other kinds of models than just LMs. I really want to look at diffusion models, but it's impractical for me at this moment to jump unto those. I'm not going to talk about the material, because the topics are pretty much alien to me (feel free to have a look yourself!).
10-625 Convex Optimization
This is a weird one. I knew about this class cuz a friend had some trouble getting in, and we talked about it at the time (it was quite an amusing chain of events). At a glance, it looked like this would be an important class for learning more about training. I asked him whether it's worth it, and he said:
He's a ML guy and I don't know much about this, so I'll have to take his word on that. But it doesn't hurt to learn, right? So I might do this one too.