DiscordLM (In Progress)
Designing and training a decoder-only transformer language model trained on Discord chat logs with the end goal of being a sort of Adit text generator. Built from scratch in Pytorch inspired by Karpathy's nanoGPT, OpenAI's GPT paper, and the original transformer paper.
Things I'm interested in finding more about:
- Character tokenization vs word/subword/n-gram
- Vocabulary size and its effect on model speed and efficacy
- Optimizing model performance given limited data
- Implementing optimizations like FlashAttention and AdamW Optimizer
The Future:
- Pivoting from training from scratch to finetuning something like Mistral 7B or Llama 3.1 on chat logs
- Optimizing and abstracting away training to release as a Discord bot
RunTune
A tool to augment running training sessions with playlists of music synced to your steps. Built in Python and React.
Things I learned:
- API endpoint design
- Frontend/backend separation
- OAuth2.0 framework for SSO integration (Spotify)
Potential Future:
- Integration with Strava for more accurate and personalized cadence data
Monkey Mash!
An animated top-down multiplayer sandbox shooter videogame built from scratch in Java using only standard library.
I designed and built the networking pipeline to make the game multiplayer using a custom packet format, multithreading, and sockets.
Shout out to Gabriel Kan for building the game mechanics and to Iris Tsai for drawing all the sprites and helping with animation!
Things I learned:
- Client side and server side networking system design
- Use of different internet protocols (TCP, UDP)
- Using multithreading to parallelize processes