Building Small GPT AI Models at Home Part 2

September 8, 2026

Abstract

This article is part 2 of building a tiny GPT-2 model, alan, using data from Project Gutenberg, tokenizing the text (with a small byte pair encoding model), and training the model on a consumer grade gaming PC with a 16GB 5080Ti GPU. You can read about the methods and initial results in part 1. After a week of cursed training attempts, the full process is completed, and a final, completed simple model is presented.

Introduction

As most people of action know (or learn eventually), the first iteration you do of something is likely going to be rubbish. You probably wont feel like that at the time, but over time you will come to understand that the first pass was likely pretty bad. In my experience, it is good to get those first versions out of the way so that you can iterate, and get somewhat good at whatever task you’re trying to do. Get stuck in; you wont truely know anything until you actually do it.

To set some expectation, that was the point of this exercise: to setup a process to build a GPT language model from end to end, try some training processes that would take days to run, and in the end have something that mostly works.

If you’re a TL;DR kind of person, you can play with the results here, but the following is the story of the last few weeks training the model.

Base Training

Where we left off in part one can be seen on the following graph as the black line. After I wrote the last post, I decided to resume training (the purple line). I messed up the learning rate when firing off the training resume which was the reason for the loss starting much higher than it should have. I decided to let it ride and see if the results would improve, and they seemingly did - dropping perplexity from $21.88$ down to $20.85$.

Blowout

It’s difficult to compare my little GPT2 model vs. the original GPT2 paper as Alan is about 20 times smaller, but it was seemingly headed in the right direction.

But then I tried to use the model’s last checkpoint (purple line):

prompt="What is the capital of France?"

What is the capital of France? **********___********** SECOND ARTICLE [II-II, Q.  
15, Art. 1, Art. 1, Art. 1, Art. 1, Art. 1, Art. 2, Art. 1, Art. 1, Art. 1, Art.  
1, Art. 1, Art. 1, Art. 1, Art. 1, Art. 1, Art. 1, Art. 1, Art. 1, Art. 1, Art.  
1, Art. 1, Art. 2, Art. 1, Art. 2, Art. 2, Art. 1, Art. 1, Art. 1, Art.

Yikes. Not only was it stuck in loops (which is going to happen with a small model), but it was becoming clear that if I wanted any kind of final demoable test I’d have to go back and clean the data (see original post for what that means).

In my defence I didn’t think I would push on for this long (half a week at this point), but, hey, that’ll teach me not to be lazy.

So I went back and made some scripts to clean up the data as best I could.

I then tried something that was also likely stupid, but I wanted to see what would happen. After I cleaned the data, I decided to resume training using the weights I had already trained. I figured, it might need to dig itself out of some gradient hole, but then would hopefully continue to learn.

That is the orange line in the graph above. Initially it looked like it was going to work, however when I woke up the next day:

Topout

It was clear it was folly. As you can see on the updated orange line, it was likely just going to oscillate back and forth.

Raw-ong, Do it Again!

Ok, so I blew away everything and started fresh training with fresh clean data.

Train Loss

Eval Loss

After about 4 hours the loss was already down to 3.3 - much better than the blow out from before. I set the training run up to train for several days, but I didn’t think it would have any real gains past a day or so of training. I figured it’d get to the point where it just hits the minima and then just hovers there.

Which you can see in the graphs above, is exactly what happened. When I woke up the next day it had just past that bump and didn’t look like it was going to go any lower. It looked like the model wasn’t improving, and was just going to oscillate again. Having a look at the grad_norm:

Train grad norm

It looks like it jumped out of whatever minima it was in. So I just stopped it at that point and grabbed the last best checkpoint: 197500.

Training Results

With the cleaned up data, the results of the model learning English structure was - unsurprisingly - much better. While the model still hadn’t learned any facts, you can see that the structure of it’s sentences, while still not very good, makes some sense:

======================================================================
  Text Generation
======================================================================
  Prompt     : 'What is the capital of France?' 
  Max tokens : 128 
  Top-k      : 40 
  Temperature: 0.7 
  Seed       : 42
======================================================================
What is the capital of France? She is the only man that her country has ever 
seen. She is the most impulse to tell you, and she is the most precious of all 
her life, and she is the most precious." "Ah, ah!" he cried, "you have never 
seen me, and you have always seen me. I know not how to tell you. I have seen 
you, and I have never seen you. If I had been a little older than you, I should 
have known you; but I have seen you, and you have seen me once." "What has happened? 
You have gone to the house
======================================================================

======================================================================
  Text Generation
======================================================================
  Prompt     : 'Once upon a time in a forest' 
  Max tokens : 128 
  Top-k      : 40 
  Temperature: 0.7 
  Seed       : 42
======================================================================
Once upon a time in a forest, he saw the man standing at the station, with gaze 
fixed on his face. "What's the matter with you?" "I don't believe you're going 
to go on." "You've got the pick of your way back to the office. I do believe it 
means something." "I've been telling you," said the man. "I've got one of the 
men who have been here in the cavalry and the cavalry and the men who have been 
here in the office were here." "You are going to look at that man? You've gone 
to the office
======================================================================

Oddly, I find this kind of nonsense generation much more creatively valuable. When I read some of those non-sensical answers, they inspire some more complex stories in my imagination.

“Ah, ah!” he cried, “you have never seen me, and you have always seen me.

and

Once upon a time in a forest, he saw the man standing at the station, with gaze fixed on his face.

Those seem like interesting starts to books to me.

Q and A Training

The last step in this pipeline test was to train it on questions and answers to try to make it act more like a chatbot.

I did another run using the Alpaca dataset, but didn’t actually record any graphs. And while the source code isn’t available, these were the settings I used (learning rate, epochs and scheduler) after converting the Alpaca dataset into my tokens, and basically just continuing the training run but only with the Alpaca data.

SFT_EPOCHS:=3
SFT_LR:=5e-5
python src/train.py \
	--dataset-path $(TRAIN_DATA)/sft_data.arrow \
	--output-dir $(CHECK_POINTS_SFT) \
	--tokenizer $(TOKENIZER) \
	--init-from $(CHECK_POINTS)/final \
	--max-positions $(MAX_POS) \
	--epochs $(SFT_EPOCHS) \
	--lr $(SFT_LR) \
	--lr-scheduler cosine \
	--warmup-steps 100 \
	--save-steps 500 \
	--eval-steps 500

Keeping in mind the underlying model isn’t that smart in the first place, you can see it gives it a go to answer all my questions immediately:

User: What is the capital of France? 
Assistant: The capital of France is a strongly populated county in Macedonia. 
[21 tok, eos]

User: Write a short sentence about the sea.
Assistant: The sea was still at the shore of the sea. 
[12 tok, eos] 

User: List three colors. 
Assistant: One color is red and blue. 
[9 tok, eos] 

User: Translate 'hello' into French. 
Assistant: Hello is a type of logistic style in which French is written. 
[21 tok, eos] 

User: Give me advice for a rainy day.
Assistant: I would like to have the rainy day to get a game. It's a great way 
to help you go. I am just looking for something I'm looking for. It's a great 
way to get a few times out of the world. I'm feeling a lot of work if I think 
you're not looking for something I've had to do. I'm looking for something I've 
been looking for. I'm here to help you go. [93 tok, eos]

Conclusion

While Alan has a ways to go to fit into my overall grand plan, I think this was a great success if only for the learnings alone. I have some solid things scribbled down that I want to try to improve Alan for the next run, but knowing that it’s possible to do something close is a good first step.

With out giving away everything, I think the two big things that will improve for the next run is 1) better, task focused data, and 2) a slightly more complicated model. My goal here is to make something that works reasonably well while as small, and narrow in scope as possible. We’ll see how we go.

If you’d like to have an interactive play with alan you can here.

References

Addendum

To really understand the scale of the computations involved in these things, watch this section on Large Language Models explained from 3Blue1Brown