Building Small GPT AI Models at Home

Abstract

This article documents my attempt to build a GPT-2 model using data from Project Gutenberg, tokenizing the text, and training the model on a consumer grade gaming PC with a 5080Ti GPU. After about 14 hours, the result isn’t ChatGPT by any means, but it’s on it’s way to being a useful part of a larger system. The over arching point is I think the future of AI isn’t just about massive frontier models. I think there isn’t just a niche, but a need for small, specialized models. Software engineers should be able to build and deploy specific models for specific tasks instead of, for example, fine tuning other people’s models.

Introduction

There are many tasks that do not require genius level IQ to get a a good result. In fact, sometimes being too smart can be a hinderance. For example, if you just need to classify a type of tree leaf, having to run that through neurons that know the python programming language or PhD level mathematics is likely just causing noise and wasted space.

There can be some patterns we don’t understand that do indeed lead to a better result when having cross domain knowledge. For example, “thinking mathematically” can help work through other kinds of problems, but that extra knowledge comes at a cost. For a human, that cost could be more money, more food, slower result time (because they think more deeply about the problem), etc. For a machine learning model that cost is more compute cost, more training time, more data collection, more disk space, and often times a worse result.

Thats something that doesn’t get understood enough. You can very often get cheaper and far better results with a model designed specifically for a task then by just “sending it” to ChatGPT or Claude. Data scientist have been saying this all along, but in the last few years of AI hype, they often just get ignored.

It feels like the tide is turning from the large do-it-all frontier models, to more bespoke, domain focused models. A change in the wind, says I. Be it the, somewhat overblown, large data centre aversion¹, the increasing of token costs, the RAM shortage, or just the typical tech cycle - “time sharing big iron” turns into “box in my house”.

I don’t think the frontier models are bad or a waste of time mind you. They should continue to chase AGI (Artificial General Intelligence), but there is more engineering work, much better results, competitive advantage, and actual ownership building small focused models.

📝 if you want to build software, you should learn how to build models. While LLMs like Claude are invaluable to help build these models, you still have to have physical presence to understand if the are working, to collect data, and verify the results. While an AI agent can arguably replace a web developer right now, it can’t replace a model builder (yet, of course).

While I have a good amount of experience building several different types of models for different people, and I have built a GPT model that did midi music, I hadn’t tried to build a language model. I am working on something where I need one, and I thought it would be good to see if I could reproduce and train ChatGPT2.

Methods

Data

When training a model you need data. Data is the most important. Because machine learning models find patterns in data, without good, clean data, you have no chance of building anything useful.

I want to highlight the importance of data because if you come from an engineering background you likely think that an AI model is some hand crafted, complicated mathematical algorithm. Maths are involved to be sure, but it’s more like, a stock market ticker application. The application is completely and utterly useless without the stock data.

I have a very specific reason for building my language model, but for this test I wanted to use something that I could talk about publicly and test some limits of my setup so I used a copy of Project Gutenberg.

Depending on what you are doing, you might want to start with a different dataset (for example TinyStories is a good starter dataset), but the base of a language model needs to understand how text works. The first bit of training is just teaching it how, like, nouns and verbs go together. So giving it some dataset to learn how language works is step one.

I did not clean up the data in any way. This is not good practice because, for example, the start of almost all of these books is something like:

The Project Gutenberg eBook of Gambara
  

This eBook is for the use of anyone anywhere in the United States and
most other parts of the world at no cost and with almost no restrictions
whatsoever. You may copy it, give it away or re-use it under the terms
of the Project Gutenberg License included with this eBook or online
at www.gutenberg.org. If you are not located in the United States,
you will have to check the laws of the country where you are located
before using this eBook.

Title: Gambara
Author: Honoré de Balzac

Release date: October 16, 2004 [eBook #1873]
                Most recently updated: June 25, 2026
Language: English

*** START OF THE PROJECT GUTENBERG EBOOK GAMBARA ***

Which can (and does) lead to the model learning some slightly wacky patterns.

And we also run into the first hurdle trying to train one of these models. The storage:

rob@weak-gpu:/mnt/labelstudio/raw$ du -sh ./cache
13G     ./cache

While 13GB isn’t that large by today’s standards, this can, and will escalate quite quickly.

We have the raw text, but what we have to do for training is tokenize the text. Yeah, that kind of tokenize - the I pay X for Y amount of tokens kind of tokens. For GPT2 and using the HuggingFace transformers library, this is very easy:

from datasets import Dataset
from transformers import GPT2TokenizerFast

tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token

# Single streaming pass: load -> chunk -> tokenize -> flush to an on-disk
# arrow cache every writer_batch_size examples. No stage ever holds the
# whole corpus, a whole book, or a whole raw-text dataset in RAM at once.
dataset = Dataset.from_generator(
	iter_tokenized,
	gen_kwargs={
		"pg_dir": INPUT_DIR,
		"tokenizer": tokenizer,
	},
	writer_batch_size=WRITER_BATCH_SIZE,
)
dataset.save_to_disk(OUTPUT_PATH)

When you’re done (assuming you have enough RAM), you’ll end up with even more data you’ll need to store:

(base) rob@weak-gpu:/mnt/labelstudio/raw$ ls -alFh train_data.arrow/
total 13G
drwxrwxrwx 2 rob rob  32K Aug 26 11:34 ./
drwxrwxrwx 5 rob rob  32K Aug 26 20:33 ../
-rwxrwxrwx 1 rob rob 1.1M Aug 26 11:34 cache-84773c119d774495.arrow*
-rwxrwxrwx 1 rob rob 106M Aug 26 11:34 cache-b508e9ab5cd755a2.arrow*
-rwxrwxrwx 1 rob rob 454M Aug 26 11:27 data-00000-of-00028.arrow*
-rwxrwxrwx 1 rob rob 475M Aug 26 11:27 data-00001-of-00028.arrow*
-rwxrwxrwx 1 rob rob 453M Aug 26 11:27 data-00002-of-00028.arrow*
-rwxrwxrwx 1 rob rob 463M Aug 26 11:27 data-00003-of-00028.arrow*
-rwxrwxrwx 1 rob rob 465M Aug 26 11:28 data-00004-of-00028.arrow*
...
-rwxrwxrwx 1 rob rob 429M Aug 26 11:29 data-00027-of-00028.arrow*
-rwxrwxrwx 1 rob rob 1.3K Aug 26 11:29 dataset_info.json*
-rwxrwxrwx 1 rob rob 1.8K Aug 26 11:29 state.json*

But once you have the .arrow file setup, you are ready to train the model.

Training Setup

The computer used for training had the following specs:

CPU: AMD Ryzen 5 3600 6-Core Processor
RAM: 16gb
GPU: 5080Ti 16gb

📝 It’s a testament to how far frontier models have progressed. It would be impossible to train something like Fable at home (unless you’re loaded). But you can train GPT2 on a 16gb 5080Ti, and they are only on GPT5. That is incredible to me when you consider the difference and amount of time between Mac System 7, Mac OSX, and macOS 26. The acceleration seems unreal.

The python code to train a GPT2 model is ridiculously easy now. For these types of well troddened models, the code has been abstracted up to almost a single function call. If you are new to this whole thing, I think this is a nice way to start to learn. It’s very high level, and from here you can start digging into building your own models with pytorch or tensorflow. Here is the code:

import torch
from transformers import (
	GPT2Config,
	GPT2LMHeadModel,
	GPT2Tokenizer,
	Trainer,
	TrainingArguments,
	DataCollatorForLanguageModeling,
)
from datasets import load_from_disk
# Load Dataset
dataset = load_from_disk(OUTPUT_PATH)
train_dataset = dataset.train_test_split(test_size=0.01, seed=42)["train"]
# Model Config
config = GPT2Config(
	vocab_size=50257,
	n_embd=256,
	n_layer=6,
	n_head=4,
	n_inner=4 * 256,
	max_position_embeddings=256,
)
model = GPT2LMHeadModel(config)
# Training Args
training_args = TrainingArguments(
	output_dir="./checkpoints",
	per_device_train_batch_size=32,
	gradient_accumulation_steps=4,
	num_train_epochs=1,
	max_steps=100000,
	save_steps=1000,
	save_total_limit=3,
	logging_steps=100,
	learning_rate=5e-4,
	weight_decay=0.1,
	warmup_steps=100,
	fp16=torch.cuda.is_available(),
	load_best_model_at_end=False,
	resume_from_checkpoint=None,
	report_to="tensorboard",
)
# Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token
# Data Collator
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)
# Trainer
trainer = Trainer(
	model=model,
	args=training_args,
	train_dataset=train_dataset,
	data_collator=data_collator,
)
trainer.train()
trainer.save_model(os.path.join(OUTPUT_DIR, "final"))
tokenizer.save_pretrained(os.path.join(OUTPUT_DIR, "final"))

This code is not optimal (for example there is no eval_dataset), but it should work to test the overall training process and get some kind of verifiable result.

📝 Side note: this is a ~17 million parameter model. Parameters are the connections between the neurons. While the whole point of this exercise is to build smaller, more focused models, it’s easy to get excited and think you can build your own coding assistant or something. I mean, you could try, but here is a quick parameter count comparison with the latest models as of this writing:

These numbers are very difficult for humans to comprehend. A good way to think of this is using time instead. If we were to use seconds instead of parameter count:

• One million seconds is about 11.5 days
• One billion seconds is about 31.7 years
• One trillion seconds is about 31,700 years

Training Run

Most of the models I’ve built in the past few years have been able to be trained directly on my laptop (or occasionally use colab when I need more horse power). They normally take an hour or two to train, so I normally just babysit the process and watch youtube or read a book while I wait. However, I wanted to specifically test training a model at home for multiple days.

There are many ways to actually run the training code at home. In the past I have built my own Kubernetes cluster, I’ve looked at setting up a Slurm cluster, and the more home lab-ish cluster LPJS. If you are into DevOps or are building a proper lab, have a look at some of those. However, I used a very low budget route of a single old gaming PC, scp, tmux and Makefiles.

I scp’d the code to the gaming PC, started tmux on the other computer, and just ran the training code right from python:

train:
	mkdir -p $(CHECK_POINTS)
	python ./src/train.py \
		--dataset-path $(TRAIN_DATA)/train_data.arrow \
		--output-dir $(CHECK_POINTS) \
		--hidden-size 256 \
		--n-layers 6 \
		--num-training-steps 100000 \
		--save-steps 1000

dash_board:
	tensorboard --host 0.0.0.0 --logdir $(CHECK_POINT_DIR) --port 6006

You may have noticed the line report_to=“tensorboard”, in the Trainer source above. That makes the training code log data into a format that tensorboard can view. Tensorboard is a simple dashboard that one can use to monitor training, and you can see the command to run it above as well.

I had a few set backs getting this to train to completion. Initially it ran for a good 26 minutes, but then the computer ran out of disk space while writing checkpoint files. I moved the checkpoints to a 1TB drive and started again, and the second time it ran overnight for ~14 hours

You can see that is a healthy loss curve. With the loss value around 3.7 there is quite a bit of room to go still (as stated above I didn’t add a eval_dataset; I will next run).

I was surprised to find I wasn’t even pushing the GPU very hard either:

For comparison here is the card when running my local Qwen model for inference:

There is a lot of room to tune the training GPU RAM usage. I only used about half, which means you could probably train GPT2 on a card with even less than 16GB of RAM.

Results

Many people were not paying attention when GPT-2 was released. This was before the AI hype really started kicking in. The only thing GPT-2 could do was complete the next token. In order to get the model to respond to questions, it would need to be trained on a lot of question-and-answer pairs.

With that being said, here are some interesting results from the model that I hope helps you get an intuition for how these models work.

The prompt: “What is the capital of France?”

======================================================================
  Text Generation
======================================================================
  Prompt     : 'What is the capital of France? ' 
  Max tokens : 128 
  Top-k      : 40 
  Temperature: 0.7 
  Seed       : 42
======================================================================
  What is the capital of France? _______________________ FOURTH ARTICLE [II-II, Q. 33, Art. 
1] Whether it be for us to say that the kingdom of France is its own, or that it is for us 
to say that it is a part of the kingdom, and that it is for us to say that it is the 
kingdom of England. But the kingdom of England is full of all the glory of the kingdom of 
France. And it is for us to say that it is the kingdom of France, and that it is for us to 
say that it is a part of the kingdom which is the kingdom of France and of France.
======================================================================

While this is nonsense, you’ll notice that it’s bringing up data from France and England. It also seems to liken them to a kingdom. The model is, of course, not fully formed, but you can see that some relationships that make sense to us are starting to form. And you can see some of the blowback from not properly cleaning the data too.

The prompt: “Once upon a time in a forest "

======================================================================
  Text Generation
======================================================================
  Prompt     : 'Once upon a time in a forest ' 
  Max tokens : 128 
  Top-k      : 40 
  Temperature: 0.7 
  Seed       : 42
======================================================================
Once upon a time in a forest urchin he was not at home, for he had been in constant danger 
of being caught by the deer. The deer were the deer that had been killed and killed in 
several parts of the jungle. He was a hunter, and he was his father and mother, and had been 
killed. The deer were so tame that he was soon killed by the deer. But he was not so tame as 
to have killed one of the deer, nor any other hunter could have killed the deer. He was not 
an old man, and his father was killed, and he was not very tame. He was a hunter, and was a 
hunter of the
======================================================================

Similar to the France prompt, a forest seems to be close to deer, hunter, and jungle. While a lot of it is gibberish as well, it seems to be getting closer to forming complete sentences: “He was not an old man, and his father was killed, and he was not very tame.” - not great, but I get it.

The prompt: “The number of legs of a spider is "

======================================================================
  Text Generation
======================================================================
  Prompt     : 'The number of legs of a spider is ' 
  Max tokens : 128 
  Top-k      : 40 
  Temperature: 0.7 
  Seed       : 42
======================================================================
The number of legs of a spider is iced to the same size as in the case of a spider. The 
animal may be caught in the water, but the animal may be caught in the water and its claws 
are tied together with their hands. The dog may be caught in the water and after being 
bitten by the tail, the animal may be caught in the water, and may possibly be cut in 
the water. The animal is a very large animal, but not much bigger than the other animal is 
commonly killed. The animal has a large head, which he is not called a spider, and who is 
not very high, is very much alarmed at all the movements of the animal
======================================================================

Well, at least it’s a great nonsense machine.

After training was all finished, we are left with weights which are about 68M in size (unoptimized).

-rwxrwxrwx 1 rob rob  68M Aug 26 21:12 model.safetensors*

Conclusion

I was very pleased with this result. The language models that I am building next are actually going to be smaller than this model (a smaller vocab_size) so I find these results quite promising. I am also going to train this for a few more days to see what happens.

Building your own GPT model isn’t going to compete with Anthropic anytime soon, but I think there is an opportunity to go low while they are going high. Parameter count and benchmark scores aren’t really what we are here for. We’re here to solve engineering problems and chew bubble gum.

Frontier models will continue chasing AGI, and that’s great. But I think the next wave of AI won’t come from prompting a trillion parameter model, but from engineers who understand that a small model trained on the right data can outperform a generalist model if you can get to a specific task. Of course, you have to have the skills to be able to build a model yourself - which any engineer is capable of learning if they try.

I feel like we are on the verge of a new “PC moment.” The question isn’t whether you can build small task focused models on the cheap, the question is: what will you solve?

The next Woz is out there somewhere.