GPT with a Little Help from My Friends

September 26, 2026

Introduction

Alan, my Tiny GPT model, is being built to be part of a larger system. It is not trying to be a monolithic LLM, but instead the GPT portion is just the terminal interface, if you will, into other models and deterministic systems. It is borrowing from ai harnesses, cognitive architectures, blackboard architecture, micro services, brain states, vector databases and boring old classifiers.

You can play with Alan in the brain here: Alan Harness.

This post is kind of part 3 (part 2, part 1) of training the GPT part of the system. I touch on the overall brain a bit, but this post is mostly about training the Tiny GPT model.

Methods

Data

Instead of using Project Gutenberg to train the model, I switched over to using a small subset of Wikipedia. My initial thinking behind the Gutenberg selection was that it would provide ample, well written text under a public domain license. I was hoping that the model would be able to find the patterns of the English language; I didn’t really expect it to “learn” any facts.

However, after basic training and playing with it, I found the patterns it learned were kind of wonky because, I think, most of the text was written in a story format, and it used a lot of old English - hence the public domain license. Additionally, in the back of my head I was hoping it leaned towards some general facts after seeing so much data. For example, my hope was that it would learn some basic facts found in romance novels like:

User: What is the capital of France? 
Assistant: The capital of France is the capital of France.

Well, I guess it’s not the city of love, but instead the city of snarky answers. Sheesh, thanks Alan.

Wikipedia

So, it was off to downloading and cleaning up the text from a copy of Wikipedia.

One thing that might be a bit more complicated for me is that Wikipedia is under the Creative Commons BY-SA license. Which, I think, means I have to do the following:

For these experiments that is fine. I don’t mind attributing all the output to Wikipedia for testing, the hard part would be if I added another source in addition to Wikipedia. Currently, there is no way to know what percentage of what training data influenced the weight of each neuron, and then how you’d inform any one of that at inference time. “This output is 24.32% Wikipedia, and 10.3% reddit, and…” is just mathematically impossible. Which I suppose is where all those debates are coming from.

New Model

My concern with the initial architecture of Alan was that it was too small to learn properly, but I am also specifically trying to target tiny devices so I don’t have a lot of wiggle room to tweak parameters.

It’s already using a quite small vocabulary size (4,096) so I didn’t have any room there, but after revisiting some of my school notes, and a few back and forth with Claude, it dawned on me that the key to deep learning is the deep part. I of course knew that from school - I remember the lesson with this tool specifically (try to make the spiral - but for some reason I didn’t consider it in this context. I guess that is what practice is for.

So, I made the model thinner but deeper (less neurons, more layers). That way I can keep the parameter count about the same, but make more folds in space (for lack of a better way to put that).

The new model structure is now:

MODEL_CONFIG = {
"vocab_size": 4096,
"hidden": 192,
"n_layers": 12,
"n_heads": 6,
"intermediate": 512,
"block_size": 512,
}

vs. the old structure:

MODEL_CONFIG = {
"vocab_size": 4096,
"hidden": 256,
"n_layers": 6,
"n_heads": 4,
"intermediate": 1024,
"block_size": 512,
}

Training

With the new Wikipedia data downloaded, cleaned up and tokenized, I fired off the trusty RTX 5080Ti with 16GB of RAM and let it run for over 2 days:

Alan Loss

What a lovely loss curve. What this curve shows is the model going through the data and trying to guess the next token. If it guessed correctly every time, the graph would be at zero. So this curve shows that it is learning, but the learning rate slows down towards the end. Using the value parameter we can calculate perplexity:

$$ e^{2.66} ≈ 14.3 $$

That means that the model gets about a 1 in 14 chance of picking the next correct word (according to the training data of course).

For fun, here are some of the other graphs reported by the training board:

Alan Board

Aside from looking cool, they don’t really tell us much other than I have a pretty weak GPU (at least as far as I understand it).

New Model Output

From a subjective, vibe based metric, This model performs much better than the original Alan model. As you can see it can make decent sentences:

prompt="New Zealand is"

New Zealand is the biggest city in Australia. Wallace Lynch was the first to use
a battery. It was built in 1865. It started as a small, small town. It was named
New Zealand in 1778. It became a town in 1886. It became a town in 1887. During
the 19th century, it was part of the National Park and a part of the Australian
Government. The town was named after the river. Pilgrims, and the Aboriginal
people use it to tour the countryside. During the winter, the town has a hot
summit

What is says is obviously nonsensical, and likely to get itself in a fight in a pub if it wasn’t careful.

However, it does have some moments of clarity:

User:    What is the capital of France?
Assistant: The capital of France is Paris.

User:    How many legs does a spider have?
Assistant: One legs is a fat in the front of a spider's back.

User:    Give me advice for a rainy day.
Assistant: One day, the rainy day is the sun as a sun rises in the darkness. It is rising in the
           coolest night, with a bike rising in the air. The rainy day is the night's
           moment, and the rainy day is an unusual day. The rainy day is so hot, with a
           sun rising in the darkness, with a rustling night.

We will always have Paris.

Interestingly when I was training the model I didn’t have a proper setting for the learning rate. It appeared that the gradient would jump in and out of the minima because the value was too high (this is a pretty common error). As you can see in the following graph (red line first graph):

Perplexity

the high jump at the last epoch a wasn’t very good sign, in fact that check point was remarkably bad. Luckily, I had setup the training process so the last three decent checkpoints wrote to disk. Another example:

Understanding

If you notice the contextual understanding fell off a cliff. Using the final model was terrifyingly bad. I thought that I had wasted another 2 1/2 days, but luckily checkpoint-495000 was hanging around on disk ready to save the day. Save your checkpoints kids.

You can play with the raw Alan v4 model (and compare it to the Gutenberg model: Alan), and, if you like, you can play with Alan’s brains Alan v4 Viz (and the original Alan Viz). However, you will find them lacking compared to frontier models; fun, but lacking.

Cognitive Architecture… Harness… Micro service…

Alan is great, but the main thing I am trying to research here is an interconnected brain type design.

I am not sure what to call it, but from what I have read I think it’s called a cognitive architecture or a blackboard architecture. I would probably call it a micro service or something like that.

The idea is that there are deterministic systems that get routed to based on the task. Alan is just the text interface into a decision model that then routes to a subsystem. Right now there is a simple maths, physics and facts subsystem.

Here is the basic high level idea:

Alan Harness

With Alan being trained on Wikipedia text it can do some “fast thinking” answers a bit better, it just knows about Paris, for example. However, it’s intelligence is very limited. It’s better than the last version, but it’s just not great.

With this subsystem architecture, it does a bit better with some types of questions. There is a lot more to say about this (like brain states, sleeping, and memory), but I will talk more about those when I have some better stats and graphs - or the whole thing fails spectacularly.

If you want to play with Alan in the brain: Alan’s Brain

References