Lets Get Small, AI

I’ve still been chipping away at Alan this past month. I have some pretty graphs and random things to talk about here, but it’ll be a bit disjointed. The overarching point of this post, I think, is that from my really limited playing around, I think AI labs (if they haven’t already) need to explore starting over with model development. Or at the minimum, there is an entry in the market to start a new type of lab (drop me a line if you are keen to fund that :D).

I am sure you’ve seen the graphic above in some form. That diagram shows that Artificial Intelligence is a blanket term that encompasses a lot of sub-disciplines and techniques. Recently the term has been co-opted to mostly just mean Large Language Models, but in the past few years there has been some amazing progress in all of the layers of the donut.

Since this whole thing is still pretty new in the grand scheme of things, and there have been so many new ideas and techniques, that I think there is a lot being left on the table. And if the big dogs started from a fresh slate with all the lessons learned in the past few years, I think the AI story could be quite different.

Alan V4

The last version of Alan can now run on an ESP32-S3 which draws around 24mA of power fully in RAM (using my custom OS). An ESP32s3 can easily be run off of USB power:

Yeah, no, this is not a frontier model by any stretch of the imagination, but there are many things this model can already do. Alan v4 is a 6M parameter model trained on meticulously cleaned Wikipedia text, had specifically designed tool calling, and when put in a harness it can do some pretty cool things for such a small model. (This type of work is called TinyML (Tiny Machine Learning) if you were curious).

Alan V5

I decided to push things a bit more and see if increasing the model’s parameter count, focusing more on good quality data, and revamping the tool design could make things work even better.

I bumped up the parameter count in AlanV5 to 23 million parameters. For comparison (and we will do that in a minute), GPT2 is a 124 million parameter model - that means GPT2 is 5 times larger. The size of AlanV5 weights on disk are about 24 MB, and when quantized weight in at 15 MB… that’s megabytes not gigabytes.

I don’t want to talk too much about the training data (as I might want to write up an actual paper about this), but know that it was source ethically and no one’s copyright is going to be infringed. As you can see in the following charts, it only took about 2 days to train on my RTX 5080 Ti:

And then about an hour to fine tune to get it to chat:

Model Evaluation

In my subjective playing around, Alan v5 is quite useful. And it really shines when run within it’s specifically crafted harness. Later on I’ll show that I did a fine tuning against a legal corpus for a side project I am playing with, but that is the “v5 legal” you will see in the following graphs.

Right now I am only benching it against GPT-2 which keep in mind is about 5 times larger than Alan. I benchmarked the models using the following datasets which are generally from Eleuther. They are the following:

You can see the results in the following graph and table. The dashed line in the graphic represents chance (if one were to just guess). GPT-2 is the blue column, and Alan V5 is the base model without any tools and not running in the harness:

To me, this is pretty incredible. That a model that is 5 times smaller can perform nearly as well as it’s big brother by focusing heavily on data quality and using new, modern techniques that have been created since GPT2 came out. And, heck, this is all before using tools which is my real focus.

When you run the model within the harness, it can do even better on some specific tasks:

Note for those in the know: the poor result with GSM8K is due to the model having very poor reasoning (it hasn’t been trained to reason in many steps yet, it can only one shot at the moment). So take that last graph with a grain of salt.

Conclusion

Who knows what the big frontier labs are doing right now, but I think it would behoove them to maybe try to build a whole new model from scratch. I can only imagine they’ve been, likely, piling data on top of data. For example, and to be a bit snarky, I can’t see how training on Reddit posts will lead to “super intelligence”. I know that the big dogs are going for a monolithic model that can do everything, requires gigawatts of power to run, and a googol of GPUs to do inference, but, as I’ve said before, I think there is much to be said - and a lot of opportunity - for going the other way.

(makes a phone gesture with his hand) Give us a call if you want to start a lab making mini-models.

References

@inproceedings{zellers2019hellaswag,
	title = {HellaSwag: Can a Machine Really Finish Your Sentence?},
	author = {Zellers, Rowan and Holtzman, Ari and Bisk, Yonatan and Farhadi, Ali and Choi, Yejin},
	booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
	year = {2019}
}
@article{allenai:arc,
	author = {Peter Clark  and Isaac Cowhey and Oren Etzioni and Tushar Khot and Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord},
	title = {Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge},
	journal = {arXiv:1803.05457v1},
	year = {2018},
}
@inproceedings{Bisk2020,
	author = {Yonatan Bisk and Rowan Zellers and Ronan Le Bras and Jianfeng Gao and Yejin Choi},
	title = {PIQA: Reasoning about Physical Commonsense in Natural Language},
	booktitle = {Thirty-Fourth AAAI Conference on Artificial Intelligence},
	year = {2020},
}
@inproceedings{ai2:winogrande,
	title = {WinoGrande: An Adversarial Winograd Schema Challenge at Scale},
	authors = {Keisuke, Sakaguchi and Ronan, Le Bras and Chandra, Bhagavatula and Yejin, Choi},
	year = {2019}
}
@inproceedings{SciQ,
	title = {Crowdsourcing Multiple Choice Science Questions},
	author = {Johannes Welbl, Nelson F. Liu, Matt Gardner},
	year = {2017},
	journal = {arXiv:1707.06209v1}
}
@inproceedings{OpenBookQA2018,
	title = {Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering},
	author = {Todor Mihaylov and Peter Clark and Tushar Khot and Ashish Sabharwal},
	booktitle = {EMNLP},
	year = {2018}
}
@inproceedings{clark2019boolq,
	title = {BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions},
	author = {Clark, Christopher and Lee, Kenton and Chang, Ming-Wei, and Kwiatkowski, Tom and Collins, Michael, and Toutanova, Kristina},
	booktitle = {NAACL},
	year = {2019},
}
@inproceedings{paperno-EtAl:2016:P16-1,
	author = {Paperno, Denis  and  Kruszewski, Germ\'{a}n  and  Lazaridou, Angeliki  and  Pham, Ngoc Quan  and  Bernardi, Raffaella  and  Pezzelle, Sandro  and  Baroni, Marco  and  Boleda, Gemma  and  Fernandez, Raquel},
	title = {The {LAMBADA} dataset: Word prediction requiring a broad discourse context},
	booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
	month = {August},
	year = {2016},
	address = {Berlin, Germany},
	publisher = {Association for Computational Linguistics},
	pages = {1525--1534},
	url = {http://www.aclweb.org/anthology/P16-1144}
}
@inproceedings{chalkidis-etal-2020-legal,
	title = "{LEGAL}-{BERT}: The Muppets straight out of Law School",
	author = "Chalkidis, Ilias and Fergadiotis, Manos  and Malakasiotis, Prodromos  and Aletras, Nikolaos  and Androutsopoulos, Ion",
	booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2020",
	month = nov,
	year = "2020",
	address = "Online",
	publisher = "Association for Computational Linguistics",
	doi = "10.18653/v1/2020.findings-emnlp.261",
	pages = "2898--2904"
}