Local Agent Vibe Coded Keylogger

     

I’ve been doing a lot of experiments and spikes using my local lab running a local LLM (on an Nvidia 5060Ti with 16gb) using my own AI harness (written by hand in a language other than python or javascript; thank you very much), and I decided to put it through it’s paces and let it try to code something itself.

I am close to releasing an old school digital audio workstation (called a Tracker) for Mac and the Steam Deck. It’s really fun, but if you are not used to trackers you’d probably find it impossible to use. To hopefully mitigate that learning curve, I decided to make some videos showing how to use it.

Ghost Installs via AI Harness

     

This is a feature and a bug. Somewhat scary, but also has the potential for being cool.

People are doing all kinds of interesting things with LLMs, but the original use case (and the thing I find them to be the best at) is translating from one thing to another; doing what is sometimes called a stylistic transfer. In fact, I believe the transformer architecture was created by Google when they were trying to make a better translate.google.com which should be an indication.

Testing a Spatial Memory Index for Strap

     

One of the pieces I still need to build for strap is a long-term memory store. As I’ve written about before, strap compresses context to work within small token budgets. But context compression only handles what’s in the current conversation. The bigger question is how to surface relevant memories from past sessions.

The obvious approach is brute-force cosine similarity over sentence embeddings. It works for a small number of stored memories, and it’s fast enough. But I got curious whether a spatial index could do better, and whether you could make that index inspectable at the same time.

Working Local Coding Agent

     

My use of AI coding agents might not be typical. I mostly find them useful for “rubber ducking” ideas, as a search engine, to build out latex styles, and for generating small POC code or examples. I don’t generally use them to add whole features to a codebase or “vibe code” whole products - although I have been playing with that a bit more lately.

I’ve been trying to build a local setup that will do what I need using as few resources as possible. To try to accomplish this, I’ve been using an old gaming PC, and writing my own harness to try to squeeze every bit out that I can. I think I’ve finally gotten something that works, and it works better than I expected.

Your Own AI on the Cheap

     

Harnesses are all the rage at the moment in AI. A harness is the environment in which you interact with a large language model (LLM). Some examples you may have heard of are Claude Code, Open AI Codex, OpenCode and Claude Cowork but there are many others. Some are focused around developers and writing code, and some (like Claude Cowork) are focused more on office tasks, making documents, posting online, etc. There are some for designers, CAD creating, 3d modeling, etc, etc.

你的代币怎么样？(How's your tokens?)

     

I’ve been having fun playing with coding agents, and I’ve been playing with some different ways, skills calling tools mostly, to enhance them at inference time. Unfortunately, to do my experiments I need to run them within the context of a coding session which gets very costly very quickly.

To try to get to a state where I can iterate, I decided to byte the bullet and try to get a local lab running. However, like most people I am GPU poor. The only thing I have is an old gaming PC with an Nvidia 1660 Ti with 6GB of RAM. That isn’t enough to run just about anything, but I am no stranger to constraints.

Basis - A Lisp Neural Network

     

Claude Code is absolutely insane. For a builder-of-software-and-crazy-ideas, I am unsure of another way to express how incredibly addictive this tool is.

However, it could very well be that I have fallen down one of the oft described rabbit holes. Or maybe I have succumbed to AI psychosis. At this point I am not sure. Like the man who thought he discovered a new form of mathematics because ChatGPT convinced him, perhaps I am falling into that trap as well. Somebody please let me know if I’ve gone off the rails.

The Unreasonable Effectiveness of Claude Code

     

As I mentioned in my last post, I very much enjoy writing code “by hand”. It is the only thing I have consistently dedicated the last 30+ years of my life to. In fact, I have a hard time understanding what software engineering / building software products would even be if one didn’t have a solid grasp of, let alone enjoy, writing code.

However, the reality is that the point of building software isn’t to build software. The point is to make a product or a business. It’s to accelerate some real world process or enable some kind of commerce. Engineers often get hung up on linters, syntax formatting, some made up abstraction, or big O time complexity of some batch process that runs all night. While those things have a place, the end goal of software is to increase shareholder value and make the company money. It’s not my favourite thing to think about, but it’s the truth.

We'll All Vibe Code; It'll Be Anarchy

     

The past few years I’ve been busy learning the fundamentals of machine learning and building my own custom models. Small models of course. SpaCy models, some small CNN models, a few hand rolled ones, and some neat little ML processes using pre built speech-to-text and OpenCV bits and bobs. Most of the things I have been building are specific, proprietary and sensitive so I haven’t really tried “vibe coding”. I’ve used LLMs to summarize some topics, do topic searches and what have you, but I hadn’t really tried letting an LLM write code “for me”.

Better At Research & The ARC Prize

     

Getting Better At Research

Last semester I sat a course on how to properly write a research paper. The course was extremely useful as I believe my academic writing has, in the past, been quite lacking (and I refuse to use LLMs to “fix” my papers). There were many, many insights I took from the course, but one of the best ones came towards the last few weeks: “You should keep a daily journal of your research project.”