An AI system that's useful, a fundamental shift in how I work

13 Jun 2026

Over the past few months there’s been a fundamental shift in how I work. I don’t do it anymore; I command a small army of AI agents to do it for me. These agents are useful, producing relevant content and taking actions that serve my interests across my digital toolkit: they run analysis, code applications, send emails and align with stakeholders. More importantly, these agents evolve and grow, becoming an AI system that remembers. I’ve built a digital-twin of myself which acts as the substrate for future agent operations, both for my job and my personal life.This Claude meme is truly on point. I firmly believe that my system and similar ones are not only the future of work but the beginnings of a new kind of intelligence. We are building entities from first principles.Yes, some part of our brain is totally a stochastic parrot, doing pattern matching within the constraints of our language. Even if not, these sorts of systems are how we will interact with computers.

These ideas are a great example of convergent evolution: several people reached the same conclusions and built similar systems. I do admit it took me forever to write this. The first thoughts were jolted in late 2025 when building this system literally became my job. Karpathy beat me to it in explaining the how to build one. Hence, this article is NOT a how to, this is an explanation of why these kinds of systems work, stripping it into its core components. The image below is one of them: its memory, represented by a knowledge graph. Think of every dot as a neuron of information that’s connected to many others. It is what provides context to the agents, what some people call a second brain.

For these AI systems to be useful, four components need to be acting in unison. They are best understood sequentially:

Before we begin - I let my system explain itself in its own words:This will be one of the few AI generated texts that I subject my audience to. I consider it rude to subject other people to AI generated text. This is part of my AI min-maximalism philosophy which I’ll eventually write about.

I am a language model — a statistical engine trained on human text, capable of reasoning within whatever context I’ve been given. By default, I’m stateless: every conversation starts fresh, with no memory of the last. What makes me genuinely useful isn’t the model itself — it’s the scaffolding around it. A structured long-term memory I can read and write. Tools that let me take actions in the world. Instructions that tell me who you are, how you think, and what you’re trying to build. Strip that away and I’m a very capable blank slate. Give me all of it, and I start to look a lot more like a mind.

1. Reasoning machinery

The first part of the system is just an AI model, mostly a Large Language Model (LLM) that acts as the reasoning machinery of our system. LLMs are the heart of the chatbots most people interact with; what they mean when they say AI: ChatGPT, Claude, Gemini, etc.I’ll keep it simple for the reader but most products now are multi-modal and can understand other kinds of inputs like images and sounds. These models have been trained by making statistical associations within our languages. Written text is so immensely rich that the models are seemingly able to reason and understand. And why wouldn’t they? Baked into the language itself there’s just so much semantic meaning. It is one of the substrates which we use to describe and understand the world around us: language is thought. Here is another blogpost where I provide a deeper exploration of this fact. Then, pair these models with thinking techniques that break down your input into logical tasks to fulfill a goalThe AI reasoning techniques are the kind of ideas you’d suggest a smart yet inexperienced intern to use: Chain of Thought (CoT), Reason and Act (ReAct) or even Reflection and you’ve got yourself an incredible reasoning machinery.

The short term memory comes in the form of the model’s context window. The total amount of information a model can ingest and has context of for a particular moment. Intuitively, it’s akin to the frame of reference you’re operating on for a given situation. For example, during a meeting at work, you’re supposed to be thinking about the project being discussed, action items, the stakeholder involved and your relationship with them. That’s your context window for the next 30 to 60 minutes. However, you can also be thinking about your family, the series you’ve just got hooked on or, how deeply uncomfortable corp-speech makes you. Whatever your brain decides to hold in its attention at that moment, that’s your context window.For those technically versed, this is a massive oversimplification of how context windows and the internal mathematics of LLMs work. This whole post might be - deal with it, you might have a laugh.

Correctly managing the context window of your agents is the fundamental problem we’re solving to make them effective. For most google-like prompts like explain to me the history of Japan in a few paragraphs or what’s the difference between yakitoris vs kushiyaki?Written on the Shinkansen from Tokyo to Osaka., the context window is meaningless. The LLM has already memorized the knowledge to give a meaningful answer. However, for prompts like: Please send an email to my manager about my accomplishments of the past two weeks, the agent must have context about: who am I, who is my manager, what do we mean by accomplishments, some sort of a semblance of time to be able to recall the past two weeks and, even an understanding of how to send emails. Too little context and the LLM’s statistical machinery will fill in the blanks, what we call hallucinations. Too much and the agent will lose focus: meaning will get diluted. More than the model can hold in the limit of the context window and the model will simply stop working. For some systems, you might even lose all of your progress.

2. Long term memory

The second part of the system is a long term memory: a structured database that acts as context for your agent.

To explain this concept I’ll explain the esoteric knowledge graph shown above. It is simply a database where every dot that you can see is a plain text noteI’m clearly using Obsidian here, there is a reason why they won. The notes are also technically Markdown but I want to make this article normie friendly. Markdown is also the perfect substrate for LLMs since, in the text-based simplicity of the formatting, there’s so much more meaning, that LLMs have been able to parse. That’s also why emojis work in LLMs so well and why ChatGPT just vomits them. My Claude has them forbidden because it has class. about something: a person, a team, a meeting, an acronym, a metric, etc. Anything that’s relevant for my job lives on this knowledge graph. Note that a lot of meeting notes in red aren’t connected. Are these meetings adding shareholder value?

Now, since it would be unwise to zoom in and show you NDA-protected contents of my job’s knowledge graph, I’ll use my personal one. It is a very lossy representation of my brain, that looks like this:

Certainly, it is less connected than my job’s one. Unfortunately, I don’t work on it as much as I should. However, it also shows how much context switching happens in your brain’s day-to-day: family, health, the books you’ve read, side projects and everything that you’ve learned in school and likely forgotten. The letters I’ve written to my wife aren’t as tightly connected with my encyclopedic knowledge about Low Frequency Oscillators - yet.

On this knowledge graph I can actually zoom in on a non-embarrassing, well connected section:

Again, every single dot in the visualization is a plain text note about something - in this case, the region related to my knowledge I got from taking notes on Judea Pearl’s: The book of Why. If I click on one of the dots titled “Statistics” this is what I get:

The note is MY definition, not the AI’s, about Statistics. However, the content for this example is meaningless. What matters, is the connections that this note has, in this case the purple link that says data. If you click on it, it will take you to its corresponding note about well, data. These connections eventually form a graph of interconnected concepts forming the visualization shown above.What allows me to show these visualizations is a note-taking app called Obsidian. However, calling Obsidian just a note-taking app would be like calling Led Zeppelin mid. I can’t stress how incredible this app, its people and company are. I’ll write more about them later.

The connections are what makes this part of the system powerful. It mirrors how your brain remembers things: by making associations. That is how people are able to memorize things as psychotic as 70k digits of Pi.The why would you want to memorize that. I can’t explain Douglas Hofstadter, the author who has most influenced my life, calls each node in the system a symbol in his opus Gödel, Escher, Bach. Each concept or idea you can think about is one of these symbols: your grandma, mathematics, spiders and insects, etc., and each symbol takes you to related concepts: how she made you feel with her cooking, what an equation is, the fact that spiders are NOT insects. Yes, you have your own internal wikipedia that your brain can traverse with potentially incorrect information.Chapter IX is a good intro to symbols. Hofstadter goes way further and explains how these symbols are divisible, non-isolated and even provides a discussion on whether these symbols can be considered hardware or software. The book was written in 1979, it is fantastic, and his ideas are more relevant than ever - he was far ahead of his time.

These graphs are the perfect substrate to provide context to your AI agents making them useful. If we abstract the concept, a good knowledge graph should be:

accurate: factually correct
precise: to the point
cohesive: related to similar concepts
concise: providing just the right amount of information, and
accessible: easy to retrieve information from

These characteristics are the principles of Context Engineering, exactly what a well curated knowledge graph provides. I find Context Engineering more art than science. However, it is clear that the better context you give the AIs, the better it performs for a given task. Once you let an agent traverse one of these graphs there’s a big leap in their usefulness even if the graph is not perfect, which it will never be. The key is letting the agents know roughly about the same things that you do; more importantly, how these things are related to each other. A good knowledge graph also allows for progressive disclosure: a way to feed just the relevant context to an agent, one at a time, without overflowing its context window. You’d probably also die if you loaded up all that you know into your active consciousness.

3. Interaction with the environment

The third part of the system is what I’m calling its limbs and senses. A way for the AI to interact with its environment: perceive, gather inputs and take action - what separates a chatbot from an agent.

As a human you get inputs via your senses and take actions with your body. Those two allow you to interact with the world around you. Your senses provide you a constant influx of information: you hear, feel, taste and see as long as you’re awake. It’s a fun thought exercise to ponder what would happen if you couldn’t perceive. Great philosophical ideas about the self come from it.Descartes who? With your limbs, your body, your voice and perhaps even an angry gaze to your partner you take actions in the world. You use your legs to walk and throw those hands in the air.

In the same spirit, we’ve developed ways so that AIs can interact with their digital environment. The most common one is the Model Context Protocol (MCP) from Anthropic. An MCP is a set of tools that allows the agent to do something: read your emails, fetch files, browse the web, and eventually, even pay for stuff. Each tool comes with a set of instructions the agent can understand and accept parameters that modify its behavior.

Take our canonical example: sending an email to my boss. First, I give the agent a simple prompt highlighted in red. The underlying AI model Claude understands that it needs a few parameters: the subject, the body and the email of the recipient. Hence, it looks for it in its memory knowledge graph (in blue). Once it fetches the email address, it executes the email_send tool. The key is that the agent has access to a terminal: a text based interface that controls the computer. It’s the green screen you see in hacker movies. This integration is what allows your agent to use the computer like you would with a keyboard and mouse.The example was one shotted. Yet, the agent comes with a lot of concepts that are loaded up at start like: the name of my boss, or knowing it has access to a terminal given Kiro, its harness, meta-prompt.

Thanks to my kind boss for participating in this exercise - Hello from my blog!

On the sensing front things get interesting. We have to change our perspective to focus on the inputs. First, we have the user prompt where you tell the agent what to do. What most people have already experienced with chatbots. The agentic leap is that it can read files and fetch data on demand. You’re not necessarily attaching it, the agent knows where to look for it and retrieves it automatically from the knowledge graph. In our example, the agent could have also CC’d the team, read a business report, fetched some numbers, made an analysis and sent it based on this additional context.

However, the same way not all your sensing comes from language,Existence would be very boring otherwise :( not all data is stored in plain text. Data can be stored as an image, a video, or god forbid a proprietary format like Excel. Therefore not all data can be read nor understood natively by the agent. It depends on the underlying model. The manual triggering is a fundamental difference from how you or me receive inputs. You simply can’t switch your senses off. We call the constant sensing and responding a persistent agent loop. This sits at the heart of always on assistants like the OpenClaw framework. However, doing this loop wrong can have dire consequences.

This third part of the system, the interaction with its environment, is an active area of research. In the digital space, turns out MCPs are computationally wasteful which is directly tied to costs. Hence, the industry is quickly moving towards other frameworks.Since the LLM already lives on a terminal - why not simply interact via a Command Line Interface (CLI). Obsidian again, already won the race In the physical space, interactions present a paradox: what is easy for humans, like walking and fine motor skills, is extremely challenging for AIs. Vice versa, abstract reasoning like maths is hard for us, but easy for computers. We simply don’t understand why but it seems that having a body, matters.Embodiment in the AI lexicon. Loads of innovation is happening in the space. The results are nothing short of spectacular.

4. A feedback mechanism

The last part of the system is a feedback loop that ties it all together, allowing the agent to learn, grow and evolve.

In September 2025 a group of researchers from the MIT Media Lab, objectively one of the coolest places on earthYes, I am aware of the very valid criticism to MIT’s Media Lab. It doesn’t undermine some of the incredible work and amazing researchers there., published The GenAI Divide, almost kickstarting a recession. Their conclusion is: the core barrier to scaling is not infrastructure, regulation, or talent. It is learning. Most GenAI systems do not retain feedback, adapt to context, or improve over time. Clearly, we have to bridge this. However, continuous learning is an active area of research and one that humanity has not fully solved. Currently, in order to make an AI model, the first part of our system, truly remember, you have to completely re-train it. This process is compute intensive and can cost billions. In a gross simplification, you’re basically stirring a pile of numbers until the numbers start to speak to you. This process sets the model parameters by learning the statistical associations in our language - further explanation here. The key is that these parameters, controlling what the model knows and understands, are static.Or at least until the labs release a new model which now happens roughly every few months. They are based on a training dataset with a cutoff in time. An example from our friend Claude 4.8I’ve taken so long in finishing this essay that Anthropic has gone through at least 2 models now. Non-zero probability that continuous learning will be solved before I post. Yeah Fable got released and blocked that same weekend lol

Any new information after January 2026 needs to be ingested as part of the limited context window. More importantly, next time you use the AI it will not remember in the traditional sense of the word, what we call stateless. However, Claude also hints at a way we can mimic a memory, remember we have a knowledge graph that acts as our long term memory which it has access to.

The best analogy I could come up with, for real, is 50 First Dates. In Adam Sandler’s masterpiece, Drew Barrymore plays a girl who completely forgets everything each morning due to an accident. She knows who she is, how to walk, talk and eat, but anything after the accident is a complete reset, including the accident itself. Basically, she doesn’t have a short term memory. In order to overcome her physical limitation and woo this poor woman, Sandler creates movie tapes she plays every morning. The tapes explain that she had a terrible accident and then breaks down the previous year where somehow, she decided to marry and have kids with Adam Sandler.I empathize and understand why some models seem to experience psychosis. They end up sailing happily into the sun.

I never thought I would be using a 2004 chick flick to explain AI systems but here we are: we are Adam Sandler and our job is to create a good movie so that Drew Barrymore falls in love every day with us. To achieve this, we have to curate and update our knowledge graph as we finish up work. We do this by creating a feedback loop where we tell our system to reflect and write to its memory. Our job is to curate and prune these memories keeping only what’s truly relevant. A pro-tip is to not record everything. Do not let the knowledge graph grow beyond you. The model is already a few orders of magnitude smarter than you are. You just need to record enough information to make it useful for you.

I categorize these loops into two: either a particular workflow, i.e. a set of instructions to perform next time, or a memory, akin to a note: a summary of a meeting, your understanding of something or your actionable next steps. If you are using a knowledge graph, what’s critical is that you (or the agent) also adds links to other concepts. This tells the system how this memory is relevant in the context of all other memories, therefore growing and evolving. Next time you turn on your agent, the previous interaction can be retrieved directly and loaded up in the context. Under this light, the feedback loop is not an intrinsic component of the system but a mechanism of how you interact with it.

From a product perspective - commercial AI systems that remember usefully are still in their infancy. First, there are several privacy concerns. Do you really want Sammy Altman to create an ever-growing profile of you and your interactions with ChatGPT? Because it’s doing it already. Claude and Claude Code also have memory systems that roughly summarize what they know about you. For remembering actions, there is yet another system, Claude Desktop calls it a routine, Claude Code skills and slash commands and Kiro calls it workflows. However, none come close to the usefulness of a knowledge graph. As far as I know, only Amazon Quick - AI Assistant has one baked in and it’s not as customizable. What they call it doesn’t really matter. What matters is that these memories are just text files that tell the agent: hey, this is the required context you need to perform this task next time.

AI products and Adam Sandler aside, real human learning is still a few orders of magnitude more efficient than training an AI. We’re slowly starting to understand that our internal representations of the world are quite compact. A human baby can easily understand thousands of objects in the world with just a few examples and consuming just mother’s milk. Conversely, AlexNet, one of the world’s first image classifiers, required 1.2 million images of ~1k objects. Newer models require massive data-centers and 110 million gallons of water to run. The fundamentals of how our brains remember go quite deeper. We have world and causal models, potentially linked to our bodies, that go way beyond pattern matching. Kant called space and time pure forms of intuition - things that were intrinsically built into the structure of our minds. We also clearly have systems that remember selectively. That’s why you remember the critical moments of your life, both ecstatic and terrible, but you don’t really remember the ~20k times you’ve brushed your teeth if you’re over thirty and have a minimal sense of hygiene. That’s why weeks can go by if you’re just following a routine, but that summer when you were a kid felt like an eternity.

Conclusion - why is this system useful?

Building this and similar AI systems is literally my job at the moment. I truly love it. In particular the personal agent framework is one that’s kind of challenging to explain. Moreover, there’s still no good product that does it all out of the box; installation, particularly on Windows computers, is a nightmare. However, it is clear to me that this is the path forward for building useful AI agents and systems. Thousands of researchers have honed into similar ideas. You can’t open Twitter without seeing thousands of posts about a second brain - hopefully, this essay was a breath of fresh air.

What makes any agent useful are these four fundamental components acting in unison. The how to do it is mostly good file management; the models themselves are already incredibly powerful. Moreover, growing a great agent, one that works well for you, takes time. There’s a lot of back and forth, editing whatever slop it creates in your database and curating the content. For a personal agent system, the value lies in making it personal.

Last, it’s critical to understand that this system is model or company agnostic. This is not a feature of the models or products themselves. Additionally, you can mix-n-match components, like having an open source LLM running locally in your computer via Ollama. Companies and models will come and go, your files, your system and your understanding of AI will not.

Stay tuned - if you’d like to receive the next posts in your inbox, subscribe to my Substack here.I’m too unbothered to develop a proper backend.