The Bitter Lesson: How We Build and What We Build is about to change

--- type: transcript date: 2025-12-27 youtube_id: GyuwH3Q_FlQ --- # Transcript: The Bitter Lesson: How We Build and What We Build is about to change I think in 2026 we begin to split into two paths with software development. I think everything changes simultaneously. In this video I'm going to be going over the bitter lesson from Rich Sutton and specifically what I wanted to focus on is from these principles and this general philosophy how this will trend into software development given where we are and where we're quickly I think going to be in 2026. Now if you haven't read the bitter lesson before I really encourage it. It was out about 7 years ago now. And there are an incredible amount core ideas. In particular, there's about three paragraphs at the end here after going through a little bit of the history in terms of how AI has evolved over the past 70 years. And there are, I think, some critical lessons that are very true, especially given the environment today. First, what I want to touch on is what is the bitter lesson summed up in one sentence. General methods that leverage computation are ultimately the most effective and by a large margin. So if we go back to 1997, we had Deep Blue that beat Kasparro and this was through a brute force search process. Fast forward to 2016, Alph Go from Deep Mind wins through selfplay and scale. The thing with this moment is as soon as these systems became good at beating the best player in the world at Go, it quickly became better at beating any player at Go. Now, if we fast forward to today, if you've leveraged any of these AI tools that are out there, you'll know that the AI systems of today are increasingly capable. Now, you can argue, we can argue about where exactly that is. Is it at a junior developer? Is it below a junior developer? Is it an intermediate developer or something in between or something not at all? Now, one thing within the coding domain is the AI tools that are out there and the large language models that they're based off of as well as the different methods in terms of how to actually leverage these within the agent harnesses and what have you. These are increasingly more powerful and I don't think anyone denies that. So now to circle back to the bitter lesson. Now what exactly makes it bitter and this is one of the hard things actually to sort of gro initially but it's actually encoding knowledge. And what do I mean by that? This could be your knowledge. This could be you you reading different papers. This could be other people's ideas and actually encoding that knowledge within an AI agent. But when we try and encode our expertise and we design a perfect system, often times what we're doing is we're optimizing for what we know and the approach that we think we would take if we were say an AI agent. And the progress with this it can often feel smart, right? You're imparting your knowledge on the system and you can watch the system run autonomously and largely take actions as if you were taking them. And one of the issues with this is on a long enough time horizon this approach will lose in the end. Now what will win as a result actually leveraging computation. What do I mean by that? What if we just instead of imparting all of that knowledge is we just gave the agent general capabilities. Let's say we gave the agent access to something like a computer. Let's say we gave the agent access to particular tools to solve whatever we're trying to do within that domain. Further let's say we actually give that agent the capability to learn from the data. Maybe it's within the short term. This could be things like you might have seen within cloud code where it's offloading context to markdown files or actually a system that maybe has a little bit more memory or what have you. And over time it can actually discover what it's missed. And there can be that flywheel of continuous learning. And why this can win by an enormous margin is right now the capabilities of large language models and agentic systems once you give them the tools and you don't give too too much direction in terms of how to leverage the tools or what exactly to do and you just try and have them be determined. What's interesting is in observing these systems, especially when they have access to all of the different commands and the internet to be able to research and what have you, it's fascinating to see all of the different tools that it can reach for. Think about how you might reach for an open source repository or or a tool that you might plug in to just make your project go that much faster. An AI agent can do that just as well. And like I mentioned within the chess and go example, I think the lesson is actually inevitable that this is going to take place within code. I think LLMs and agents are going to ship better than the frameworks that we previously encoded. For instance, I might have built an AI agent a year ago and the approach that I would have taken a year ago is completely different to what I would do today. Today, in my opinion, the better approach is just understanding the capabilities of the model, banking on that they're going to get considerably better. And with that, actually taking that and using that as a method to understand the general methods and the compute that you can actually equip to these different agents and give them the ability to actually act, but not actually being too rigid in terms of what you encode for the agent. The idea with this is you can almost think of it like a self-driving car. put in a destination, say you want to go to the airport, and let the agent figure out how to get to the airport. Don't actually give it all of the instructions in terms of the particular route that it has to follow. Just let it find its way on its own. And the inevitable conclusion of this is this isn't a preference. It's not laziness. It's not mathematical. Any domain where data exists, the general method at scale is going to beat encoded knowledge every single time. Okay. So now let's touch on the history of coding tools. Over the past few years, we had tools like GitHub Copilot. This came out in 2021. We had line by line completion. We had things like being able to complete and write functions. It worked pretty well. And honestly, it was one of the things that I first saw that I was incredibly impressed by. From then, we quickly grew and over about 18 months to 2 years into tools that had the capabilities have multifile edit. Think things like cursor and a lot of other similar systems that were out there. Today we have systems like cloud code. We have Manis, we have codecs, we have things like Droid or AMP. And all of these different tools is an agent harness. And what those agent harnesses allow us to do is we can actually spin off these systems and they can run for minutes autonomously and even hours if we prompt it correctly. We give it the proper tools and hooks and iteration and goal. It can actually run for a considerable amount of time. What's incredible with this is in just four or five years, we've gone from the tab completion all the way through these autonomous systems that can run for hours at a time. And I think this is really just the beginning. And 2026 is going to make today's agentic systems really look like autocomplete. To borrow some inspiration from the bitter lesson, I think in 2026 we begin to split into two paths with software development. I think everything changes simultaneously. I think there's two streams how I'm thinking about this. There's going to be the how we build and then there's going to be the what we build. First up, how we build. This is changing very quickly. The fastest growing companies in the world are actually in and around code generation. Think things like cloud code. Think things like cursor, Devon, even some of the consumerf facing things like lovable or bolt. All of a sudden, we have all of these different agentic systems. And depending on the use case, we can leverage one or many of these tools to actually accomplish many of the same tasks. Now, what do we build is, I think, going to be the big one, probably arguably bigger than even the how we build in 2026. I think the products and the systems that we're going to start to see is going to go from no code builders to really agents just writing software. And I don't just mean software for software sake. I think there's going to be a lot of vertical use cases where the agents are going to write bespoke software even at the time of when it needs to do the task. Now, what do I mean by that? Let's say you're making accounting software. What we could do is we could say, "Okay, here are the particular goals and the outcomes that we're looking for within the system." And then let the agent determine all of the different steps in between to actually get there. Now, in the accounting step, you might go, "Okay, that's too much to actually feed to an AI agent." But if you give it the ability and the autonomy to actually validate its work and equip it with all of the different tools, especially in and around actually equipping it with something like a computer, you're going to actually be able to get, I think, very, very good results. First up, let's just dig into the two different streams, the how we build. So, personally, I like to leverage tools that are really strong agent harnesses right now. Think things like cloud code, codeex, there's tools like Droid, there's tools like AMP, there's open code, there's a number of different ways in terms of how you can actually leverage them. Now, a lot of them have the same sort of capabilities. They're very heavy on file operations. So, they can GP, they can write, they can read, they can leverage web search, code execution, so on and so forth. Next, in terms of the second stream, and this is one that I'm arguably more excited about, I think the progression, we can arguably all say that we've gone from manual code to having developers do everything. And I think the big difference was in and around the sonnet 3.5 era. Now, shortly after that, this is when tools like lovable and bolt came out. We had these noode AI platforms emerge and all of a sudden we had a model that could generate code reliably. And if we look at models now like between 12 and 18 months out these models are increasingly good at this much much better and able to run much more autonomously with that level of autonomy all of a sudden the way that we think about writing software and the back end it's all completely different now and I think the next era is going to be about agents actually writing tools for themselves or other agents. If you think about it, if you think about the accounting example, it might go through and take a very long time and stumble and really have a hard time to get to that end goal. But once it actually finds and leverages and writes different tools to get there and actually have a mechanism that encodes that within its system to actually leverage those and potentially even improve those tools over time, I think that is the type of system and environment that we're going to see much more of in 2026. Okay, so next up in 2026, I think the script is going to be a little bit flipped. Now, what do I mean by that? So, right now it's very top- down human-driven. We have humans that design and then the agents build, right? We might use something like cursor, what have you. We choose the framework, we design the architecture, the agent executes the plan, and then we fix all of the different approaches and sort of steer it along its way. But I think in 2026, the focus is going to be in and around agents and what agents want to use. An agent might decide on the fly it needs to build a web application. Another agent might call it as a tool or might leverage it as a tool and then within an example of an API, an agent might build an API where an agent becomes the infrastructure and agent driven instead. We're not going to be too deterministic in terms of what the agent do in terms of how we actually get there. We're going to keep it general and the agent is going to figure out, okay, does it need a web app? Does it need an API? doesn't need to build or provision or actually become the infrastructure that would have otherwise been a backend. It's sort of that type of idea. Now, just to close this out, there's a couple different streams, but I think 2026 and beyond, we really have only scratched the surface in terms of the capability of the models that we have right now. I think one of the core things to remember is that computation at scale is going to win. I think agents that generate tools and architecture that emerge from need instead of what we think should be how things are built is going to be one of the big paradigm shifts that we see in 2026. And right now this is just the beginning of what's possible. And just to close this out, I know a lot of people can be nervous with how quickly things are changing. But I think for those that adapt and actually take these lessons and begin to incorporate it in how and what they build are going to be the ones that are going to be at the forefront in actually understanding and being able to leverage and have a lot of leverage in what's actually built in the coming years. Otherwise, that's pretty much it for this video. I'm going to link the bitter lesson within the description of the video. But if you found this video useful, please like, comment, share, and subscribe. Otherwise, until the next