Two and a half weeks ago, I cleared the calendar.
Not entirely. But enough.
Enough to do something I’d been putting off since the first time I tried to build a real AI workflow and found myself managing it more than using it.
I wanted to go all the way in.
Read the research. Rebuild from scratch. Figure out what I’d got wrong. And come out the other side with a cleaner answer about what building with AI agents actually means when you’re running a company, not just experimenting in a sandbox.
So that’s what I did, and what I learned wasn’t really about which AI model is best.
It was about architecture.
I’d Built an AI Workforce Before. It Didn’t Work the Way I Expected.
I’d done a version of this before.
Early last year, I built what I thought was a sophisticated agentic workforce inside Claude. Specialized agents for different functions. Workflows between them. A Guardian agent to keep everything organized. On paper, it looked impressive. And for a while, it worked.
Then it started asking more of me than it saved. Costs crept up. I was stepping in more often to manage decisions that were supposed to run themselves.
The thing I’d built to give me time back had made me its manager.
That bothered me for months before I did anything about it because the problem wasn’t that AI didn’t work. The problem was that I had built the system around the wrong mental model.
What I Actually Did for Two and a Half Weeks
The intensive I ran is called the Harvard Data Science Intensive Agentic AI program.
It was two and a half weeks of structured deep work. Not passive learning. You build as you go. The curriculum forces you to make real architectural decisions in real time, which is the only way you actually learn them.
I went in with a specific question: Had I built the right system in the wrong way, or had I built the right system and simply stopped caring for it well enough?
Those are very different problems. They also have very different answers. The short version?
It was both.
But the more interesting finding came in the second half.
What I Got Wrong the First Time
The architecture I’d built was impressive on paper and expensive in practice.
I’d treated AI agents like employees. Specialized. Siloed. Each one responsible for its own piece.
What I had actually created was a coordination overhead problem. Every time something needed a judgment call, it had to escalate, and because I’d made the system complex enough to look autonomous, the escalations were invisible until they had compounded.
That was the first big shift in my thinking.
Agents aren’t employees. They’re infrastructure.
You don’t manage infrastructure the same way you manage people.
You design it, test it. Then you trust it or you rebuild it.
The micromanagement pattern that keeps creeping in when an AI workflow doesn’t behave how you expected it to is usually a sign that the workflow was designed for your anxiety, not for the task.
That one took me a while to understand.
I Was Also Too Precious About the Tool
The second thing I got wrong was simpler. I’d been too precious about the tool.
When something wasn’t working, I would usually try to prompt my way out of it instead of asking whether I was using the right model for the job.
That’s not a principled position. That’s familiarity bias dressed up as loyalty.
And once you start thinking about AI as infrastructure rather than a collection of favourite tools, that becomes much harder to justify.
I Ran the Same Prompt Through Six Different Models
At one point during the intensive, I ran the same evaluation prompt through six different models.
- Claude.
- ChatGPT.
- Gemini.
- Perplexity.
- Grok.
- Manus.
Same prompt, context and task.
I wanted an objective read on what I’d built and where the gaps were.
They didn’t agree on a winner and that was probably the most useful thing I learned.
Not because it proved one model was better than another. It didn’t.
It showed me that “What’s the best model?” is the wrong question.
The models are genuinely different in ways that matter.
- Reasoning.
- Synthesis.
- Memory.
- Speed.
- Cost at volume.
The right question is: Which task needs which capability, and does your architecture make it easy to route the task correctly?
My original system didn’t. It used one model for almost everything because I’d built it when I had strong opinions about one model.
When the models started diverging, my system didn’t adapt.
That’s an architecture problem.
Not a model problem.
What I Built Differently the Second Time
The rebuild focused on portability over preference.
That sounds technical. It actually changed the entire way I thought about the system.
Before, I was building toward a specific tool. This time, I was building toward an outcome. Those are meaningfully different things. Practically, that meant lighter coordination overhead. Clearer lanes for what requires human judgment and what doesn’t. Defined escalation logic.
The system should surface the right decisions. It shouldn’t quietly get stuck and wait for me to notice.
I also built a cleaner mental model for what I actually want from an AI workforce.
I want it to handle the information layer. The routing, first-pass synthesis, things that are well-defined and repeatable.
I want to keep the judgment calls, the relationship decisions, strategic assessments where competing values create real trade-offs.
That 20/80 split I’ve written about before?
The 20% is harder now.
Not easier.
Because the administrative noise goes down and the judgment gets harder.
The judgment is all that’s left.
This Isn’t Really a Productivity Story
This is where I think the conversation around AI agents gets interesting.
This isn’t a productivity story. I know it can read like one. But that’s not the point.
The pattern I keep seeing across the companies I work with is the same one I found in my own system.
They’ve added AI tools. They’ve made some things faster.
But the architecture underneath hasn’t changed. The information still flows the same way. The decisions still escalate the same way.
The strategic layer still depends on the same two people it always did.
So now there are more tools sitting on top of the same underlying system.
And that creates a problem.
Adding tools to a broken architecture makes the architecture more complex, not more effective.
That’s the distinction I think a lot of companies are going to have to confront.
A Tool Is Not the Same as a System
The intensive forced me to think about architecture in a way that generic tool evaluation doesn’t.
You’re not picking a winner.
You’re designing a system for the actual work that needs to happen.
That is a different question. And it requires more honesty about what is really happening inside an organization than most teams are used to.
- Where does information actually go?
- Who makes the decisions?
- Where do things get stuck?
- What requires judgment?
- What is simply repetitive?
- Where are humans creating value?
- And where are humans acting as expensive middleware between systems that should be able to communicate?
Those are much more useful questions than:
“Which AI tool should we buy?”
What This Means for Operators
For operators, the shift is less about adding another tool and more about deciding what you actually want the business to look like with AI built into it.
That starts with the work.
Look at the processes that consume the most time. Find the decisions that repeatedly land on the same people. Identify the information that gets copied, reformatted, summarised or passed between teams.
Then ask a harder question:
Does this need a person, or does it need a better system?
Some work should disappear into automation. Some should be handled by agents. Some should stay firmly with humans because the value is in the judgment, context or relationship.
The mistake is treating all three categories the same.
And this is why I think the next phase of AI adoption will be less about experimentation and more about operating design.
The companies that get the most from AI won’t simply have more agents.
They’ll have a clearer understanding of where those agents belong, what they’re responsible for and when they should hand something back to a human. Because the architecture has to survive changing models, changing tools and changing assumptions.
If your entire system only works because you picked your favourite model and built everything around it, you haven’t built an AI system.
You’ve built a dependency.
What I’d Do Differently If I Started Again
I would start with the work.
Not the tool.
I would map the decisions first.
- Then the information required to make those decisions.
- Then the repeatable tasks around them.
- Then I’d work backwards into the models and agents that could support that system.
That sounds obvious.
It wasn’t obvious to me the first time. I started with what the technology could do. Now I’m starting with what the business actually needs to happen.
That’s a much better place to build from.
The Real Lesson
Two and a half weeks of intensive building didn’t leave me with a favourite AI model.
It left me with a much stronger opinion about architecture.
The tools will keep changing. The models will keep getting better. The pricing will change. New platforms will appear. Some of the tools we’re obsessed with today will be irrelevant surprisingly quickly.
That part is inevitable.
What matters is whether the system you build can adapt with them.
That’s what I got wrong the first time.
I built around a tool.
This time, I built around the outcome.
And I think that distinction is going to matter a lot more over the next few years.
Don’t build an AI workforce that needs you to manage it.
Build the infrastructure that lets the right work happen
Read the Comments +