Heavy AI Agents Are an Anti-Pattern: Why Fewer Agents With More Skills Wins

The shape of an AI system that holds up: lean agents, deep skill libraries, and a clear separation between rules and tools.

In this article, I want to convince you that the AI agent rosters most people are building today are an anti-pattern. The shape that actually scales is the opposite: fewer agents, with more skills. Let me explain why, and how I think about this in my own setup.

Introduction

Every week, I see someone go all-in on AI and proudly announce their new lineup of fifteen specialized agents. A research agent. A writing agent. A reviewer agent. A planning agent. A "senior engineer" agent. A grumpy critic, a caveman, etc.

Then they spend more time choosing which one to use than actually using AI to get work done.

This is NOT just a productivity problem. It's a design problem. And the longer I work with AI Agents and AI Agent Skills, the more convinced I am that heavy, deeply specialized agents are the wrong default. Most of the leverage comes from going the other way.

TL;DR

Fewer agents, paired with rich skill libraries, beat heavy specialized rosters on every axis that matters: cognitive load, flexibility, Context Budget, determinism, and maintenance. Agents should hold the rules of the game. Skills should be the tools. The two evolve at very different rates, and conflating them creates fragile systems you eventually stop using.

• Heavy agents create cognitive overhead. The roster itself becomes a thing you manage.
• Specialists are brittle. Real tasks straddle boundaries.
• Skills load on demand, keeping the Context Window focused and lean.
• Skills add the determinism that probabilistic models lack: templates, scripts, embedded programs.
• Skill descriptions auto-trigger loading based on the request, so context stays relevant.
• Different prompts pull different skills into play. Same agent, different result.
• Agents = rules of the game. Skills = the toolbox.
• Updating one skill improves every agent that uses it (see AI Skill Composability).
• Rosters of specialists fragment context and force multi-agent shuffles.
• Lean agents plus deep skill libraries are the shape that holds up over time.

The agent zoo problem

The first issue with agent-heavy designs is human, not technical.

Once you cross five or six agents, the roster itself becomes a thing you manage. You have to remember what each one does, when to reach for it, how it differs from its neighbors, and why past-you thought it deserved its own slot. Worse, specialized agents push you into multi-agent shuffles. Spin up the research agent. Copy the result into the writing agent. Hand that to the reviewer. Pray nothing got lost in translation.

At some point, you stop using AI to do work. You start managing AI to do work. And it's quite SAD when that happens.

Specialization is brittle

The pitch for specialized agents is that domain knowledge makes them better. It does, at exactly the slice of work they were tuned for.

The problem is that real tasks don't respect those slices. The "writing" task turns out to need code. The "research" task turns out to need a diagram. The "code review" task turns out to need a refactor. A specialist hits its edge fast, and when it does, you either context-switch to another agent or watch the specialist do an awkward impression of one.

A general agent with a deep skill library doesn't have this problem. It picks up what it needs as the conversation unfolds (i.e., the agent itself becomes the orchestration layer). The flexibility comes from how a session begins. The way you open the conversation, the way you frame the problem, naturally pulls different skills into play. Same agent, different framing, different toolchains activated, different outcomes. That's not a workaround; that's the feature.

Skills keep agents lean

Heavy agents are expensive in context. Every line of personality, instructions, and embedded knowledge inside an agent's definition takes up space in the Context Window before any actual work begins. A heavy agent walks into the room already half-full.

Skills flip this around. The agent starts lean. Only the skills relevant to the current request get loaded, triggered by their descriptions matching what you've actually asked for. This is exactly the Prompt Lazy Loading AI Design Pattern (PLL) applied to capabilities: defer everything until the moment of use.

The result: the Context Window stops being a bin you stuff capabilities into "just in case" and becomes a focused workspace.

Skills add the determinism

AI is probabilistic at heart. For creative work, that's the whole point. For anything that needs to be reliable, it's a liability.

Skills are how you claw determinism back. A skill can be a template. A script. A small embedded program. A precise checklist with "use this when X" baked into the description. The model still picks WHETHER to invoke a skill, but once invoked, the skill itself executes consistently.

Probabilistic routing, deterministic execution. That's the combination that makes this work in practice, not just in demos.

Agents are the rules. Skills are the tools.

The cleanest mental model I've found:

• An AI Agent Identity holds the rules of the game. How to behave, what to enforce, what to watch for, what it values. These rules should be small and stable.
• Skills are the toolbox the agent reaches into. They can be many. They can change weekly. They can be added, removed, or rewritten without forking a single agent.

The purest expression of this split is the Receptionist AI Design Pattern: an agent whose entire identity is one rule (route the request to the right handler) and whose body holds zero embedded capability. Everything functional lives in skills or other agents that get pulled in on demand. That's it. If a near-empty agent can run a whole system through routing alone, your "research specialist" probably doesn't need a five-hundred-line manual baked in either. The receptionist is the proof-of-concept for "agents = rules": strip the rules down far enough and the agent almost disappears, while the system around it keeps working.

Updating one skill instantly improves every agent that uses it. Try doing that with three deep specialists who each carry their own embedded knowledge. You can't.

My own setup

This isn't theory for me. My personal AI system runs on this shape.

I have a small set of agents (Ghostwriter, Coach, Strategist, Maintenance Worker, plus a few panels of reviewers). Each one is light: a SOUL.md that defines identity, decision framework, and boundaries; a MEMORY.md for accumulated lessons; a DEPENDENCIES.md that lists which skill categories the agent can pull from.

The skill library is the heavy part. Hundreds of skills, organized by namespace, lazily loaded by description. The Ghostwriter doesn't "know" how to write a newsletter. It pulls in the newsletter skill when the conversation says newsletter. The same agent can write an article, a thread, a book chapter, or an email, and the right capability lights up each time.

When I want to add a new capability, I write a skill, not an agent. When I want to change behavior, I update the relevant skill, not three different agents that might or might not have the same logic.

It's the difference between owning a single multi-tool with great attachments and a drawer full of single-purpose gadgets you can never find when you need them.

What about specialization?

Specialized agents excel at one slice and parallelize beautifully when you have orthogonal work. A "review Python microservices" agent really is sharper than a generalist. Multi-agent teams that work in parallel can ship more in less time.

But IMHO, specialization really belongs in the SKILLS, not the AGENT. You can have a deeply specialized "review Python microservices" skill that the generalist agent loads when the task calls for it.

As for parallelization, when I genuinely need it, I prefer two general agents sharing one skill library over ten bespoke specialists. The skill library stays the source of truth. The agents are just lightweight runners/"controllers".

Conclusion

If you're staring at a list of agents and feeling vaguely tired just looking at it, that's the signal.

Collapse them. Keep one or two with clear, stable rules. Move the actual capability into skills. Let the agent's behavior change based on how you open the conversation, not based on which icon you clicked. Over time, the skill library compounds. The agents barely change. That's the win.

Fewer agents. More skills. The rest follows.

That's it for today! ✨

PS: I write more on these every week in DeveloPassion's Newsletter. Subscribe at https://dsebastien.net/newsletter to follow along.

Related

If you want to go deeper, look at related ideas:

• AI Agents
• AI Agent Skills
• AI Skill Composability
• AI Agent Identity
• AI Agent Routing
• AI Agent Distribution
• AI Agent Permissions
• Context Budget
• Context Window
• Prompt Lazy Loading AI Design Pattern (PLL)
• Context Engineering
• Receptionist AI Design Pattern
• My Content Strategy
• My Content Types

About Sébastien

I'm Sébastien Dubois, and I'm on a mission to help knowledge workers escape information overload. After 20+ years in IT and seeing too many brilliant minds drowning in digital chaos, I've decided to help people build systems that actually work. Through the Knowii Community, my courses, products & services and my Website/Newsletter, I share practical and battle-tested systems.

I write about Knowledge Work, Personal Knowledge Management, Note-taking, Lifelong Learning, Personal Organization, Productivity, and more. I also craft lovely digital products and tools.

If you want to follow my work, then become a member and join our community.

Ready to get to the next level?

Want to use AI as a real thinking partner?

• 🤖 AI Ghostwriter Guide — Make AI write like you
• 🧠 AI Master Prompt Workshop — Craft prompts that actually work
• 🎙️ Knowii Voice AI — Privacy-first voice-to-text powered by AI
• 🎯 Join Knowii — AI workshops + community + all courses