The AI CoE Blueprint

Somewhere in your company there's a spreadsheet of AI ideas. It started with a dozen rows. Now it has a couple hundred, a priority column nobody agrees on, and at least one idea that three teams are building separately without knowing it.

That spreadsheet is usually how an AI center of excellence gets started. Someone looks at it, then at the invoices from four different AI services, and asks for one central team to set how AI gets built, ship the first systems, and teach everyone else to build on them.

I like centers of excellence. I've also seen plenty of them turn into a waiting room, where ideas sit in line for a review that never quite happens. So this is how we set one up, and what we want in place before we hand it back to you.

Stop ranking the list

The instinct is to rank the spreadsheet. Score every idea on value and effort, draw the two-by-two, take it to a steering meeting. That eats a quarter. By the end, the people who asked for help have found their own way around the problem.

We skip it. For each idea we ask who owns it, what number it should move, and when it should be running. The ideas with clear answers get built. The ones without an owner can wait, and they're rarely missed.

Pick that number carefully. A coding agent that opens more pull requests than anyone on the team looks great on a dashboard, right up until your reviewers spend a whole week cleaning up after it. Pull requests were just the easiest thing to count.

Look at the whole job around each idea, too. Dropping AI into one step of an old process tends to save a few minutes and change nothing else. The bigger wins come from redesigning the work, and that usually means a few steps disappear.

Most everyday AI use doesn't need the center at all. Writing, research, summarizing a long thread. Give people good tools and get out of the way, and if a vendor already does something well, buy it. Save the center for the work that's specific to your business.

Build the front door first

The first thing we build is boring: one front door for AI.

Right now every team probably connects to AI services its own way. You pay for the same work more than once, and you can't see the spend until the invoice shows up. Worse, nobody can say for sure what data has left the building.

Send every request through one shared gateway and most of that goes away. Personal information gets stripped before anything leaves, and cost shows up by team. You get a record of what was asked. And it's the one place to decide which tools and data each agent can touch.

The catch is that the front door has to be the easiest way in. If it's slower than a personal account and a company card, people will use the personal account. I would too. Then the usage you most needed to watch happens where you can't see it.

Behind the front door, AI needs one shared way into your data and systems. An agent is only as useful as what it can look up. Build each connection once so every team can reuse it, and have it carry the permissions of the person asking. If you can't open a file, neither can the agent working for you.

Security checks run on every change, automatically, so a review happens whether or not anyone remembers to ask. It's how we build everything (we call it secure by design), and every center we help set up starts that way.

Size the reviews to the risk. If a low-risk change has to wait two weeks for a committee, people learn fast, and suddenly everything they submit is low risk.

Let the users write the tests

I care more about the first system than anything else a center does in its first year. It should run on real data, for people whose work depends on it. A showcase for the steering committee doesn't count.

People use a system when they trust it, and the fastest way I know to get there is to hand them the testing. Give them questions from their actual work and let them grade the answers. Then fix whatever they flag. After launch, those questions become your evals. Run them every day, so you hear about a problem from the evals before you hear about it from a customer. When your AI provider ships a new version, run them again before you switch.

A lot of AI work dies right here, somewhere between the pilot and everyday use. We wrote about why in The Pilot Isn't the Product.

The second system tells you more than the first. If it comes together faster, mostly out of the same parts, the foundation is working. If it takes just as long, something underneath needs another look.

Train on what you built

I've always thought the best technical training happens while you build something real, and AI hasn't changed my mind. Your engineers build the first system alongside ours, and the classes come after, about the thing they just built. One of your people becomes the practice lead and works next to one of ours the whole way.

For the rest of the company, find your champions. They're the people already using AI well in their own jobs, the ones coworkers quietly go to for help. Give them a few hours a week to help their teams and bring the next ideas forward. I'd skip the mandate, though. People pass a useful tool around on their own.

Treat agents like new hires

Agents are where I'd slow down a little. They take a step, look at what happened, and take another, often without a person in between.

I think about an agent the way I'd think about a new hire. A person in the business is responsible for it. It has a clear job, and access to what that job needs and nothing more. It gets its own login, never a borrowed one, so you can see what it did and shut it off without locking anyone out. When its manager changes roles, someone else takes it over. And it gets an end date, so you have to decide, on purpose, whether it's still worth keeping.

Agents are easy to start and easy to forget. Someone builds a useful one, moves to another team, and six months later it's still running on their old access. No one is watching it. You find out it's there when it does something odd.

So the habits that matter are dull ones. Keep one list of every agent and who's responsible for it, including the agents that came bundled inside software you bought. Keep a record of what each one did and why. Have a person approve anything an agent can't undo. And give every agent an off switch you've actually flipped.

Then hand over the keys

This is the part I care about most. It's also the easiest part to put off, because every month there's a good reason to stay a little longer.

A center of excellence should get smaller as AI spreads through the business. If yours keeps growing year after year, everything probably still has to go through it.

So we agree on how we'll step back at the very start, before anyone gets attached. By the end, the architecture lives in your code and the checks run in your pipeline. Every system has its evals and a runbook for when it breaks. The agent list is yours, and so are the off switches. Your people approve what ships next.

Each system gets judged on the number it was supposed to move. We judge the center on four: how many systems are in production, how long it takes to go from yes to running, how much each new one reuses from the last, and how many of your engineers are shipping without us. That last one is my favorite.

One team, start to finish

Strategy, delivery, and training often end up with three different firms, and the gaps between them become nobody's problem. I'd rather we own the whole thing. We do the strategy and build the first systems, secure by design. Then we train your team to run them. When a specialist would help, we bring them in as part of our team, so you still have one plan and one group accountable for it.

I'm biased, of course. This is what we do. But if you have an AI center of excellence with a long list and not much running yet, I'd love to hear about it. It's why we built AI Edge.