A customer walked up to a fast-food kiosk last year, asked it to write Python code instead of taking their order, and it complied. Somebody else figured out how to chain together three different brand assistants and use them as a free distributed coding environment. Somebody else asked a customer service chatbot for help with an order and got the contents of a user table printed on the receipt.
These aren't theoretical risks. They're already production incidents at companies that thought their AI deployments were governed. The governance was theater. The bill is starting to arrive.
I had this conversation with Paul Lewis, CTO of Pythian, on Business Disruptions in Tech recently, and we landed on the same uncomfortable observation. Of all the things that have changed in the way enterprises approach AI in the last fourteen months, the one thing that hasn't changed is governance. It's still last on the Gartner hype cycle. It's still last in the workshop agenda. It's still last in the budget. And the deployments are accelerating anyway.
That gap is closing the wrong way. Production is racing ahead. Governance isn't catching up. The space between them is where the next round of incidents lives.
We Don't Even Agree What This Means
Part of why governance keeps trailing is that the term itself is undefined. Ask ten enterprises what AI governance means and you'll get ten different answers, most of them focused on the part of the problem the speaker happens to own.
The data team will tell you AI governance is data governance. Make sure your training corpus is clean, your lineage is documented, your access controls are enforced. That work is necessary. It is not sufficient. It also has nothing to do with what happens when a customer asks your kiosk for a Python tutor.
The security team will tell you AI governance is prompt injection defense. Patch the obvious exploits, sanitize inputs, monitor for anomalous queries. That work is also necessary. Also not sufficient. Also has nothing to do with what happens when your model provider quietly ships a new version that changes how your agent reasons about ambiguous customer intent.
The legal team will tell you AI governance is regulatory compliance. EU AI Act, sector-specific requirements, disclosure language, audit trails. Necessary. Not sufficient. The regulators are still figuring out what to ask, and they're going to ask harder questions next year than they're asking this year.
None of these definitions are wrong. All of them are partial. And as long as nobody owns the whole thing, nobody is responsible when the whole thing fails. Which it will.
The Vendor Lie You're Buying
The most expensive assumption in enterprise AI right now is that the tool you bought has guardrails built in. The pitch sounds reassuring. The demo looks safe. The product page mentions enterprise-grade controls. The contract has the right indemnification language.
In practice, most of those guardrails are configured for the average customer, not for your specific deployment. They protect the vendor from the most embarrassing public exploits. They don't protect you from the exploits that haven't been publicly embarrassing yet. And in some cases, the tool is actively opening up more authority than your security team realizes, because the vendor's commercial interest is in showing you everything the model can do, not in narrowing what it should do.
This is the part where your CISO walks into a meeting and discovers that the marketing team has deployed an agent that can read from the customer database, write to the support ticketing system, and trigger refunds, all without going through any of the change-control processes that govern every other system that can do those things. Nobody hid it. It just wasn't asked about, because the agent is "just an AI tool" and AI tools live in a different organizational compartment than the systems they're touching.
That compartment is going to disappear in the next twelve to eighteen months, either because somebody inside the company forces the issue or because somebody outside the company exploits the gap. Either way, the bill arrives.
What You're Actually Paying For
The cost of weak AI governance shows up in three places, and most enterprise AI ROI models account for none of them.
The first is reputational. The chatbot that solves Python problems instead of taking burger orders is a screenshot, and screenshots travel. So does the chatbot that surfaces a competitor's pricing, or the one that says something offensive in a customer's voice, or the one that confidently cites a regulation that doesn't exist. The reputational cost of one bad screenshot can dwarf the entire value of a year of AI productivity gains, and you don't get to argue with how the screenshot is interpreted.
The second is regulatory. The regulators have not yet started enforcing aggressively, but they are watching. The EU is building case law. US sector regulators are circling. State attorneys general are asking questions. The companies that get hit first are going to be the ones that look like they weren't trying. "We assumed the vendor's guardrails were sufficient" is going to be a sentence somebody has to say under oath, and it will not go well.
The third is financial, and it's the one nobody talks about. Every time your agent does something it wasn't supposed to do, you paid for the tokens. Every time someone jailbreaks your chatbot into solving an unrelated problem, you paid for the tokens. Every time your model burns cycles on inputs that should have been rejected at the gate, you paid for the tokens. Token cost overruns from poor governance are real, they're already showing up in cloud bills, and they're invisible to most of the people who would care if they could see them.
What Governance Has to Cover
Real AI governance has to span four things at once, and most enterprises are doing one or two and calling it good.
The data governance piece, which most teams have made progress on. Training data, retrieval data, knowledge management, access controls. Necessary foundation, doesn't address runtime behavior.
The intent governance piece, which almost nobody has touched. What is the system supposed to do? What is it not supposed to do? When a user asks it for something out of scope, what's the correct refusal behavior? Most deployments have no documented answer to these questions, which is why the kiosk happily wrote Python.
The output governance piece. What can leave the system? What gets logged, redacted, blocked, escalated? Are the outputs being inspected for the categories of mistake that would damage you most? Most observability stacks aren't asking these questions, because they were built for software that didn't have opinions.
The chaos governance piece. What happens when the model provider ships a quiet update that changes the model's reasoning? What happens when somebody discovers an exploit you've never seen? What's the kill switch? Who has the authority to throw it? How fast can you actually take an agent offline if it starts doing something embarrassing in production?
If you can't answer all four, you don't have AI governance. You have a partial defense and an exposed flank.
Three Things to Do This Quarter
Pick someone whose job is AI governance, end to end. Not the data person, not the security person, not the legal person. Someone whose accountability spans all four of them and who has the authority to stop a deployment that doesn't meet the standard. Until that person exists, governance will keep trailing because no single function owns it.
Audit your existing AI deployments for the authority they actually have. Not the authority the vendor's slide deck described. The authority the agent actually possesses inside your environment. Read access. Write access. Trigger access. Cost authority. Most enterprises will be surprised by what they find. Some will be horrified.
Build the kill switch before you need it. Every production AI asset should have a documented, tested, fast-acting way to take it offline. If your incident response runbook for "the agent is doing something it shouldn't" is "we'll figure it out when it happens," you've already failed the next incident. Test the kill switch. Time it. Make sure the right person has the authority to pull it without scheduling a meeting first.
The Bill Is Already Arriving
The era of "we'll figure out governance later" is ending. It's ending because the exploits are getting public, the regulators are getting serious, and the cost of not having the conversation is starting to exceed the cost of having it.
The companies that get ahead of this are going to look paranoid for about six months and prescient for about ten years. The companies that don't are going to spend the next several earnings cycles explaining a screenshot. Pick which one you'd rather be.