The gateway isn't the whole governance story. Eight mistakes enterprise teams make, and theidentity and action layers a gateway was neverbuilt to cover.
‍
Enterprise AI gateway adoption is accelerating. Enterprise AI spend hit $37 billion in 2025, up from $11.5 billion the year before (Menlo Ventures State of GenAI, 2025). An AI gateway (the middleware layer routing requests between your enterprise applications and LLM providers, enforcing rate limits, managing costs, and handling provider failover) is now standard infrastructure for any team running AI at scale.

The problem is that these same eight misconfigurations show up again and again.

Over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value, or inadequate risk controls (Gartner, June 2025). Identity and access gaps are one common form inadequate risk control takes. To reduce the chances of it happening in your organization, you need to build infrastructure around 1) who authorized the agent, 2) what it could do inside enterprise tools, and 3) whether the controls in place went deeper than token counts and provider routing. 

The eight mistakes below show up most consistently.

‍

What Each Layer of AI Infrastructure Governs

The AI agent governance stack has three distinct layers. Understanding which one your gateway covers is the first step toward every fix.

‍

Enterprise teams typically build the AI gateway first. The identity and access layer is the foundational piece, and the eight mistakes below all trace back to skipping it.

Mistake #1: Treating the Gateway as the Whole Governance Story

An AI gateway (software like LiteLLM, Portkey, Kong AI Gateway, or Cloudflare AI Gateway) governs the connection layer. 

It controls which applications can call which LLM providers, at what rate, under what cost budget. Absolutely necessary controls. But not sufficient. 

The governance gap that gateways do not close is the action layer (what the agent does inside each connected tool). 

An enterprise agent might be permitted to call Linear via the gateway. 

Can it read any record, or only records owned by the user who invoked it? Can it write to production data or only staging? Can it delete? 

The gateway sees the API call. It does not see the data the agent touches, the scope of the action, or the human identity responsible for authorizing it.
‍

Most organizations understand what a gateway covers when they deploy it. Fewer have a plan for what it doesn't cover. 

The action layer is what the enterprise will be asked to audit. For example, by regulators, security teams, or anybody conducting incident post-mortems. 

Building governance at the connection layer only leaves you blind to much of what an AI agent does. The tool calls and the multi-agent collaboration. It also prevents you from being able to define and confine how an AI agent can behave. 

The fix is to treat the gateway as one layer in a stack, with identity and access controls handling what agents do once inside each tool.

Mistake #2: Underestimating the Gaps Left Outside Rate Limits and Cost Caps

A rate limit caps how many calls an agent can make and how much it can spend. Cloudflare, Portkey, Kong, and LiteLLM all offer rate limiting and cost-budget controls, and every enterprise running AI at scale should use one.

A rate limit answers "how many calls can this agent make?" and can slow down overzealous bad actors or AI agents running awry, which may limit some damage purely because there are only so many actions they can take before hitting a wall. 

Capping token spend protects the AI bill. If a bad actor finds an API key, it limits the blast radius and damage they can do to your token spend. But it only takes a single dangerous action to cause serious financial and reputational damage to your business (data leaks, hijacked systems, etc.).

Mistake #3: Deploying Agents Before Identity Is Wired In

In practice, teams tend to provision AI agents and their access to the tools they need first, and figure out identity later. 

By the time governance becomes urgent, the agents are already running with API keys and access to MCP servers that have no IdP binding, no scoped permissions, and no expiry logic.

The average enterprise runs 3,891 SaaS and AI environments, with 23,021 applications operating outside centralized IT visibility (Grip Security, 2026 SaaS + AI Security Report). 

Every agent deployment without a bound identity adds to that invisible surface.

The right sequence is to organize identity first, and access route second. 

When an agent inherits a real employee's identity through the existing identity provider (Okta, Entra ID, JumpCloud, etc.) that manages who your employees are and what they can access, it gains real scoped permissions, group memberships, and role constraints. These propagate into every tool it touches. 

The gateway then enforces the access route, and the identity layer enforces what the agent can do at the destination.

Mistake #4: Confusing API Keys with Identity

Two-thirds of organizations carry risky OAuth permission scopes (grants that give AI tools broader data access than any governance policy formally authorized) (Grip Security, 2026). 

This is exacerbated when the identity layer is left out of the gateway stack. 

It looks like OAuth tokens and API keys that persist after the employee who created them has left, changed roles, or moved to a different team.

On-prem had Active Directory. SaaS had Okta. AI agents need the same governed identity layer underneath them. A virtual key in a gateway carries none of that (no live IdP binding, no scoped permissions, no tie to a real employee identity).

Gateways manage API keys and virtual credentials. Tokens that tell a model provider "this application is authorized to call you." 

Those keys identify the application, but say nothing about the person behind the action. 

When an agent takes an action inside Jira, Slack, or Salesforce using a virtual key, there is no record of which employee's authority it was acting under, which data it was permitted to touch, or whether that employee even still works at the company...

Mistake #5: No Audit Trail Beyond Token Counts

AI agents still fail roughly one in three tasks on Stanford's OSWorld benchmark of general computer-use tasks across operating systems (Stanford HAI, 2026 AI Index). When an agent fails, or when it succeeds in a way that later becomes a compliance or security concern, the audit trail determines whether the incident is resolvable or unresolvable. 

Gateway observability typically reports 1) which provider was called, 2) how many tokens were consumed, 3) what latency was measured, and 4) whether a rate limit was hit. All useful operational data. Some gateways also log tool calls, parameters, and responses, but none of it ties an action to the employee identity that authorized it, which is what a compliance audit requires.

Action-level logs are the difference between knowing an agent ran and knowing what it did.

An action-level compliance audit trail for an AI agent answers “What did this agent do, inside which enterprise systems, on which records, under whose authority, and at what time?”

SOC 2 audits generally expect access to be traceable to who did what and when. HIPAA's audit-control standard (45 CFR 164.312(b)) requires recording and examining activity in systems that handle protected health data. SOX-driven internal controls require an auditable trail for changes to financial systems. And the EU AI Act's record-keeping requirement (Article 12) mandates automatic action-level logging for high-risk AI systems. None of this is satisfied by gateway-level connection logs alone. 

‍

Mistake #6: Assuming the Gateway Handles Permissions

The version of Mistake #1 that shows up in production often looks like a team deploying a gateway, granting the agent access to a tool, and assuming the gateway provides the ability to define and enforce what the agent can do inside that tool. 

It's a complicated issue, because many platforms focused on connecting and controlling agentic AI contain a broad mix of features, including gateway functionality. There are pure-play gateways. But there are other platforms that combine shadow AI detection, identity, access, MCP connection marketplaces, reporting, and more into one bundled platform. 

The takeaway is that not all platforms that provide gateway functionality will also enable permissions handling. Others, which provide gateway functionality alongside a host of other features, may. 

So here, we are talking about the gateway layer, not any platform that happens to include gateway functionality. 

Gateway-level permissions answer “Can this agent call this API endpoint?”

App-aware permissions (the action-level layer that governs enterprise agent behavior) answer “Can this agent read this record, write to this field, delete this file, or reassign this ticket?”

The practical consequence is that an agent granted access to "Salesforce" via a gateway may be able to read any customer record in the CRM, not just records associated with the user who invoked it. 

A governance policy that says "agents should only access customer data relevant to the current task" cannot be enforced at the gateway layer. It requires app-aware, action-level permission scoping tied to the agent's identity context.

If your agents are already SCIM-provisioned in Okta and your IdP groups map to your enterprise tools, you can bolt action-layer permissions on top of existing gateway routing. 

If your agents run entirely outside your IdP boundary, the identity layer has to come first. 

Mistake #7: Set-and-Forget Agent Permissions

Gateway credentials tend to get issued once at setup and never reviewed again, which is a particular risk for agents running long-timeframe background tasks. Unless access is clearly tied to an identity, provisioned through your identity provider, that permission-profile may not update when an employee changes roles, a project ends, or a data classification shifts. 

AI-related SaaS attacks increased approximately 490% year over year, and the attack surface is shifting from infrastructure to identity. Attackers are increasingly targeting OAuth integrations, delegated access, and non-human identities rather than compromising systems directly (Grip Security, 2026). 

The identity-first approach solves this. When an agent inherits its authority from a real employee's IdP identity, the agent's permissions live and die with that employee's account. 

Deprovisioning the employee in Okta automatically revokes the agent's access to every connected tool, with no separate key rotation workflow, no manual audit of standing gateway credentials, and no ticket to the platform team.

‍

"When we started, the question was whether enterprises would govern AI agents at all. That's settled. The real question now is whether they'll govern them one provider at a time, or once, across all of them." - Eyal Ben Ezra, Co-Founder and CEO of Willow

Mistake #8: No Visibility Into What Agents Do Inside the Tools

Gateway observability shows you the high-level view of which tools or apps were called. It does not give you action visibility (what the agent read inside that tool, which records it modified, what data it ingested into the LLM context). This is fine for singular well-defined scripts or workloads. You know what is called because it’s built into an unchanging process or script. 

But AI agents themselves decide how they’ll complete the task provided to them. They’re not confined to a single, predictable script. 

This can mean different tool calls each time, a different order of operations, autonomously adapting to the changing underlying data landscape, and sometimes, simply hallucinating or making a mistake. It is not predictable. It can also mean dozens, hundreds, or even thousands of tool calls during a workflow or task. The whole time, the agent is building context, memories, and more. Each time, it produces a different internal result before providing the output. 

Action-level permissions are protection against out-of-bounds activity. Action-level logging is how you capture the bulk of what an AI agent does that simple gateway observations miss. 

The second layer of the problem is assuming your gateway observability measures actually capture all the AI-related calls in your org. 

Shadow AI (unauthorized AI tools used by employees, running outside IT-governed systems) is difficult to govern. And your gateway isn't a sufficient defense. It is now pervasive enough that ISACA flags shadow AI as a fast-rising enterprise risk that bypasses security and compliance controls (ISACA, 2025). Most detection efforts stop at the connection layer. A tool is either approved or unapproved, so what an approved tool's agent does inside enterprise systems is often invisible.

Closing that gap takes discovery that runs below the connection layer. Endpoint sensors and a browser extension surface every tool and skill actually in use, including rogue MCP servers, personal API keys, and unsanctioned agents that never route through a gateway. Detection alone only tells you an unmanaged tool exists. The identity and access layer is what lets you act on it: bind the tool to a real employee, scope its permissions, and log what it does inside your systems, or block it outright.

What This Means for Your AI Agent Tool Stack

An AI gateway is necessary. This is not an effort to deter you from using one. You need one. Every mistake above is about highlighting the layers a gateway does not cover, and how to build them.

Provision identity before routes, tie agent authority to real employee IdP groups, and log at the action level. 

And importantly, engineer the infrastructure needed to handle agent permissions as living state rather than static configuration. When an employee's role changes, the agent's scope changes with it.

Willow, the Agentic Access Platform, builds the identity and access layer for AI agents at work. 

Every agent inherits a real employee's identity through the existing IdP (Okta, Entra ID, JumpCloud), operates within action-level permission scopes, and writes every tool call to an immutable, per-action audit trail.

Willow sits above any gateway, so you can keep LiteLLM, Portkey, or Kong for routing and add the identity and action layer on top.

The gateway controls which providers the agent can reach. Willow governs what it can do at the destination, and who is accountable for it.

Willow also ships its own MCP gateway and a marketplace of pre-built integrations, so teams that want to consolidate can route through Willow directly rather than adding it on top of an existing gateway. The marketplace carries over 1,000 connectors, plus pre-built skills and plugins, each installed scoped to an identity and policy, with Slack approvals where a team requires them. With Willow, any MCP-compatible agent (whether Claude, Cursor, or ChatGPT) can reach approved tools through one governed entry point.

The practical impact is that agents can proliferate across your org while staying governed and decommissionable. 

Wix uses Willow to run agent access today, across nearly 5,000 weekly users and almost 600 governed tools, with more than 300,000 governed tool calls a week, every one tied to a real identity (Willow, Wix case study).

Frequently Asked Questions

AI gateway governance raises the same questions across enterprise teams. 

Further Reading

• Menlo Ventures State of Generative AI, 2025: https://menlovc.com/perspective/2025-the-state-of-generative-ai-in-the-enterprise
• Grip Security 2026 SaaS + AI Security Report: https://grip.security/blog/ai-governance-statistics
• Gartner, Over 40% of Agentic AI Projects Will Be Canceled by End of 2027, June 2025: https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
• Stanford HAI 2026 AI Index: https://aiindex.stanford.edu/report/
• ISACA, From Shadow IT to Shadow AI, 2025: https://www.isaca.org/resources/news-and-trends/newsletters/atisaca/2025/volume-19/from-shadow-it-to-shadow-ai-navigating-the-new-frontier-of-enterprise-risk
• EU AI Act, Article 12 (Record-keeping for high-risk AI systems): https://artificialintelligenceact.eu/article/12/
• Willow, Identity and Access: https://withwillow.ai/platform/identity-access
• Willow, Governance and Compliance: https://withwillow.ai/platform/governance-compliance