Why Uber built one gateway for 60+ internal LLM use cases instead of letting every team integrate separately

Uber's Michelangelo ML platform team published a writeup of a problem that a lot of companies are quietly running into right now: what happens once "add an LLM to the product" stops being one team's project and becomes 60 different teams' projects.

By the time Uber wrote this up, more than 60 distinct internal use cases existed, spanning process automation, customer support, and content generation. Each team had been building its own integration to whichever vendor it picked, OpenAI, Vertex AI, or an internally hosted model. As the post puts it, "disparate integration strategies adopted by different teams have led to inefficiencies and redundant efforts." That's a familiar shape of problem: not a broken system, just a lot of duplicated, uncoordinated effort that gets worse as it scales.

The fix was the GenAI Gateway, a centralized Go service sitting between every internal team and every LLM vendor. Two design decisions stand out as genuinely useful, independent of Uber's specific stack.

First, the gateway mirrors OpenAI's HTTP/JSON API rather than inventing a new interface. That single choice means tools built for the OpenAI ecosystem, LangChain, LlamaIndex, existing internal code, all work against the gateway with no changes, whether the request actually routes to OpenAI, Vertex AI, or an Uber-hosted model. The team explains the reasoning directly: a proprietary interface "would risk becoming quickly outdated," while aligning with the most widely adopted interface keeps the gateway compatible with an ecosystem that's still moving fast. It's a good example of choosing the boring, already-popular interface over a custom one, specifically to avoid becoming the thing teams have to work around later.

Second, and more interesting: the gateway includes a PII redactor that anonymizes sensitive data before it leaves Uber's infrastructure for a third-party vendor, then un-redacts the placeholders in the response before handing it back. Names become ANONYMIZED_NAME_0, phone numbers become ANONYMIZED_PHONE_NUMBER_0, and so on, with the mapping restored afterward. For any company sending user data to an external LLM API, this is a real, working answer to a real question: how do you get the benefit of a hosted frontier model without handing that model raw customer data.

The result, as reported: close to 30 internal customer teams, 16 million queries per month, peak QPS of 25. That's a meaningful amount of consolidated traffic that used to be scattered across ad hoc integrations.

What makes this post worth reading past the summary is that Uber doesn't stop at the win. They call out directly that the PII redactor "introduces challenges to both latency and result quality," and that redacted placeholders complicate caching and retrieval-augmented generation, since the model is now working with anonymized tokens instead of the original context. That's an honest tradeoff, not a footnote: privacy protection and model quality pull against each other, and Uber chose to eat some of the quality and latency cost rather than skip redaction.

Where this generalizes: any company with more than one or two teams independently calling LLM APIs will eventually hit the same duplication problem, and centralizing behind a well-known interface (rather than a custom one) is a low-regret way to fix it. Where it doesn't generalize as cleanly: building and maintaining a PII redaction layer is a real engineering investment, and it only makes sense once you're actually sending sensitive data to third-party vendors at meaningful scale. A smaller team might get more value from simply restricting which fields are allowed to reach an LLM call in the first place, rather than building a redact/un-redact pipeline.

Full writeup: Navigating the LLM Landscape: Uber's Innovation with GenAI Gateway.