PayPal says its internal AI assistant improved question-answer accuracy from 43% to 95% in an internal evaluation, and the underlying model never changed. “The LLM is still the same,” Ren Zhang, PayPal’s vice president and global head of AI product, said in a VentureBeat AI Impact Series discussion. The mechanism was a separate knowledge layer built to carry the organizational context the model lacked.
Data Access Is Not Business Understanding
The assistant could retrieve records but could not interpret them correctly without organizational definitions.
The assistant had access to vetted data before the improvement. What it lacked, according to Zhang, were the business definitions, query patterns, and analytical procedures employees use to make sense of that data. The term “new accounts” illustrates the gap: an agent can retrieve account records and still produce a wrong answer if it applies the wrong organizational definition of “new.”
Golden Queries and Analytical Playbooks
PayPal’s knowledge layer is designed to remain useful across model upgrades.
The knowledge layer contains what PayPal calls “golden queries”: specifications for how data should be retrieved for a given purpose. Alongside those sit analytical playbooks that describe how to investigate questions across segments, products, and behaviors. Zhang characterized the layer as PayPal’s intellectual property and said it was built to be model-agnostic, so the company can substitute a better model without rebuilding the accumulated business logic. That design addresses the cost of model migration, which a VentureBeat Intelligence survey found led 69% of respondents who attempted a model swap to spend several weeks on the work or abandon it before completion.
Agent Workflows and Governance
Governance infrastructure determines what agents can decide independently and when human review is required.
Deepika Khera, PayPal’s senior director of AI, described a seven-agent workflow that investigates alerts across more than 33,000 services, drawing on Datadog, ServiceNow, and internal logs to separate signal from noise. A separate security-review workflow uses as many as 18 agents. The count is determined by the roles, context, and tool permissions the task requires. “Every decision that is proposed is vetted against the source of truth,” Khera said at VentureBeat’s Agentic Infrastructure Stack Summit, adding that many decisions still require human verification before execution.
Procedures Fail When Split Into Fragments
Retrieval design matters as much as what gets retrieved.
An internal HR assistant ran into trouble when procedural instructions were stored as separate chunks; the assistant could not reliably reconstruct the correct sequence. PayPal reported preserving procedural steps as coherent units rather than isolated text fragments. In some workflows, the company also forces memory retrieval at a specified step rather than leaving that decision to the agent.
Performance Claims Come With Their Own Caveats
Volume metrics alone do not confirm productivity gains.
The assistant handled more than 1,000 questions and saved an estimated 1,600 hours over 10 weeks, according to PayPal. Zhang offered her own caution: question volume does not prove productivity. A system can also score well in offline evaluation while failing disproportionately on the questions employees actually ask most often. PayPal uses AI-Powered Websites synthetic question generation, single-turn and multi-turn testing, model-based judging, human assessment, and direct employee feedback to surface failures that benchmark scores miss.
Industry Survey Data on Production Failures
Independent survey data shows that production failures after successful internal tests are common across the industry.
A VentureBeat Pulse survey found that 42% of respondents whose organizations test AI before release said a feature passed internal tests but later failed in production within the past year. Among those reporting failures, 49% said the most significant incident reached customers. An additional 39% of respondents identified no automated checks or monitoring alerts as a first means of discovering incorrect AI output, relying instead on people to catch problems.
PayPal’s reported approach treats business knowledge as the durable asset in its AI infrastructure. The knowledge layer, with its definitions, query standards, and analytical procedures, is designed to persist across model changes. The model itself, by design, is the replaceable part. Organizations currently focused on model selection may find that formalizing their own business definitions and query standards is a more direct path to accuracy than changing the underlying LLM.
