- NBER Working Paper 35677 by Bick et al. found correlations as low as 0.10 between survey-reported AI task usage and platform chat logs from OpenAI and Microsoft, meaning the two measurement methods barely agree on what workers actually do with AI.
- OpenAI and Microsoft chat logs record 0.0% usage for “direct organisational operations”, a category covering 43% of workers, making vendor telemetry structurally blind to management and operations work, where enterprise headcount is concentrated.
- A June 2026 Dynatrace study found most organisations rely on logs despite considering them insufficient for understanding AI workload behaviour, leaving audit trails too thin for HIPAA and GDPR traceability requirements.
Enterprise AI business cases built on vendor chat logs are systematically miscounting the work that matters most. A September 2026 NBER working paper (Working Paper 35677), by Alexander Bick of the Federal Reserve Bank of St. Louis, Adam Blandin and Tyler Schumacher of Vanderbilt, and David Deming of Harvard Kennedy School, matched what 13,920 workers said they used AI for against what platform logs from OpenAI, Anthropic and Microsoft actually recorded. The two pictures barely overlap, and the direction of the distortion has direct consequences for how enterprises are sizing their 2027 AI investments.
Two Measures, Two Realities
When the researchers matched task-level AI usage from a nationally representative survey against task distributions in platform telemetry, the correlation was 0.11 for OpenAI data, 0.34 for Anthropic and 0.10 for Microsoft. A perfect agreement between the two methods would score 1.0.
The gap exists because the two methods are looking at different slices of reality. Surveys capture what workers believe they are doing with AI across their full role. Chat logs capture the visible action inside a conversation window. For enterprise planning, the direction of the distortion matters as much as its size.
What Chat Logs Miss
The skew runs in both directions, but one direction is more damaging for investment planning. More than 15% of OpenAI chats are classified as “edit written materials or documents.” According to O*NET occupational data, only about 2.4% of workers hold jobs where that task is a defined primary function, meaning logged data inflates the apparent importance of editing work by roughly sixfold relative to its actual share of the workforce.
The inverse problem is starker. “Direct organisational operations, activities, or procedures” accounts for 4.1% of all reported AI uses in the survey and covers 43% of workers, a large, operationally central category. In both OpenAI and Microsoft logs, that category registers 0.0%. Anthropic’s logs record 0.2%. Chat log analyses are structurally blind to the ways AI touches management and operations work, which is precisely where enterprise headcount is concentrated. An AI investment thesis built on vendor usage data will systematically undercount that contribution, and enterprises already miss several cost layers once pilots move to production.
Survey Limits Run the Other Way
Surveys overstate breadth where logs understate it. As of May 2026, the NBER paper puts generative AI work adoption at roughly 45% of workers, a figure the authors treat as a lower bound, since embedded or passive AI use often goes unreported. Adoption spans more than 80% of occupations and 40% of tasks at the 20% usage threshold. Depth is thinner: only 2.8% of tasks show adoption above 50%, and none cross 70%.
Social desirability compounds the problem. Many workers conceal their AI use at work even while believing colleagues should disclose theirs, which means survey figures likely overcount confident, visible usage and miss the quiet, habitual kind.
The Observability Gap
Neither surveys nor raw chat logs give enterprises what they actually need: granular, real-time visibility into what AI agents are doing inside production workflows. Token counts and latency figures say something about infrastructure cost; they say little about whether an agent completed the right task, touched sensitive data it should not have, or produced output that required human rework.
A June 2026 Dynatrace study found that most organisations rely heavily on logs despite finding them insufficient for understanding AI workload behaviour. The practical failure mode is concrete: a support engineer pasting customer records into an external LLM for summarisation may leave no operational trace unless audit logging is explicitly configured to capture it. That same gap makes HIPAA and GDPR traceability requirements difficult to demonstrate in a regulatory examination. Audit logs that reconstruct the full interaction, what data entered the system, what the model returned, what action followed, are the unit of proof regulators and security teams need.
Rebuilding the Business Case
The measurement distortions the NBER paper identifies carry direct budget consequences. A business case built primarily on chat log analysis will overstate AI’s impact on content and information tasks while undercounting its role in operations and management, the categories chat logs miss entirely. ROI projections built on that foundation are calibrated for the wrong work, a problem that becomes harder to ignore as boards increasingly tie AI funding to demonstrated returns.
The paper proposes an approach for establishing accurate AI performance baselines: measuring throughput, cycle time, quality and verification effort using explicit signals rather than self-reported usage or raw platform telemetry. That prescription is straightforward; the harder problem is that most enterprises have not yet instrumented their AI deployments to produce those signals at all.
