About that MIT report…

Last month, the internet was abuzz about an MIT report with a dramatic headline: “95% of generative AI pilots at companies are failing.”

Fortune had the exclusive, and paywalled the write up. The report itself, published by MIT’s NANDA1, could only be accessed by filling out a Google Form. I don’t think many people actually read the report, but the headline was enough. Here’s what happened the next day:

Shares of megacap tech and big-name chipmakers declined. Nvidia shares lost 3.5%, while Advanced Micro Devices and Broadcom slipped 5.4% and 3.6%, respectively. Shares of high-flying software stock Palantir dropped more than 9%, making it the S&P 500′s worst performer. Other major tech-related names such as Tesla, Meta Platforms, and Netflix were also under pressure.

Since then, many have criticized the methodology and conclusions of the report. Too few executives were surveyed, those that were didn’t represent the entire market, and the report (on the whole) reads as an advertisement for NANDA’s mission rather than a peer-reviewed research paper (because it’s not).

Someone could probably start a pretty good investment fund that just reads the papers behind the headlines that move the market.

You can read the actual report here, without filling out any Google Forms. It’s worth skimming, as there are a few datapoints more interesting than the headline claim.

From those, I want to highlight these two figures (emphasis mine):

For all the criticism of the NANDA report, it is a survey of many business leaders. We can treat it as such. So while we might take that 95% figure with a grain of salt, we can trust that business leaders believe the biggest reason their AI pilots are failing is because their employees are unwilling to adopt new tools… While 90% of employees surveyed eagerly use AI tools they procure themselves.

A Simpson’s classic comes to mind:

"Am I out of touch? No, it's the children who are wrong." – Seymore Skinner

The subject of employees using their own ChatGPT or Claude accounts at work has been heavily discussed for years. It’s frequently referred to as the “Shadow AI Economy,” and is a source of anxiety for IT leaders and inside counsel.

Just this week, OpenAI published a paper on ChatGPT usage that validates this specter:

OpenAI’s report is excellent and provides a rare look at how people use ChatGPT2: ~80% of usage is for learning, searching, and writing. Often to help them perform their work!

Thinking about the two plots above, I am reminded of the iPhone’s arrival in the enterprise. When the iPhone arrived, it was not seen as a work device. IT organizations continued to provide BlackBerrys, with their IT-controlled email and messaging. Nearly all IT teams didn’t think this would change. More than once, I heard IT managers reply to iPhone support requests with, “Just wait for the BlackBerry Storm.”

But you know who loved iPhones? The C-suite. And they asked their IT leaders to support the device. IT caved, “Bring Your Own Device” became a thing, and four years later Apple was an option in the enterprise.

Which brings us back to the charts above: employees are using ChatGPT while managers grumble that their AI projects aren’t adopted. If I had to guess, I’d wager there are a few things going on:

  1. Most companies adopt AI products slowly, bottlenecked by legal and security. There’s a reason you see Llama 3.1 continue to show up in McKinsey surveys: once teams win approval to use a model, they are loath to go back to compliance to seek an upgrade. New models emerge monthly, but security reviews take many months. This applies to AI applications as well: if a company buys one and employees tell them it’s not great, no one’s eager to take on legal again.
  2. Bundle deals are poor substitutes for great chatbots. I’ve heard from many friends that their workplace-provided chatbot were selected for security and trust reasons (think Microsoft Copilot and others). Rather than wrestle with bad UX or bad answers, these people opt for BYOAI (bring your own AI), IT concerns be damned.
  3. It’s hard to separate personal from business use. This is a classic IT problem: when people can’t be bothered to switch accounts before asking a question. We see it with email, browsing, and more. Savvier users quarantine accounts in separate browsers, but most people just use what’s there.

The topic deserves further study – I don’t think this will be as easy as the iPhone and BYOD was. But I do think the dominant bottleneck here is IT and compliance. If enterprises don’t stand up continual review processes, they’ll be doomed to be stuck with last year’s tools and models… Then wonder why no one is adopting their AI.

Until then: employees will continue to opt for BYOAI.

  1. NANDA stands for, “Networked AI Agents in Decentralized Archtecture.” ↩

  2. To Anthropic’s credit, they’ve already published several usage reports. ↩