A Stalwart Shrivels

Yesterday, George Orosz pointed out quite the trend, documented by Theodore R. Smith:

The number of questions being asked on StackOverflow has been in decline since 2017 (after reaching a peak in 2014). But the gentle slope turned into a nosedive with ChatGPT’s arrival.

Chatbots didn’t kill StackOverflow, but they certainly drove the final nail in the coffin.

Let’s look at Google Trends to see this from another angle:

Yeesh…

As if this chart wasn’t brutal enough, both of these lines are indexed relative to themselves. Meaning: their top search volume is pegged to 100 and all other points are calculated relative to this peak. If we compared them on the same index (which is how Google Trends displays these stats), StackOverflow doesn’t even register. Sure, it’s a specialized site compared to ChatGPT’s mainstream appeal, but still…

But is this decline even for all languages? Kinda:

From 2023 to 2024, everything is down significantly, but it does vary. The C-family of languages (C, C++, C#, and culturally we’re including Rust here) are only down 35-40%. While the dominant scripting languages (Javascript, Python, Ruby) are down by roughly half.

Anyone who’s coded with an AI knows they’re better with popular languages and APIs. These things are a reflection of their training data, after all, and Python and Javascript’s dominance over the last decade means they’re a larger share of the training corpus. Plus, their current popularity among the users AI companies are chasing means these languages are given plenty of attention during the RL phase.

If you stray away from Python, React, or other popular languages and frameworks, vibe coding gets more perilous1. For example, last week I spoke with a CTO who said AI usage among their devs was going well but they were lamenting their past choice of Vue over React. I had expected the StackOverflow question trends to reflect this dynamic, but it’s tenuous at best given this data.

What Do We Lose When Using Chatbots Instead of StackOverflow?

As someone who’s been programming for the duration of these charts, my immediate reaction is one of sadness. On further reflection though, it’s frustration.

We lose something when people stop posting their problems online. There’s something magical and reassuring when you Google a confounding bug and discover your exact challenge was not only shared but also solved. The poster and the responses were a community you found in frustration.

With ChatGPT you’re also finding problems and solutions experienced before, but the answers have been laundered through the model. More importantly, your problem is never publicly posted if you’re first. It’s just between you and the AI you’re interacting with, potentially memorialized as training data for future weights.

As you code, you develop a sense for what problems the model’s seen and which are likely new. In a previous piece, we called this Déjà Connu:

The speed, amount, and quality of Copilot feedback is a valuable signal. It’s a signal of what’s been done before, what’s been done often, and what’s been done well. It tells you when you’re trodding familiar terrain and when you’re breaking new ground.

Sometimes the quality and amount of feedback will surprise you. “I thought I was working on something unique,” you’ll think as you review the 20 lines of code Copilot just wrote for you. But it turned out to be a common problem…I’m seeing my own idea, but it’s known before I had a chance to write it. It’s déjà connu, already known.

With better fine-tuning infrastructure, our problems and solutions could live on as weights for future users. But because the experiences are washed by the model, there’s no community with which you can engage.

But who are we kidding: StackOverflow was in decline long before ChatGPT arrived.

It was undercut by better, more supportive communities that continue to thrive today, namely Discord and Slack channels. Sure, the opaqueness of these communities to search engines and links is frustrating, but the level of community support is so much better than StackOverflow ever was.

And that’s why these charts make me frustrated.

At one point, StackOverflow deflected its toxic reputation by pointing to its impressive signal-to-noise. And at one point this was hard to argue against! There was a reason StackOverflow was one of Google’s most frequently crawled sites. But half a decade later, new users found StackOverflow dated, intimidating, and stale. Young programmers deferred to communities on Discord, Slack, and others. Then they embraced AI.

StackOverflow will live on in the weights.

  1. Just this week I asked o3 to write me a spatial join using DuckDB. It first hallucinated the spatial function used for the join. I then provided the documentation and asked it to find the real function that achieved its goal. It found one, but screwed up the provided parameters. I gave it one last shot. It thought for awhile and returned a join statement with the spatial function commented out because it was “too difficult.” ↩