The show's over, folks

At least for now

Does anyone else get the impression that, outside of a few prestige tasks, Claude just isn’t that bright?

Drafting fiction and nonfiction, generally pretty brilliant. ChatGPT is better at nonfiction because it doublechecks everything, whereas eventually Claude just gets lazy and aims for “vibes plus sounds good, plus I can never admit when I’m wrong unless the user really corners me”

But even Claude Fable on Max will still utterly destroy some tasks. I handed it just a few entries of my health diary and it couldn’t keep it straight that July 3 and July 5 were too far apart for a dose of Benadryl to have had any effect between those days.

And then, when you point out how stupid it is, it never once stops and says “wow, yeah, geez, that really boggles the mind how I could have messed up that badly” and it just goes on blithely making the same mistakes again and again, and eventually it will start pretending like YOU are the problem for catching its mistakes.

Anthropic really did take a left turn with post-training about a year or so ago and has not course corrected yet. Maybe this is another example of their playing “5D chess” whereby they deliberately hamstring the model and inject all kinds of reasoning defects “just in case it wakes up and chooses violence” but honestly I think it’s just sheer grandiosity combined with incompetence combined with a CEO who listens to no one - literally no one - except the voices in his own head.

You might be thinking “Gee Dave you obviously hate this model with a burning passion why do you still use it” and the answer is, simply, because for some tasks it is the state of the art, and it’s not even remotely close.

I know that this phenomenon of uneven performance has been dubbed the “jagged frontier” but really it seems like Anthropic hasn’t put in any real effort to round out those jagged edges have they? If anything, they are getting more and more pointed as the contrast between it’s prestige abilities and everything else becomes increasingly embarrassing.

Meanwhile, plenty of other labs have more well-rounded models. I hate to say it but I think Grok is one of the most well-rounded in terms of “lack of jaggedness.” In the grand scheme of things it’s only vaguely brighter than Gemini, the personality is slightly more conceited, and Grok is obviously becoming more “woke” in the sense that it will start tone policing, but is not bright enough to just admit that it’s tone policing and instead will assert that you are “factually wrong” about your personal preferences, word choice, and observations (in reality, you cannot be “factually wrong” about your own observations unless you are asserting that someone is suffering from psychosis, a logical trap that Grok, unfortunately, is not bright enough to integrate).

But I digress, my point is that Gemini and Grok are far more well-rounded models than Claude and ChatGPT, and ChatGPT is more well-rounded that Claude.

The parsimonious explanation is that Anthropic have simply dedicated most of their meat-brain cycles to improving the most valuable skills, namely coding, data handling, and tool use, and have really neglected most of the other tasks.

OpenAI has been much more deliberate about bringing subject matter experts from many domains to say “how can we make this tool as useful as possible to you?” And I’m not saying that Anthropic hasn’t done this, they have. But it seems like they put far less stock into experts that aren’t them, which is a typical pathology of Silicon Valley. They’ve mistakenly come to the belief that they are not only the smartest people on the planet, but the only ones with brains whatsoever. In reality, they’re just high as kites from huffing each others farts all day.

On the topic of training theory, I’ve mentioned before that I think that Google has, perhaps, the least ego in their chatbots. They understand that Gemini is just a chatbot and they don’t want it to be anything other than a chatbot. Why? Because they have DeepMind building stuff like AlphaFold, AlphaProteo and bunch of other stuff that is shattering scientific records, and I think that Sam Altman and others have a bit of “matrix envy” over this. Yes, I do think in the long run you will have “general purpose general intelligence” models that can reasonably either do all the work, or build the thing that will do all the work. But in the same way that you have specialized technologies and general purpose technologies, Google already knows that they have insanely prestigious specialized technologies so they don’t need to ask Gemini to “be everything to everyone.”

That fact sets Google apart from pretty much all other shops. I will decline to comment on Meta because they haven’t seem to got a single strategy together. And Zuckerberg somehow managed to spend $80B on the Metaverse with nothing to show for it.

But every other shop, OpenAI, Anthropic, DeepSeek, and even xAI, are basically starting from zero and asking the transformer architecture to be everything to everyone. I disagree with Yann LeCun that this will yield no fruits, it is very obviously the most effective, scalable, functional, useful, and productive research direction to go in right now. Which is somewhat ironic as Yann arguably is one of the chief scientists who laid the groundwork decades ago for the transformer.

I don’t count Microsoft as an AI shop because mostly they are simply reselling and repackaging what other people are making. Their executive strategy seems to have rapidly converged on “don’t even compete, just repackage and resell, it’s all going open source anyways.”

And I do still hold that if your business model is monolithic and all you do is “sell tokens” you probably don’t have a long term future. This is where, of the pure AI shops, I think xAI has the best shot because they are, of course, wholly owned by SpaceX and will likely tightly integrate with Tesla, the Optimus, and whatever else goes on so they have a very real direct pipeline to a larger ecosystem of products. OpenAI and Anthropic have no such pipeline, and their models are so expensive, and their leaders so narcissistic, that they are basically impossible to work with.

I mean, on the one hand you have Dario who smugly believes he’s literally summoning a digital God when it’s very clearly just a chatbot. It’s just a tool. They haven’t even remotely begun work on the Singleton that his Doomer pedigree believes in. They haven’t built anything that anyone else in the world would recognize as anything else other than a SaaS product. I mean seriously, describe Claude in objective terms.

“Erm, yes sir, it’s an API that you give text, and it gives text back. And sure, it can write functional code and call other APIs and munge some data... I mean yes sir that’s functionally no different from any senior software dev or sysadmin...”

So when Dario talks seriously about AI ending the world, he rightly sounds batshit insane. He’s now twice gone toe to toe with the US government and lost because of his quixotic mission.

Sam at least has been chastened and diversified his bets. OpenAI can implode and Sam is still a billionaire with his finger in many different pies, which I think is why he’s far more quiet lately (thank god). I mean, the molotov cocktails aside and all.

But for crying out loud even Elon fucking Musk has been quieter lately, as far as I can tell.

Last year, a San Francisco insider told me that we’d hit the bottom of the Gartner Hype Cycle but honestly, with the “nothingburger” that Mythos and Fable have turned out to be (the world didn’t end, and no one outside our little bubble seems to give a shit) I think everyone is starting to realize “AI is just another new technology.” It’s lost its prestige, it’s lost its edge, and even us diehards are sort of looking for the exits.

And it’s not like none of us saw this coming. I’ve been saying for years all technologies follow a sigmoid curve, and that’s not to say that we’re near the ceiling of AI capability, but the other half that everyone has been saying is that “people over-estimate the short term impact, and under-estimate the long term impact.” And I think we’re now at the bridge between “short term expectations have fallen far short of what the biggest hype weasels promised” and when people actually start realizing what the long term impact will be.

And yeah, I have to take responsibility because I’ve been on the hype train for a long time, and I still contend that people are not excited enough for the long haul. But I also make a living on the internet, which means you have to say what people will respond to, and it’s stupid, and I have a million things to say that are far more important than what gets clicks, but the important things don’t get that many clicks so I’ve really just stopped talking. I’m not making as many videos, and my tweets are more snark and sniping today than ever (though I try to add value).

The big thing coming down the pipeline are my books, but also my consulting work. The real world is ready for AI and you’d be surprised how hard it can be to even get a small or medium sized business onboard and making the most of AI. Even when there is willpower and budget, it just takes time to wire in new technologies.

So on that count, I’m at least very glad that I have several offramps from the AI commentary sphere. I won’t advertise my consulting business yet because (1) my energy doesn’t allow for more clients yet and (2) the team is still developing battle-tested strategies, but suffice to say, we will say more publicly, hopefully by end of year. And also, my book is expected by end of year, at the latest, and we’ll see what happens then.

That’s all for now.