The Strawberry Juice is Not Worth the Squeeze

I stress tested OpenAI's Strawberry... and was not impressed

UPDATE: This blog post aged poorly.

For some context,

OpenAI has just released their o1-preview model, which is a beta test of their first “Orion” series models that use “Strawberry.” If all that sounds somewhat cryptic it’s because they’ve deliberately chosen meaningless keywords for their secret projects in the same way the CIA and NSA does. Like the upshot knothole nuclear test. The words were more or less chosen randomly. I suspect OpenAI did this because Jimmy_Apples (a Twitter leaker) disclosed their “Arrakis” model back in 2023, and let’s just say the internet collectively disapproved of that model name.

So what is Strawberry and what does it do?

If you want to know a little bit more about it, you can take a much closer look on OpenAI’s release blog for the new model. The very short version is that it uses a form of self-prompting called “Chain of Thought” (CoT) reasoning, which is nothing new. The first CoT paper was published at NeurIPs back in January 2022.

OpenAI has obfuscated the reasoning from users, primarily because this deepens and widens its moat. Self-prompting (or what I originally called “metaprompting” way back in June 2022, is nothing new but it isn’t the easiest to integrate into a stable, commercial project. So, long story short, OpenAI have commercialized a technique that is almost 3 years old.

Great, but what does this actually do?

The simplest analogy is that it enables LLMs to “think through” or iterate on thoughts. If a single-shot from ChatGPT is like Daniel Kahneman’s system 1 thinking then Strawberry enables system 2 thinking. While this is certainly a step in the right direction, it’s nothing new, and this is the direction I’ve been saying it needs to go all along. In other words, it’s a rudimentary cognitive architecture.

Cognitive Architecture

Below is a video I released in May 2021 of a chatbot I built for Discord called RAVEN. This was my inaugural AGI project where I focused on alignment via cognitive architecture and multi-step self-prompting.

In fact, I published my very first book, Natural Language Cognitive Architecture, just over two years ago, in the early days of 2022.

You might be wondering: Dave, why are you driving home all these dates?

Great question: OpenAI has nothing new.

What this says about OpenAI

My read on this is that OpenAI has nothing innovative or new. As we’ve observed before, OpenAI makes good use of integrating the research of others and bringing it to market, but there’s no “secret sauce.”

In fact, when asked about building GPT-4, Sam Altman admitted that there was no big breakthrough, they were just good at integrating hundreds of little improvements. Almost all of which came from the scientific research community. Such improvements include papers on attention windows (basically, algorithms that allow ChatGPT to “pay attention” to the right context of the conversation).

Still Fails Simple Tasks

During my livestream to explore o1-preview with my audience, I was given some “teaser questions”

write a 10 word coherent sentence where each word has exactly 1 letter more than its previous word starting with first word having one letter and so on till tenth word having 10 letters

This first teaser question ultimately yielded no results. I killed it after 230 seconds.

My audience, however, came up with a few answers:

• I am not free today, Jordan, because galaxies regularly caramelize

• I am not some great prompt command follower brilliant mastermind

Okay, these aren’t super coherent, but close enough.

Here’s another test that it completely failed at (never rendered results)

A city adds more buses to reduce delays and overcrowding during peak hours, but this leads to increased traffic congestion, higher fuel costs, and continue passenger dissatisfaction. Despite more buses, schedules remain unreliable. Identify the feedback loops and unintended consequences causing these issues. Propose a solution that addresses the root causes without creating new problems elsewhere in the system.

I looked at this for about 15 seconds and came up with two explanations:

1. The number of buses was not the constraint, the chokepoint is elsewhere.

2. Look at other things, such as bus routes, stops, fueling depots, and schedules.

So clearly it’s “reasoning” abilities are still limited, even on simple tasks. Claude, by the way, barfed out a good-enough answer immediately. So in this case, the juice is not worth the squeeze.

Now, with that being said, there were a few tests that it did well on. Here’s one that requires a lot of reasoning, and it ultimately figured out a good solution for:

You are faced with three gods, A, B, and C. Their names are True, False, and Random, but you do not know which is which. True always tells the truth. False always lies. Random sometimes tells the truth and sometimes lies—how Random behaves is completely unpredictable. You have to determine the identities of A, B, and C by asking three yes-no questions. Each question must be addressed to one god only, but you are allowed to ask different gods different questions. The gods understand English, but will answer all questions in their own language, where the words for "yes" and "no" are "ja" and "da," in some order. You do not know which word means "yes" and which means "no."

Strawberry iterated through each God and question and concocted a hypothesis about what question it would ask each one, and what it’s answer would mean either way. I don’t know if the solution it rendered was accurate because the livestream moved on, but I was impressed with the reasoning either way. It took about a minute or two to render.

The Juice Isn’t Worth the Squeeze

So, now we have a model that uses Chain-of-Thought reasoning, whether it’s appropriate or not, that costs 100x their current flagship model. This is not an exaggeration.

4o-mini, which powers most ChatGPT sessions, costs $0.15 per million input tokens

Now, by comparison, o1-preview costs $15 per million input tokens

This is literally 100x the price

My diagnosis is somewhat grim: OpenAI are compensating for lack of underlying model progress by simply throwing tokens (money) at the problem. LangChain, an open source startup, makes Chain-of-Thought easy and practical, and has been around for more than a year already.

So, OpenAI integrating chain-of-thought prompting into their flagship model is sort of like when Google and Apple integrated Swype (slide typing) into their keyboards.

I’m also not the first person to notice that the hype is tamping down, and I’m also glad for it.

AI companies are pivoting from creating gods to building products. Good.

There have been a slew of financial and business commentators saying that the honeymoon is over, that generative AI is headed for the trough of disillusionment. AI companies can’t find product-market-fit let alone price-market-fit.

We are now solidly in the reductive spiral. Source: https://www.linkedin.com/pulse/ai-hype-cycle-2024-cowan-plus-g4jbc/

You only have to ask yourself one question:

Is the marginal improvement in reasoning worth 100x the price?

The value prop is looking thinner and thinner.

Corporations Doing What Corporations Do

I’ve made the sausage, I know how to finetune models. Here’s an example of what I mean: I was finetuning ChatGPT-like chatbots 6 months before ChatGPT was released. There’s no magic to it, but if you’ve never done it, it can look like magic to the outsider.

OpenAI is now just gobbling up cool features that other people invented and perfected. They are no longer a science or research company, which would explain the mass exodus of actual scientists and researchers.

Long story short: OpenAI is very clearly following in the footsteps of Google, Microsoft, Meta, and Apple. They are simply becoming a business productivity shop, and there’s nothing wrong with that. It’s probably why Sam Altman has been making less noise about AGI lately.

Where I’d Go From Here

I don’t envy OpenAI or the other big AI shops. I’ve been saying for a while that AI is slowing down. I got more than a little hate for that observation, and OpenAI’s flop of a Strawberry release will almost certainly be the final nail in the coffin for a lot of business stakeholders. Sam Altman, and others at OpenAI, have talked about “intelligence too cheap to meter” which has fallen completely flat with the release of o1-preview.

Open source models are not that far behind frontier models. Meta is working on commoditizing AI. If your business is predicated on a single tech (LLMs) your days are probably numbered. Unlike CPUs, which have dependencies and architectures, LLMs are totally fungible: they all use natural language. Most of the data comes from the internet. Most of the research comes from arxiv.

• OpenAI: Being joined at the hip with Microsoft sealed their fate long ago. Anthropic threw in with Amazon, which makes sense. The Big Tech companies, the actual companies with a durable business model, were always going to win. If I were Sam, I’d be looking for the exits, to start something new and fresh like what Ilya did. I bet this is why Greg Brockman is taking an extended leave of absence: they got nothing, their mission is winding down, and the noose is closing.

• Microsoft: If I were Satya, I would probably start to tighten my grip, and start squeezing. OpenAI is not any closer to AGI, and the marginal utility of keeping OpenAI separate is diminishing. Microsoft has some stellar in-house R&D teams, plus experimental product teams. The usefulness of OpenAI is falling off a cliff. However, due to inertia and optics, Microsoft will have to play nice and let this play out naturally. OpenAI is running out of money, surviving on investment, and someone is going to have to pay up, one day.

¯\_(ツ)_/¯

UPDATE: I realized I didn’t quite say where the path forward it. The path forward, to me, is very obviously multi-agent frameworks, such as Microsoft’s AutoGen or AgentForge.

From a business POV, you’ll have more transparency, explainability, and flexibility with multi-agent systems. They also allow you to bake in model routing (basically performing model arbitrage to optimize costs) and tool use, all baked in.

Advantages of multi-agent systems versus these expensive models include:

• Model fungibility: you can swap out faster, cheaper models when it makes sense, and use big expensive models only when it counts. This does several things: saves a ton of money, and makes the overall framework much faster (rather than relying on a single gigantic model). This is a permanent feature of optimization. TLDR: Exploit the fast/cheap resource as much as possible.

• Tool use: Pick models that know how to use specific tools (function calling, API calls, etc). That includes writing code, calling APIs, and basically creating an extensible framework. It’s sort of like a microservices architecture with AMQP routing.

• Flexibility of instantiation: When you have a bunch of agents, each with specializations, you can spin up or spin down agents just like we already do with containers (tiny virtualized applications) and microservices. In other words, enterprises already have a model to manage fleets of ephemeral worker nodes. Agents are the new worker nodes.

• Transparency and traceability: You can monitor the communication between agents, with supervisor layers, manager layers, resource and RBAC layers, etc. Building one monolithic model that does all its reasoning in a blackbox is untraceable. Especially with OpenAI obfuscating the reasoning. I don’t think enterprises, governments, or militaries will be willing to adopt that.