End of an era: Haven't seen anyone do this analysis, and I was curious
This is my personal history with the chats of Claude and GPT over more than a year.
Take the journey with me ๐งต๐
You think you use the best products but you're susceptible to marketing like anyone else. We can see predictable spikes in usage with releases (except for 3.5 sonnet, was overseas when that happened)
But real usage/engagement peters out over time. But man is there a new king
Tokens exchanged is interesting. Anthropic crossed the 'non-lazy-model' barrier first (and with a higher context length) which means a lot more tokens given and got.
4o solved this problem on the OpenAI side, but by then they didn't have the best model so it didn't matter
Actually I lied (or realised I was computing things wrong). Anthropic puts your input files into attachments somewhere else in the object. If you add these into the tokens, it looks wildly different
Overall gpt usage starts to condense into a tiny line
Looking deeper into tokens, you can see just how much more output you can get from Claude. Chat is a good place to check this, since you can't automate it like you can with APIs. Everything's rate limited by you as a human.
What's interesting is that the chart above has openai user tokens including both what I typed and what I copied, and for anthropic it's just what I typed. If I add in attachments we get this!
We can measure tokens per second using the server timestamps. We see a clear improvement in stability and speed after 4o for openai.
However, Claude feels faster. If you test with the request, it actually is! Weird. I'm getting 75 TPS on Sonnet compared to even 4o.
Had to recheck the results, but it holds up. Claude (Sonnet I'll add) gets a later start than gpt-4o but it gets done with the diagram first.
Initially I'd thought it was just me using gpt-4 instead of 4o, but no Claude is faster.
Looking at the request, it holds up. If you just look at it from the point tokens start streaming, Claude actually gets 95 TPS. But they take a while to allocate your request and start compared to openai.
It's really weird to see your life in numbers - but since 3.5 Sonnet it's become so easy I've started doing more of these
Follow if this was fun!
Hrishi
@hrishioa
In SF in March - Building artificially intelligent bridges at Southbridge, prev-CTO Greywing (YC W21). Chop wood carry water.