Mistral123 = Haiku > Llama405.
It's hard to trust benchmarks so I have some of my own. This one's the best to look at: Drawing capability between Mistral 123, Llama 405, Haiku, 4o, and Sonnet.
Before I tell you which is which, lock in an answer about which one you like.
First off, here's mine, and here's Sonnet's. I can't draw but I like the Sonnet version.
Next is 4o and 4o mini. 4o is the one with a ton of lines. It got the logic exactly right, and the lines are a personal choice I don't like too much.
4omini is also pretty perfect except for the floating nodes.
Mistral 123 (or Mistral large 2) is one of my favorites. Good layout, good colors, got most things right except the storage layer (which isn't **wrong** per se but it's weird).
At 123B params this is super impressive
Haiku too does a pretty good job of not overcomplicating things, and structuring everything well.
Llama405 was a complete disappointment. Multiple tries and prompt edits, it just couldn't make a mermaid without syntax errors (even on multiple providers too). This is after 10 minutes of fighting it (and getting Haiku to fix the syntax issues). All the others are zero-shot.
Same story with Nemo (syntax issues), but it's a much, MUCH smaller model.
All of the prompts are using this blog post as the source, and given the same instructions (except llama405 just to get it working) to build a diagram.
What am I doing wrong? In all the tests I've run, llama405 is slightly worse than Haiku at instruction following.
x.com/hrishioa/status/1808109617408626722
Hrishi
@hrishioa
In SF in March - Building artificially intelligent bridges at Southbridge, prev-CTO Greywing (YC W21). Chop wood carry water.