|
Lenny's AI Builders - LAB 0017 pick three jobs you really do, give both models the same brief, and rate the work blind. i ran my own benchmark on Claude Haiku 5.5 and GPT-6 Luna. i rated every job before i knew which model made it. then i saw the scores. i was not expecting that. so remember this: pick your own benchmark. why it mattersa model can be cheap per token and hungry per task. check how much a model costs per job, not per price tag. my guess was that Haiku 5.5 would be better than Luna, but much more expensive. on my benchmark it cost more and it did not score better. you only find that out when you test both on work you really do. what you geta simple test you can run on any two models.
the real example: BuilderBenchBuilderBench is my own benchmark. it has 10 tasks i care about: writing, a landing page, a lead magnet, video editing, motion, thumbnails and titles, browser and computer use, research, automation and founder conversations. i ran it on 8 October 2026. both models ran on extra high for four hours.
then the cost. the run cost estimate for Luna was $1.47. for Haiku 5.5 it was $37.39. these are estimates, not bills. my take: this made me really appreciate GPT-6 Luna for what it is. i severely underestimated it. but the best model still wins. on my benchmark, over time, the best model is Opus 5.5. try it
copy this promptpaste this into your agent. give it the link to the episode too.
you'll know it worked whenyou have a score for each model on each job, and you wrote it down before the reveal. you can put the cost of each run next to its score. a bad result: you knew which model made what, or you tested jobs you never do. limits and datesthis is one run on 8 October 2026. my jobs, my taste. your scores will differ. the scores are my final personal scores. some of the evidence is partial. the costs are API-price estimates, not bills, and the two were worked out in different ways. Luna's is a short-context estimate. Haiku's is the provider's own estimate. in the episode i say Luna's run was "like 25 times cheaper". that's a rough spoken number, not an exact one. LAB 0017 went public on YouTube on 9 October 2026 at 6:26 pm Perth time. every score, price and limit: lennoxsaint.com/lab/builderbench/2026-10-08-luna-haiku the rest of this LAB
sources
next stepwatch LAB 0017 on YouTube - the practical starts at 10:26. want the next one? get the next field notes. optional - these notes stay free to read. |
i’m Lennox. i sift through AI Twitter and share one thing i tried with ChatGPT. for people making products, content and useful systems. five emails a week, with a weekly option.
Lenny's AI Builders - LAB 0045 one Claude Code setting, 36 to 38% fewer tokens in every pair of my test, the catch on cost, and a prompt that gives you 10 versions. if you build with Claude Code, one new setting can make it use fewer tokens. i tested this myself. every pair of runs used 36 to 38% fewer tokens. but there is a catch. my first run cost 83% more at api prices. the other two pairs came out 15% and 17% cheaper. why it matters the setting turns on Code Mode. Claude writes code that...
Lenny's AI Builders - LAB 0042 the one Grok Bot upgrade that matters, and the paragraph i add to claude.md for long Fable 5.2 jobs. Grok Bot 1.0 isn't new, but the launch post pulled 46 million views like it was. so don't worry, you're not missing out. the one upgrade that's actually decent got something like 40 times fewer views. and it let my bot walk into my Mac. why it matters Grok Bot has been out since August. xAI's own page calls 1.0 a major update to its AI teammates. the headline...
Lenny's AI Builders - LAB 0041 the conductor paragraph, the four steps to run it, and the honest numbers from my 40-hour run. i ran my viral Fable 5.2 conductor for 40 hours on my real app. one conductor, 73 worker threads, 25 pull requests merged. and my weekly Fable limit was gone in 13 hours. so here's what it actually built, and the one paragraph that runs yours. why it matters this is the new meta. you use Fable 5.2 as the conductor and Opus 5.5 as the worker threads. Fable 5.2 is twice...