test two models blind on your own jobs | LAB 0017 field notes


Lenny's AI Builders - LAB 0017

pick three jobs you really do, give both models the same brief, and rate the work blind.

i ran my own benchmark on Claude Haiku 5.5 and GPT-6 Luna. i rated every job before i knew which model made it.

then i saw the scores. i was not expecting that.

so remember this: pick your own benchmark.

why it matters

a model can be cheap per token and hungry per task. check how much a model costs per job, not per price tag.

my guess was that Haiku 5.5 would be better than Luna, but much more expensive. on my benchmark it cost more and it did not score better.

you only find that out when you test both on work you really do.

what you get

a simple test you can run on any two models.

  • the same jobs. both models get the same brief.
  • a blind score. you rate the work before you know who made it.
  • cost next to quality. you see what each run cost beside its score.

the real example: BuilderBench

BuilderBench is my own benchmark. it has 10 tasks i care about: writing, a landing page, a lead magnet, video editing, motion, thumbnails and titles, browser and computer use, research, automation and founder conversations.

i ran it on 8 October 2026. both models ran on extra high for four hours.

  • GPT-6 Luna got 44.58 out of 100.
  • Haiku 5.5 got 39.18 out of 100.
  • Luna leads by 5.4 points.

then the cost. the run cost estimate for Luna was $1.47. for Haiku 5.5 it was $37.39. these are estimates, not bills.

my take: this made me really appreciate GPT-6 Luna for what it is. i severely underestimated it.

but the best model still wins. on my benchmark, over time, the best model is Opus 5.5.

try it

  1. open your agent's prompt box.
  2. paste the prompt below.
  3. add the YouTube link to LAB 0017, so it can see how i ran mine.
  4. choose three jobs you really do.
  5. rate the work before you know which model made it.

copy this prompt

paste this into your agent. give it the link to the episode too.

Pick three jobs you really do. Give both models the same brief. Rate the work blind.

you'll know it worked when

you have a score for each model on each job, and you wrote it down before the reveal.

you can put the cost of each run next to its score.

a bad result: you knew which model made what, or you tested jobs you never do.

limits and dates

this is one run on 8 October 2026. my jobs, my taste. your scores will differ.

the scores are my final personal scores. some of the evidence is partial.

the costs are API-price estimates, not bills, and the two were worked out in different ways. Luna's is a short-context estimate. Haiku's is the provider's own estimate. in the episode i say Luna's run was "like 25 times cheaper". that's a rough spoken number, not an exact one.

LAB 0017 went public on YouTube on 9 October 2026 at 6:26 pm Perth time.

every score, price and limit: lennoxsaint.com/lab/builderbench/2026-10-08-luna-haiku

the rest of this LAB

  • Elon's Grok Bot will use Claude. Elon said that, going forward, Grok Bot will use the best back-end model for any task, including Claude Opus 5.5. right now it won't tell you which model it's using. my take: pick your model, or pass.
  • Grok Bot can watch X. it can search, read and monitor X. paid users get free X API credits to start. the X API costs half a cent for every post you read. 10,000 posts is $50.
  • Claude Haiku 5.5. 10 cents in and 50 cents out per million tokens, up to 100,000 tokens. above that it's five times more. Artificial Analysis scored it 43 against Luna's 38 - that's their score. at max effort it uses about three times Luna's tokens. my take: let Haiku do grunt work.
  • monthly API credits for Max and Team. Max 5x gets $100, Max 20x gets $200 and Team gets up to $500, pooled. they expire every month. my take: claim it before it expires.
  • GPT-6 and Intelligent UI in ChatGPT. GPT-6 can now answer with text, visuals and interactive elements. it's in the Chat tab only. the models in Work and Codex don't change. my take: plain text answers are dead.
  • OpenAI's day 3 reset. every paid account got a banked reset. my take: i'd rather have my 20x usage back. the Pro limits drop from 20x to 10x at the end of October.
  • Theo's $400,000 lesson. Theo says he burned about $400k of Codex tokens and got nowhere, then about $20k of Opus and got there in two weeks. those are his numbers. my take: the wrong model burns $380k.

sources

next step

watch LAB 0017 on YouTube - the practical starts at 10:26.

want the next one? get the next field notes. optional - these notes stay free to read.

Lenny’s AI Builders

i’m Lennox. i sift through AI Twitter and share one thing i tried with ChatGPT. for people making products, content and useful systems. five emails a week, with a weekly option.

Read more from Lenny’s AI Builders

Lenny's AI Builders - LAB 0045 one Claude Code setting, 36 to 38% fewer tokens in every pair of my test, the catch on cost, and a prompt that gives you 10 versions. if you build with Claude Code, one new setting can make it use fewer tokens. i tested this myself. every pair of runs used 36 to 38% fewer tokens. but there is a catch. my first run cost 83% more at api prices. the other two pairs came out 15% and 17% cheaper. why it matters the setting turns on Code Mode. Claude writes code that...

Lenny's AI Builders - LAB 0042 the one Grok Bot upgrade that matters, and the paragraph i add to claude.md for long Fable 5.2 jobs. Grok Bot 1.0 isn't new, but the launch post pulled 46 million views like it was. so don't worry, you're not missing out. the one upgrade that's actually decent got something like 40 times fewer views. and it let my bot walk into my Mac. why it matters Grok Bot has been out since August. xAI's own page calls 1.0 a major update to its AI teammates. the headline...

Lenny's AI Builders - LAB 0041 the conductor paragraph, the four steps to run it, and the honest numbers from my 40-hour run. i ran my viral Fable 5.2 conductor for 40 hours on my real app. one conductor, 73 worker threads, 25 pull requests merged. and my weekly Fable limit was gone in 13 hours. so here's what it actually built, and the one paragraph that runs yours. why it matters this is the new meta. you use Fable 5.2 as the conductor and Opus 5.5 as the worker threads. Fable 5.2 is twice...