AI did not remove the need for product judgment.

It removed the excuse for not having any.

When agents can produce five flows, three prototypes, two launch plans, and a passable landing page by Friday, the hard question changes. It is no longer “Can we build it?” It is “Which version is worth shipping?”

That scarce work is taste.

By taste, I do not mean aesthetics. I mean product discrimination: the ability to tell useful from noisy, sharp from generic, finished from merely working, and necessary from extra.

The taste tax is the bill that comes due when execution gets cheap.

Subscribe now

When building gets cheap, choosing gets exposed

At Notion, the AI shift did not start with designers making prettier static mocks.

It moved the work into a shared prototype playground.

Brian Lovin, a designer on Notion AI, built an environment where designers could turn ideas into working prototypes with Claude Code. The point was not that every designer should become an engineer. The point was that AI products are hard to judge from static screens.

You have to feel the system behave.

A prototype that answers can still be wrong. It can use the wrong tone. It can hide the moment where trust breaks. It can complete the task and still feel generic in the user’s hands.

That is the right scene for the taste tax.

AI made the first version cheaper. It did not decide whether the interaction was good.

The new bottleneck is not making the artifact.

It is judging it while there is still time to change it.

Taste is not decoration

Taste is often treated as a soft word. That makes it easy to ignore.

Do not ignore it.

Taste is the operating standard underneath product work:

• What should we cut?

• What should we polish?

• What should we call this?

• Which version helps the user think less?

• Where does the feature cross from useful into trying too hard?

• What would make this feel like us if the logo disappeared?

Those are not decoration questions. They are product questions.

Before AI, weak taste was partially hidden by execution cost. Building was expensive, so fewer bad ideas made it all the way to users.

Now the gate is lower.

Bad taste ships faster too.

The failure mode is generic abundance

The first-order effect of AI is more output.

More drafts. More screens. More tests. More strategy docs. More prototypes. More “pretty good” versions of almost everything.

That sounds like leverage. Sometimes it is.

But abundance has a failure mode: teams stop discriminating. The roadmap only adds. The interface gets busier. The product starts to look like every other product built with the same models, the same templates, and the same unwillingness to say no.

This is the taste tax in the wild:

• The team can name twenty things to build and nothing to remove.

• “The model wrote it” becomes a defense.

• Code review catches bugs but not mediocrity.

• The definition of done ends at “it works.”

• Nobody can name a product they admire and explain the standard behind it.

The product is not broken.

That is the problem. Broken work gets fixed. Generic work gets shipped.

The signal is what the team refuses

The easiest way to see taste is to ask what the team refused.

A team with taste can show you the versions they rejected. They can explain why a feature was removed, why a label changed, why one interaction got another hour and another got killed.

A team without taste can only show you the output.

This is why “move fast” became more dangerous in the agent era. Speed without discrimination converges to the mean. If everyone has access to similar generation tools, the difference is not who can produce more.

The difference is who can choose better.

James Bessen’s work on automation points to the broader pattern: technology often shifts work rather than simply removing it. When the task gets cheap, the judgment around the task becomes more valuable.

The artifact got cheaper.

The standard did not.

Install a quality bar, not a taste lecture

Do not tell teams to “have better taste.” That is not a process.

Make the standard visible at the moment work is about to ship.

Use one review with three questions:

• What did we refuse? Tests whether the team can cut. A good answer names a rejected version, feature, message, workflow, or interaction — with a clear reason.

• What did we elevate? Tests whether the team improved beyond “it works.” A good answer names one change that made the output clearer, simpler, safer, more useful, or more distinctive.

• What standard did we apply? Tests whether taste is shared or personal. A good answer is a reusable principle the next team can apply without the original reviewer in the room.

This keeps the advice industry-agnostic. The artifact might be a product flow, pricing change, policy answer, data workflow, onboarding moment, service script, or internal tool. The standard is the same: the team can explain why this version deserves to exist.

Over time, save the best answers. That becomes the team’s taste library: not a mood board, but a record of good decisions.

What changes Monday

Pick one AI-assisted feature your team is about to ship.

Do one thing before launch: run a refusal review.

The launch decision should not be: “Does it work?”

It should be:

Can the team explain what it refused, what it elevated, and what standard it used?

If the answer is mostly silence, do not call the work finished.

Call it generated.

The market will not punish you because your team is slow.

It will punish you because everyone is fast now, and your product looks like theirs.

Sources

• Lenny’s Newsletter — I haven’t written a single line of code in six months: https://www.lennysnewsletter.com/p/i-havent-written-a-single-line-of

• Lenny’s Newsletter — Why cultivating agency matters more than cultivating skills: https://www.lennysnewsletter.com/p/why-cultivating-agency-matters-more

• James Bessen — Toil and Technology: https://www.imf.org/external/pubs/ft/fandd/2015/03/bessen.htm

• Paul Graham — Taste for Makers: https://www.paulgraham.com/taste.html

An AI demo is not an eval.

That is the mistake behind a lot of brittle enterprise AI rollouts. The demo proves the model can answer one polished question in one controlled room. An eval — a test set of real cases, scored against a written standard — tells you whether the system can survive actual users.

Most teams still ship the demo.

That is what I am calling straw prompting: building an AI feature out of a hand-tuned prompt, testing it on the cases that already work, and launching before the messy cases arrive.

It looks like a house.

It is not ready for weather.

Subscribe now

The happy path is not the product

Air Canada learned this in public.

In 2022, a customer used Air Canada’s website chatbot while booking travel after a death in the family. The chatbot told him he could buy a full-price ticket, travel, and then apply later for the airline’s bereavement fare.

That was wrong.

The airline’s actual bereavement policy said the discount could not be requested after travel had already happened. The chatbot even linked to the page with the correct policy, but the answer it gave in the conversation was still misleading.

The customer relied on the chatbot, travelled, and later asked for the fare difference. Air Canada refused. In 2024, British Columbia’s Civil Resolution Tribunal found Air Canada liable for negligent misrepresentation and ordered it to pay damages, interest, and fees.

The product lesson is not “chatbots are bad.”

It is sharper than that.

Air Canada had the correct information on its own website. The failure was that the interactive answer surface did not reliably behave like the policy it was supposed to represent.

That is the straw-prompting pattern in the wild.

The company did not just need a chatbot that could answer bereavement questions. It needed a system that could notice policy conflict, ground answers in the current source of truth, and escalate when the answer affected money, rights, or obligations.

A working answer was not enough.

The answer had to be governed like part of the product.

The wolf is variance

The wolf is not a smarter model from a competitor.

The wolf is variance: the messy spread of real-world inputs that your demo never saw.

Users misspell things. They paste half a policy. They use internal acronyms. They ask two questions at once. They omit context. They ask in anger. They ask in formats your prompt writer did not imagine.

OpenAI’s eval guidance makes the point plainly: AI systems need task-specific tests that reflect real-world conditions, edge cases, and continuous change. “Looks good to me” is an anti-pattern, not a launch gate.

That is the trap. A demo that works ten times in a row does not tell you how the system behaves on the eleventh thousand.

The launch review should expose the failure mode

You do not need a 40-point governance checklist to spot straw prompting.

You need a launch review that separates symptoms from causes. Run it against five patterns:

• Polished examples, not error rates — the team tested the demo, not the system. Require a real-input eval set before launch.

• No one can name the top failure modes — the risk model is missing. Write the failure list before tuning the prompt.

• Output quality judged by vibes — the team has no shared standard. Create a rubric before reviewing answers.

• “Guardrails later” in the plan — controls are being treated as cleanup. Define refusal, escalation, and source-of-truth rules now.

• The prompt writer also wrote the tests — the eval is likely overfit to the happy path. Add messy cases from real usage, edge conditions, policy boundaries, and expert review.

The point is not to punish the team for moving quickly. It is to find the root cause while the cost of fixing it is still low.

Build the test before the prompt

The fix is not a giant governance program. It is a different order of operations.

Start with the test.

For any AI feature, collect 100 to 300 real or realistic inputs from the hardest part of the workflow. Include short requests, long requests, ambiguous requests, boundary cases, policy conflicts, and cases where the right answer is “I do not know.”

Then write the rubric.

What counts as correct? What counts as unsafe? When should the model refuse? When should it escalate to a human? What evidence should it cite? What must it never invent?

Only then tune the prompt.

The prompt has to earn its way through the mess. Otherwise you are not improving the product. You are decorating the demo.

The brick house has gates, owners, and reruns

Once the root cause is clear, prevention is boring on purpose.

Brick-house AI rollouts put four controls in place before launch:

1. A golden set. The small, trusted collection of examples that defines what “good” looks like. It includes normal cases, edge cases, and failures you never want to see twice.

2. A named owner. Someone owns the eval set, the rubric, and the launch threshold. Without an owner, quality becomes a meeting mood.

3. Escalation rules. The system knows when to refuse, when to cite the source of truth, and when to send the user to a person.

4. Scheduled reruns. Models change. Prompts change. Retrieval changes. Product policy changes. The system that passed Tuesday may not pass next Tuesday.

What changes Monday

Pick the next AI feature waiting for launch approval.

Do one thing: replace the demo review with an eval review.

The launch decision should not be: “Did the examples look good?”

It should be:

Did the system pass the real cases, against the written rubric, with clear escalation rules and an owner for reruns?

If not, do not approve the launch.

Approve the demo, if you want.

Just do not mistake it for a brick house.

Sources

• OpenAI — Evals: https://platform.openai.com/docs/guides/evals

• OpenAI Cookbook — Evals design guide: https://cookbook.openai.com/examples/evaluation/evals_design_guide

• NIST — AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework

• Deeth Williams Wall — BC Tribunal Finds Air Canada Liable: https://www.dww.com/articles/bc-tribunal-finds-air-canada-liable-for-inaccurate-advice-given-by-website-chatbot

• American Bar Association — BC Tribunal Confirms Companies Remain Liable: https://www.americanbar.org/groups/business_law/resources/business-law-today/2024-february/bc-tribunal-confirms-companies-remain-liable-information-provided-ai-chatbot/