A few months into building an AI product at pace, I noticed my usual PM toolkit was no longer keeping up. We had three or four dedicated engineers on the build. A PM and a Designer each split across it, both of us also pulled onto the go-to-market work that came with building a zero-to-one product: decks, demos, narrative, partner conversations. Whatever artefact product was producing for engineering (PRDs, research, documentation) was being produced at a fraction of the speed they were shipping. New features were arriving on a daily basis, each one the kind of thing that would have taken weeks or months to build before AI, and arriving faster than we could scope the UX and UI around them. The demo kept looking good, because demos always look good: you pick the inputs. The harder question, whether the product was genuinely getting better at the thing it was meant to do, had become difficult to answer with confidence. Product was a bottleneck the team was quietly routing around, and I couldn't fault them for it. The honest response wasn't to work harder at the old playbook. It was to accept that the playbook couldn't work at this ratio.
I don't think I'm alone in this. Most PMs working on AI in the last eighteen months have had a version of this moment. Nobody's saying it out loud, which is why I want to.
Here's what I think is actually happening: specification has moved from prose to executable artefacts. The PRD as a long, static document has stopped doing the core work of a spec, because the work now has to live somewhere it can be run. And that somewhere else has been hiding in plain sight. It's a golden dataset.
What the team was actually asking for
I started writing PRDs in February. By April, most hadn't been finished, and the ones that had weren't being widely read. But the thing the team was asking for in retros, repeatedly, was more focus and sharper direction on what we were prioritising next. They wanted the thing a PRD was meant to give them. They just weren't getting it from the PRD.
The demand was real. The supply was broken.
Why the PRD stopped working
Three things broke the PRD's contract for AI work, and they broke it together.
1) Engineering has become faster than product. A competent engineer with modern tooling can ship an AI feature in an afternoon that would have been a two-sprint project a year ago. The traditional cadence of discovery, PRD, scoping, sprint, review is slower than the build cycle. By the time a PRD is written, three versions have been prototyped. This isn't disrespect; it's physics.
2) PRDs were always written in the wrong register for AI work. A functional requirement describes intent: what the feature is for, what it should do, who it's for. For deterministic software that was sufficient. Engineering closed the gap between intent and behaviour, QA caught misalignment. For AI systems the same intent can be satisfied by thousands of output distributions, most of them not what you meant. No gap-closing layer rescues a document written in the old register.
3) Early-stage AI products don't have stable targets. Ours is shaped continuously by shifting conversations with stakeholders and users. A PRD assumes a fixed thing to specify against. Written specs degrade faster than AI products evolve.
What a golden dataset actually is
A golden dataset is a curated collection of real inputs paired with examples of good outputs, or clear criteria for what would make an output acceptable when outputs are harder to pin down. It's the artefact an AI system is run against every time a meaningful change is made.
On our project, one of the engineers started building ours as sets of images: examples of brand-compliant creative alongside examples that clearly weren't, organised so any change to a model, prompt, or workflow could be tested against the same reference set. That curation is itself product work. Deciding what belongs, what counts as a good output, where the edge cases live: this is where "what good looks like" now lives. It lives in the dataset, not in a document describing the dataset.
Eric Schmidt put it sharply: "The 10x advantage is no longer execution. It is defining what counts as success. A programmer writes a spec and an evaluation function, runs it at 7pm, and wakes up to what was invented overnight. The advantage now belongs to whoever can specify the problem precisely." The uncomfortable version of that quote, for a PM, is that specifying the problem precisely has always been the job. We got away with being approximate because engineering filled in the gaps. It doesn't anymore.
Three things a golden dataset does that a PRD cannot:
1) It encodes "good" in a form the system is tested against automatically. A document describes intent. A dataset is specification.
2) It gets run rather than read. Every new feature, every iteration on an existing one, every change to a prompt or workflow gets exercised against it. The spec is alive in the development loop in a way a document never was, which means whoever shapes the dataset is inside every decision automatically.
3) It improves with the product rather than decaying against it. New edge cases from production go in, stale examples come out, the artefact gets sharper over time. PRDs do the opposite.
One thing I want to be direct about: the dataset is at its best when it isn't owned exclusively by product. On our project, Engineering sets it up in the way they find most useful and Product curates the criteria and edge cases. A dataset gatekept by Product recreates the problem PRDs had.
Where this goes wrong
A piece that said "PRDs are bad, golden datasets fix everything" would be dishonest. Three failure modes worth naming:
Overfitting to the dataset. A system tuned to ace a fixed set of examples doesn't generalise; it just gets good at passing your specific test. The remedy is continuous replacement of stale examples with fresh ones pulled from real usage. Static datasets rot fast.
False confidence from small samples. Thirty examples is enough to catch obvious regressions. It is not enough to tell you the system works. Treat early datasets as directional, not definitive.
Divergence from real-world use. Your dataset reflects what you thought users would do. Production reflects what they actually do. The gap widens quickly. Pulling production samples into the dataset on a regular cadence is the only thing that closes it.
These aren't arguments against golden datasets. They're arguments against treating any artefact as self-maintaining. The PRD had its own version of this problem. Nobody updated that either. At least with a PRD it was obvious when it had gone stale. A dataset can look healthy while quietly lying to you.
The hard part
You have to articulate what "good" looks like concretely enough to put it in a spreadsheet, which means a sharper definition than most PRDs ever forced out of us. You have to tie eval scores back to commercial outcomes, whether that's task completion, support load, or revenue, or you're optimising for a metric that only speaks to itself. And you have to accept that the dataset is a living artefact that needs continuous tending, not a document you write once and forget.
That last one is where PMs earn the seat the dataset puts them at.
Where the PRD still earns its keep
The team's request for focus and direction was the correct request. The PRD wasn't the wrong instinct; it was the wrong form. There is still real value in a shared source of truth: what the product is, who it's for, why it exists, what its commercial shape looks like, what we're prioritising next. Without that, the team drifts, and no golden dataset fixes drift.
What's changed is the form. A twenty-page PRD nobody reads is worse than no document at all. A short, visual-first overview that the whole team can orient to in a minute is the thing that earns a read. A microsite, a living one-pager, a small internal hub. Something that sits above the work rather than trying to specify it.
Prose and pictures for why and for whom. Dataset for what.
Starting on Monday
Pick one AI feature you're responsible for. Write one sentence describing what a good output looks like. Find ten real inputs. For each, write the criteria that would make an output acceptable. Run your current system against it. Grade the outputs. Talk to the engineers about how they'd want to extend and maintain it.
The shift happens the first time you run the dataset against a change before it ships. That moment, a numbers conversation instead of a taste conversation, is where the power relationship over "what good looks like" moves back to something shared, measurable, and alive. Not by writing another document. By exercising a spec.
The job didn't get smaller. It got sharper.
