Datasets

Datasets let you curate sets of inputs and expected outputs that you can replay against prompt or model changes to catch regressions before they reach production. Instead of hoping a prompt tweak didn't break anything, you keep a growing collection of the cases you care about – the tricky ones, the ones users complained about, the ones that broke last time – and re-run them.

Because every change is versioned, a dataset is also an audit trail. You can always see what a dataset looked like at the moment you ran a test against it.

Why use datasets?

• Catch regressions before shipping – Replay real cases against a new prompt, model, or agent version.
• Turn production failures into test cases – Add a trace to a dataset directly from AI observability the moment you spot a problem.
• Keep tests reproducible – Every dataset change creates an immutable revision, so a test run can be pinned to exactly the data it used.
• Work programmatically – Build and update datasets from agents and scripts, not just the PostHog UI.

How datasets work

Datasets have three layers:

• Dataset – A named collection of items, scoped to a project. It has a description and a free-form metadata object.
• Dataset item – A single test case with a stable ID. An item's content lives in its versions, not on the item itself.
• Item version – An immutable snapshot of one item's content. Editing an item never overwrites anything, it appends a new version.

On top of item versions sit dataset revisions: monotonically increasing snapshots of the whole dataset. Every item mutation creates a new revision, so revision 12 describes exactly which version of every item was current at that point.

A brand new dataset has no revision. Its first revision is created along with its first item.

Item fields

Both expected_output and source_output are optional, so you can save a case you know is wrong before you've decided what the right answer looks like.

Creating a dataset

You can create a dataset in the PostHog UI or with the PostHog MCP server.

Adding items

From a trace

The fastest way to build a useful dataset is to harvest real traffic. On a trace, click Add to dataset and pick a dataset. PostHog pre-fills the item with the trace's input, its output as source_output, and the trace provenance fields, so you can jump back to where the case came from.

Manually

Open a dataset and add an item directly, filling in Input, Expected output, and Metadata as JSON.

Versioning

Every item change creates an immutable version and a new dataset revision. You can view past revisions and restore an earlier version of an item without losing its history.

Archiving

Datasets and items can be archived instead of deleted. An archived dataset stays readable but rejects item mutations until you restore it. Archived items are filtered out of the default view – and out of exports – but their history is intact.

Exporting a dataset

Exports let you run a dataset through your own harness, in CI or on your machine, and keep the results comparable over time.

Click Export on a dataset to generate a JSONL file. Exports are:

• Asynchronous – The export is prepared in the background, then made available to download. Exported files expire, so download or re-export as needed.
• Pinned to a revision – Exporting from Latest pins to the current revision, and exporting while viewing a historical revision pins to that one. Because revisions are immutable, re-downloading the same export always gives you identical data.
• Active items only – Archived items are excluded.

Each line is one item, with the revision recorded on every row:

{"dataset_id":"018f...","dataset_revision":12,"item_id":"018f...","client_item_id":"ticket-4821","version":3,"input":{"messages":[{"role":"user","content":"How do I cancel?"}]},"expected_output":"Point the user to the billing settings page.","source_output":"You can cancel any time.","metadata":{"category":"billing"},"source_trace_id":"trace_abc","source_event_id":"event_def","source_timestamp":"2026-05-04T10:12:33+00:00"}

Pinning to a revision is what makes an export worth storing next to your test results: "this run scored 84% on revision 12" stays meaningful even after the dataset has moved on.

Programmatic access

MCP

The PostHog MCP server exposes datasets to AI agents like Claude Code and Cursor, so an agent that just debugged a bad generation can add it to your regression set without you switching windows.

Pass a client_item_id when an agent creates items. An identical retry returns the existing item instead of creating a duplicate, which makes agent loops safe to re-run.

API

The same operations are available over REST at /api/projects/:project_id/datasets and /api/projects/:project_id/dataset_items, including creating exports and downloading their content. Use a personal API key with the dataset scope.

CLI

CLI access is on the way, so you'll be able to pull a dataset revision and run it as part of a build without writing API glue. It isn't available yet.

Limits

PostHog applies limits to datasets to prevent abuse. These limits are subject to change.

Coming soon: offline evals reporting

Today datasets are the curation and export half of the loop. You export a revision, run it in your own harness, and read the results wherever you run them.

We're working on closing that loop. Soon you'll be able to report the results of an offline evaluation run back into PostHog, scored per dataset item and grouped by run, so you can compare a prompt or model change against the previous run and see which specific items regressed – all next to the online evaluations already scoring your production traffic.

Further reading

• Evaluations overview – online evaluations that score production generations automatically
• Trace reviews – the manual review workflow for finding failure modes worth adding to a dataset
• Traces – where dataset items usually come from