The “Market for Lemons” Problem in Data Platforms

Organisations often measure the success of their data platforms by volume.

How many datasets have we onboarded? How many tables are available? How many data assets can users access?

The assumption is simple: more data means more value.

I think that assumption is often wrong and, worse, that acting on it can actively erode value.

This is my central claim: past a certain point, adding technically accessible but poorly understood data to a platform doesn’t just fail to add value. It can make the platform less trustworthy overall, including its high-quality data.

Economist George Akerlof explored a similar problem in his famous “market for lemons” theory. In a market where buyers cannot reliably distinguish a good used car from a bad one, known as a “lemon”, uncertainty affects the value of every car in the market.

It’s worth being precise about the analogy, though. In Akerlof’s market, the information asymmetry is adversarial: sellers know more about the quality of their cars than buyers and may have an incentive not to reveal what they know. In data platforms, the asymmetry is rarely adversarial. More often, it’s the residue of organisational memory loss. Nobody is deliberately hiding the fact that a data asset is no longer maintained. The person who understood its history has simply moved on, the documentation explaining its limitations was never written, and ownership has changed hands more than once.

The mechanism is different, but the resulting problem is similar: users cannot reliably assess which assets are trustworthy and fit for purpose. When the cost of making that assessment becomes too high, uncertainty begins to affect the platform’s value.

The data asset quality problem

Imagine a data platform containing thousands, perhaps hundreds of thousands, of data assets.

Some are highly valuable. They contain current, well-understood data, have clear provenance and ownership, and answer important user needs. Others may be technically accessible but poorly documented: static snapshots of historical programmes, assets with unclear ownership, or dependencies understood only by a small number of people.

From a technical perspective, all of these may appear equally “available”.

When users cannot easily distinguish between trusted, maintained data products and assets whose quality or purpose is uncertain, they have to perform that assessment themselves, working out what a table actually contains, whether it’s still current, and who, if anyone, owns it.

The cost of answering those questions becomes part of the cost of using the data. When every new asset introduces another question mark, the cumulative cost can become significant.

Availability is not the same as value

This is where data platforms can fall into a quantity trap.

Moving a data asset to a new platform, aligning its schema or making it technically queryable does not automatically create a valuable data product.

A useful data asset needs context.

At a minimum, users need to understand its purpose, provenance, population, coverage, limitations and relationship with other assets. Ideally, there is also clear ownership and an expectation of how the asset will be maintained.

Without this context, organisations risk creating increasingly large catalogues of technically available but difficult-to-evaluate data.

The consequences go beyond inconvenience. Users spend time independently validating assets that could have been documented once, different teams reach different conclusions about the same data, and analysts create their own copies or workarounds because they don’t trust what already exists. Valuable assets become harder to discover among hundreds of uncertain ones.

As uncertainty grows, so does the cost of finding and validating the right data. That’s the point at which more data can begin to create less value.

The hidden cost of undocumented data

The problem becomes particularly visible during organisational handovers, when a team inherits a large portfolio of data assets without sufficient documentation, lineage or knowledge transfer.

At that point, the receiving team faces a difficult choice. It can treat every asset as equally important and attempt to maintain, migrate or modernise everything, or it can first ask a more fundamental question: which of these assets still create meaningful value for users?

This distinction matters because maintaining data products has a cost. Every asset may require engineering effort, quality assurance, documentation, governance, domain expertise, and ongoing service support.

There’s also an opportunity cost. When resources are limited, spending time maintaining every historical asset means less time for the data users actually need.

A decision to maintain everything is therefore not neutral. It is also a decision about where the organisation will not invest.

Curation is a product decision

This is why I increasingly see data curation as a core part of data product management.

A good data platform should not simply maximise the number of assets it exposes. It should help users find, understand and trust the right ones.

That requires making deliberate decisions about which assets deserve investment, based on factors such as:

• user and research value;
• strategic relevance;
• data quality and completeness;
• currency and future coverage;
• uniqueness versus duplication;
• availability of appropriate domain expertise;
• and the cost of sustainably maintaining the asset.

Some assets will justify significant investment. Others may remain available as historical or legacy datasets, clearly labelled as such. And some may no longer justify continued investment at all.

That isn’t a failure of a data platform. It’s portfolio management.

The alternative, treating every asset as equally valuable simply because it exists, is also a product decision. It’s just an implicit one.

From data assets to trusted data products

The goal should be to create enough transparency that users can understand the difference.

A smaller portfolio of well-understood, documented and actively maintained data products may create significantly more value than a vast catalogue whose quality and purpose users must discover for themselves.

In data, as in Akerlof’s market, information asymmetry matters.

If users cannot distinguish assets they can confidently use from those whose fitness for purpose is uncertain, the problem is not simply that some data assets are less useful. The uncertainty itself creates a cost.

Over time, that cost can reduce adoption, duplicate effort and erode trust in the entire data ecosystem.

Perhaps, then, the most important question for data organisations isn’t:

“How much data have we made available?”

but:

“How much of the data we make available can our users confidently understand, trust and use?”

Because, in my experience, there is a point at which the question changes from “Which asset should I use?” to “Can I trust this platform at all?”