So, you’re building a data product.

This could be a personalization engine, a business dashboard or a clustering algorithm (among some real-life examples). Something that deals with data and/or numbers. Big or small, doesn’t really matter.

You’ve made your initial assumptions, talked to users and validated your problem. You have an idea for a solution and need to validate it. Your next step is to build a prototype, show it to users and see if they’re willing to pay for it.

The catch with data product: if you want really actionable feedback from your potential users, you need to use their own data. You can get some bits of feedback from a fake/generic data prototype but most likely you will fall into one these two traps:

False positives. Your users seem to like what they’re seeing. “Oh yeah, that’s really cool, can we have it?”. You go on and spend months building the actual product. After weeks of work, you ship the actual product to the users. They like the wireframes, they’ll surely like it. Instead, you start getting feedback like “actually, I’m only interested in half of this data, can we filter by X?”. Or worse “of course there is always going to be a customer with 100% open rate, what a stupid metric”. Or “that’s some nice trivia but what do I do with it?”. Oops, you just wasted some valuable engineering time on an un-validated solution.

False negatives. Your users are indifferent to what you’re showing them. You might as well be showing them a chart of your weight compared to the GDP of Mauritania. “Yeah, nice chart. What else do you have?”. Doesn’t seem promising, you move on. And waste a possibly great product opportunity. I’ve seen users previously indifferent to a generic analysis completely light up once the number on the screen were theirs.

In both cases, there is something that clicks only once the user sees the prototype with their own data. After all, the value of a data product is mostly in the data (doh!). If you’re validating only the way the data will look or how the user will interact with it, you’re missing the main point. Only once they see the results with their own data will users start giving you useful feedback.

It will obviously require you some extra effort (development-wise + working around the Catch-22 of convincing the user to give you their data in the first place), but if you’re testing a data product with users, go feed your prototype with some real numbers.

Update: Marty Cagan, author of the great book Inspired, wrote about something the same subject in his article Flavors of prototype.