Tips And Tricks For Data-driven Product Managers

Q: How does a data-informed PM make decisions, balance between data & intuition & go about developing a new product where there is no historical/supporting data?

‍

What is data?

It doesn't have to come out of an analytics dashboard or a spreadsheet.

Great PMs feed off the universe of evidence. In this arena, quantifiable product metrics is just a subset.

Evidence can come from an interview, a competitive user flow, even a casual conversation.

Sure, basing a decision on an isolated anecdote can be risky but that doesn't mean it's not data.

Be evidence-driven, not just data-driven.

‍

What a smart PM will do before analysing data?

Data is RARELY pure & unadulterated. It's never served on a platter. There are 6 things a smart PM will do.

Treat bias

They are able to detect forms of biases e.g. tool bias (like GA samples data at certain thresholds), confirmation bias, availability bias etc.

Contextualize data

e.g. A 100% increase in conversion rate on a trivial base (from 1 conversion to 2) isn't necessarily a eureka moment.

Benchmark data

e.g. we got 500 transactions this quarter vs. 12% more transactions than the same quarter last year.

Know what metrics are "vanity"

e.g. Pageviews can be indicative but matter less if they don't correlate with conversions.

Reduce their scope of measure to vitals

Rather than measuring hundreds of parameters, they will setup a hierarchy of few metrics that actually move the business dial.

Realize the impact of over-optimization & law of diminishing returns

Optimizing one metric too far can lead to a failure in another KPI or doesn't scale beyond a certain point.

PMs also know that "measurement" is a means to an end.

If you can't possibly convert a "trend" to an actionable insight, then you're tracking noise.

PMs obsess first about how the data point will inform action, then think about the tooling. (not the other way around)

‍

How to align field of vision with the business?

PMs will need to align their vision for the company if they want to prosper the company's development. Here are what they can do:

• Start from the company OKR
• Identify a product strategy that supports it
• Derive roadmaps & initiatives from the strategy
• Conceive product-level & feature-level metrics

Intuition vs. data?

I don't take intuition to mean "gut feeling", rather an educated perspective developed through observation.

Intuition & evidence work like an adjustable dial.

When data is unavailable, thin, biased or skewed, you need to apply intuition to provide balance.

When it's abundant & irrefutable (e.g. a rigorous A/B test once showed me a solo form performed better than a 3-part wizard), then you dial down intuition in favor of data/evidence.

Usage metrics

Finally, when you're starting out a product, then you don't have one form of data: usage metrics. However, the idea wouldn't exist if it didn't have potential prospects.

So, generate copious data from them.

Derive evidence from customer research, focus groups, indirect competition, surveys etc. to fuel your MVP's hypothesis. Then, test & iterate.