The Various Layers of Startup Feedback
After working in tech for a few years, I’d like to share one of my biggest lessons so far. It’s incredibly simple, yet something startups are constantly getting wrong.
I’ll define feedback in this post as information collected that influences what needs to be improved. This manifests itself in many ways. It could be a phone call, survey answer, a collection of analytics events, user tests, etc.
Feedback is also the primary source for understanding where you need to improve your product. It’s also a great way to improve as a manager too. If you’re a startup (or a large company), this is your lifeblood. So let’s talk about it.
But first, a tale of two founders…
Steve is the CEO of a startup. His team just launched a software product and they promoted it heavily across the interwebs. Steve scrambles to setup Mixpanel and Google Analytics to track how people are finding and using the product. Great news! Lots of people seem to be interacting with it.
Steve then realizes that a specific part of the product seems to be causing quite a bit of drop-off, so he changes a page, and sits back to wait for more feedback to come in.
Jane is the founder of a coffeeshop. She just opened her doors, and promoted it heavily across town and in newspapers. Jane scrambles to brew coffee and fulfill orders. Some customers arrive, and ask where the decaf coffee is. Jane forgot that some people are insane and drink decaf coffee. She apologizes and says she’ll have it tomorrow.
The next day, Jane brews a pot of decaf coffee and waits for those customers to arrive and give feedback on the cup of joe.
What’s the Difference?
In both scenarios, the founders received a form of feedback. The difference was the richness of the feedback. For Steve, the feedback was abstracted away and stored in the form of analytics events. For Jane, the feedback came straight from a customer (in-person vs. online).
Also notice how each business owner took action. Steve & Jane both made changes to their business, but Steve guessed on how to improve, while Jane didn’t. She knew exactly what needed to change.
Why Rich Feedback?
Many startups avoid doing basic business-building tactics, and talking to users in person (or over the phone) is one of them. A founder may say, “only one person’s feedback can steer me wrong” and they are right. That’s why you need to collect it from multiple people.
In the early stages of a company or project, you need the richest feedback you can possibly get. If I view Google Analytics, the metrics serve as guideposts, but they really can’t tell me on what to do next.
If you find yourself in a position where you’re guessing on what to do next, ask yourself if you’ve collected enough rich feedback.
The Richest Feedback Sources:
- User test in real life (not usertesting.com – you need to guide the conversation)
- Talk to people on the phone/in-person
Make sure to zoom into an individual level, where you are able to steer conversation and ask questions.
- Screen capture (Inspectlet)
- Email Feedback
In these examples, you’re still collecting feedback from an individual, but emotion is filtered out, and you can’t steer the conversation nearly as well as talking to people in real-life.
- Survey Data (aggregated)
- Metrics (aggregated)
Analytics serves a very important purpose, but it won’t tell you what you need to do next. I see metrics as a way to confirm or deny what someone might tell you in-person. If you’re making a large number of decisions exclusively via Mixpanel data you’re setting yourself up for failure.
So what does this look like in practice? Here’s how I approach this…
High Ambiguity -> Get Rich Feedback
If you’re making a large change to your startup’s business model, or pushing a brand-new feature for an age-old company, the same rules apply. If you encounter pockets of “unknowns” it’s almost a sure sign you need to collect richer feedback.
Time Constraints -> Get Rich Feedback
Many funded startups only have a finite time to figure out what they are doing, or they will go out of business. That means you don’t have a ton of time to A/B test everything under the sun. Speed is your most important weapon.
You don’t have time to wait three weeks to achieve statistical significance.
Low Risk -> Okay to Abstract
Not every change requires that you talk to five people and ask for in-person feedback. Let’s say you’re a big company, and you just tweaked a feature. Sure, it would be nice to collect feedback in-person about it, but there’s not very much at stake. Fire up analytics software, and see how it performs.
But if you see some crazy changes, it may mean you need to talk to customers about what they think.
Work up the ladder
I try to work my way up the ladder, starting with the richest feedback possible.
First, I will try to talk to people in-person, then I will see what they are doing via Inspectlet, and finally, I’ll look at metrics. I try to mentally de-risk things. If I started by digging into metrics, I could be lead to all kinds of strange conclusions about what to do next.
Make life easier on yourself by digging into rich feedback sources, and don’t make decisions off watered-down feedback.