In the second part of our interview, Wizeline Head of Product Matt Pasienski discusses numbers, customer experience, and education with Optimizely CTO, Pete Koomen.
Pete’s product management insights include the following:
- Creating customer trust and how entrepreneurship is a delicate balance between envisioning something that doesn’t exist and being ultra realistic
- Revenue dollars as the strongest market signal that you’re building something people want
- The importance of creating a feedback loop between customers and the people actually making product decisions
- “Statistics has a usability problem” and why academia should play a part in building techniques and tools for companies using data to make product decisions
With Optimizely, Pete maintains the biggest part of their job is education — not only in creating software for “A/B testing you’ll actually use” but in educating the broader community on the importance of making data-driven decisions.
Check back for the final installment of Pete Koomen’s interview next week!
Matt Pasienski: Obviously if you were getting revenue on your first day, which is something that you can do. You're saying, "Hey somebody please take a flyer on this one, please come in and invest in us as a customer". How do you project trust though? How do you build that trust? Obviously you can start with your friends but as you get further along you're asking people not just to buy your product today but you're asking them to buy it as a work flow change that they are going to have to integrate into their company for years to come. How do you project trust?
Pete Koomen: Well a lot of that has to do with spending some time and thinking about, "What is the vision that you want to accomplish here?" Entrepreneurship is a bit of a balancing act between imagining something that doesn't exist and forcefully projecting that on to the world. Also being really, really, ultra realistic and trying your best to understand what the world is like now. That's a really hard thing to balance.
When were talking to customers early on, I don't want to say we were selling vapor ware because we were completely open about where the product was but it was a combination between saying, "This does not exist now but this is what we want to exist. Will you invest with us in it?"
Matt Pasienski: I think a real concrete example is pricing of that. When you go out and you're charging someone on day one, what would you rather accomplish, would it be to pick the price you know they would pay and then build until they'll pay it, or start small and then steadily optimize and grow your platform? I think those are two valid approaches that might be good in best in different situations. What is your opinion on that?
Pete Koomen: When we started we really weren't even thinking on that level. With Carrot 6 we made a small amount of revenue selling software subscriptions to parents and over the months a few parents would subscribe so we had money trickling in, which we were excited about.
With Wisely we found a couple paying customers on day one who were willing to commit $1,000 a month to get access to early versions of the product. Which was mind blowing for us at the time, that was an unfathomable amount of money and I don't think ought of trying to optimize that at that point really entered our mind. We were using the revenue number not so much as something that would fuel the growth of the company but more a proxy for customer interest, for the need that they had for what we were doing.
Both Dan and I are big fans of Y Combinator and Paul Graham and his mantra, "Build something people want" is exactly what we were trying to do there and their dollars were [crosstalk 00:02:34]
Matt Pasienski: When you're first starting the company, is revenue actually like the best ... you say start charging from day one, is that the best metric of our customers are happy and engaged and ready to follow us into this crazy vision?
Pete Koomen: At the risk of over simplifying I would say yes. At that stage the dollars are the strongest signal you have that you're building something people want.
Matt Pasienski: I want to drill down into something like dollars because obviously you have these giant goals that are really important to the company. We need to make revenue number, we need to grow our team, there's a lot of big level goals but then you have that middle ground between those goals which are clearly 100% important to the company and things that you can actually measure. Impulse to response, when you put in a small amount of effort somewhere, the revenue might go up, might go down. There might be a connection, there might not be. How do you start to build those things and start to build that connection between the day-to-day tasks, and the features that you're adding to your product, and the decisions you're making on a day to day level, and those giant numbers that can be moved for 20 different reasons, or 100 or 1 million?
Pete Koomen: That's a great question. There are some areas where it's pretty easy. If you're a sales person, if you're on the sales team it's relatively easy to attach your performance to a number, it just falls out. I think it's a lot harder to do for a project manager, for example they may be trying to optimize for many things at once, and many of those things may only be measurable well after the fact. It's tough.
One thing that were big fans of here at Optimizely is the idea of NPS, net promoter system. We've built something internally called Optimizely Promoter System, which is this idea that through a very very simple question you ask your customers, you can put a number on the quality of experience you're giving them. That gives product managers, it gives engineers, and designers something concrete to grasp when they're thinking about the customer experience. If that goes up it means we're doing well. If we follow up with customers that are giving us poor scores and understand what really went wrong, that's a direct feedback loop with customers.
I think one of the big challenges as you're scaling a company is keeping that feedback loop with customers with the people who are actually making decisions. That's one way to do it, that's one way we had some success with.
Matt Pasienski: I want to talk about the role of data science, and it's a very big topic. Obviously the last few years in technology and Obama just appointed a Chief Data Scientist for the United States, DJ Patel. Where do you think the role of academic science, and you come from a math background, plays in forming a company like Optimizely. Obviously you've seen a lot of deep mathematics to produce results for the customer. How much of that comes from academics or just having really smart people reexamining the problem in light of what the customers need today?
Pete Koomen: That's a great question. We build products that help our customers make the best possible decisions about the experience they give their customers. That's hard to do without solid tools for interpreting data. Something that we've been saying now for a while is that statistics has a usability problem. The statistical techniques most of use to make decisions were designed roughly 100 years ago for statistician. There are a lot of reasons that they don't hold up well in a world where you can always get more data and in a world in which people who have no training have access to this data and are making decisions based on it.
It's my view that the academic world actually needs to make some changes and build tools for people who don't necessarily have training but are making decisions based on data.
Matt Pasienski: That's one of the things I noticed, we're huge fans in our head of marketing Adam, who you saw a little earlier. He uses Optimizely everyday. He always come back with these big wins like, "Look we just increased our click through 30% by changing this button" its fantastic but there's lots of really clear graphs. I feel like you started with education, you're educating people on what it means to be statistically rigorous. Through visualizations of your confidence band and through things like helping them design tests. Where do you see yourself as an educator and a community builder? Obviously you need a lot of people really focused on statistics in order to be successful, how do you view yourself as building the community around that?
Pete Koomen: I see education as one of the big parts of our job and it has been for the past 5 years. We built a product that made A B testing easy enough that anybody could do it. A B testing you'll actually use is what we said. What that meant was building software that was usable but it also meant educating the broader marketing community about the importance of A B testing , the importance of making data driven decisions. That was something we grew up with, so to speak at Google. That wasn't commonly done outside of some of the bigger software houses. That I think has hanged and a lot of that is because of our efforts.