Tweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+


With valuable experience as a Google product manager and co-founder of two companies prior to Optimizely, Pete Koomen graciously agreed to let us pick his brain on product management.

Wizeline Head of Product Matt Pasienski digs deep into the following topics with Pete:

  • The importance of defining product success with the entire team using clear goals and metrics
  • How to define success when it’s difficult to set clear goals and metrics
  • A/B testing and how it can work in conjunction with creativity and intuition
  • Hard-learned lessons in the journey from failure to success

Related Video: Keval Desai explains how Optimizely is changing B2B software buying patterns.

Watch the video to learn more about the role of A/B testing and metrics in a product or company’s success and why Pete says that “metrics-driven does not mean customer ignorant."

Create Your Free Wizeline Account


Full transcript:

Matt Pasienski: Hi, I'm Matt Pasienski. I'm here with Pete Koomen who is the CTO and co-founder of Optimizely, a AB testing and personalization platform. A optimization,  personalization and AB testing platform. Hi, I'm Matt Pasienski. I'm here with Pete Koomen, the CTO and co-founder of Optimizely a personalization and AB testing platform.

Pete, thanks for joining us today.

Pete Koomen: Thanks, Matt.

Matt Pasienski: I know you're big on defining success. I think everyone agrees that defining success before you start working on a product is essential. What do you do here to get everyone aligned along some definition of success.

Pete Koomen: It's a great question, Matt. We spend a lot of time at the beginning of a project talking about what success means because it's really ... It's one thing to have a clear notion of success in your head. It's a wholly other thing to make sure that everybody else understands what you understand and that's the really difficult part and so, for us we try our best to come up with clear metrics where possible to sort of set the end post before we really start.

Matt Pasienski: What do you do to get everyone aligned on those metrics, especially when it's complicated because it's one thing to say, "Hey, we need to drive this many more views." But when you have something complex, when you have multiple angles or when you're developing something new, those metrics that you start with might not be what you end up with as your final notion of what it means to be successful.

Pete Koomen: You're absolutely right. When you're developing a new product for example, it's a little difficult at the outset. Sometimes it's hard to pick metrics. Sometimes it's hard to pick goals for those metrics. I mean, you're starting from a blank slate. It can be difficult and so there are other ways you can tackle this problem. One way is, I think it's popularized by Amazon, we do this, we write a press release on day one. Which is, it's a really easy way to help capture in a real simple format, what is the end customer benefit here.

Matt Pasienski: Even though you're running a very metrics driven business you start out with a kind of public facing, human description of what you want to accomplish.

Pete Koomen: Yes. Metrics driven does not mean customer ignorant here. It really means that we do our best to capture the numbers when we can but our business is driven by a lot creativity and intuition as well and for us the right way to capture that when you've got a larger team working on something is to try and distill down to what is the ultimate customer benefit that we're all working on here. Press release is one way to do that. Another way to do that is to write a little script for a demo that should exist when we're all done with our work here.

Matt Pasienski: That's great. Yeah. I think we're talking about, when you're trying to measure success there's such a giant universe of different ways to get there. Obviously once you're down to the fine grain of personalizing and creating that AB test which one of these landing pages plays better, what's the right position on this button, that's where you can get a lot of gains at the end but early on it's really about telling a story and defining a vision that helps you cut through all that noise.

Pete Koomen: That's exactly right. AB testing and being numbers and metrics driven in general is not a replacement for creativity. It's not a replacement for intuition or vision. It augments it and so what we try to do at Optimizely we build products that achieve the best possible man-machine merger. We built software that does what computers do well, which is measuring things, communicating numbers, trends and patterns but that doesn't replace the need for somebody with a real clear understanding of their own business, their own customers and we think the same way about our products.

Matt Pasienski: That's what kind of struck me about ... I've watched ... You've started three companies now. You've gone through this process several times. I think you learned ... You're coming at it as an engineer but you also have a very people-centric vision of starting a company. What's it like to come from an engineering background, a mathematics background but then be confronted with the world of, "Do people like it? Do people want to engage with it? Is it something that they want to integrate into their lives?"

Pete Koomen: Yeah. I think that mindset really developed and we learned a lot of that the hard way so my co-founder Dan and I started two companies that both failed before Optimizely. One of the hard lessons we learned there was that unless we were completely and absolutely customer obsessed, what we were doing was most often wasting time. We built our first product and one of the metrics actually that I like to use to illustrate this is that with our first company, it was called Carrot Sticks. It was an education software platform. Our time from founding the company to getting a dollar in revenue was about six or seven months and our second company was called Spreadly. It was a marketing referral business. Our time from inception to getting our first dollar was about a month and a half and with Optimizely, our time from inception to first dollar was literally one day.

Matt Pasienski: That's very interesting. Obviously, if you are getting revenue on your first day, which is something that you can do. You're saying, "Hey, somebody please take a flyer on this one. Please come in and invest in us as a customer." How do you project trust though? How do you build that trust? Obviously, you can start with your friends, but as you get further along you're asking people not just to buy your product today, but you're asking them to buy it as a work flow change that they're going to have to deeply integrate into their company for years to come. How do you project trust?

Pete Koomen: Well, a lot of that has to do with spending some time and thinking about what is the vision that you want to accomplish here. Entrepreneurship is a bit of a balancing act between imagining something that doesn't exist and sort of forcefully projecting that onto the world and also being really, really, ultra realistic and trying your best to understand what the world is like now.

Posted by on Thursday, April 23, 2015.


Leave a Reply