Analytics Lessons Learned: Free e-book with 13 case studies

The book is coming soon! It will be available in March from a variety of places.

You can pre-order it on Amazon so you get it as soon as it’s ready. We hope you’ll do that!

Analytics Lessons LearnedIn the meantime, we wanted to share some great stories from the book (and some that aren’t in the book), so we’ve put together an e-book called Analytics Lessons Learned.

The e-book is free! Just sign-up to get it below.

The e-book includes case studies from a variety of companies including Airbnb, SEOmoz, Backupify, Sincerely, Swiffer and EMI. The companies are at various stages — some are still building an MVP, others are large, multi-billion dollar corporations. And we even have a church in there! Yup, Kingsway Church, thanks to the amazing work of Dr. Ernie Prabhakar, is converting and applying Lean principles and Lean Analytics to how they operate. We were first connected to Ernie through his blog where he talks about how churches need to get more practical and analytical to succeed.

The book, Lean Analytics, has over 30 case studies, so this is just a taste of what we’ve been working on. But we’re confident you’ll find a lot of value in these stories, and look forward to your feedback.

So with that, please check out Analytics Lessons Learned by signing up below. Thank you!



* Signing up adds you to our email newsletter list. We’ll send you the e-book immediately, and only send occasional emails after that.

Lean Analytics workshop at the Lean Startup Conference

Yesterday, Alistair and I did our first workshop at the Lean Startup Conference. The event itself was fantastic. I thought the content from the speakers was incredible.

The workshop was a fairly deep dive into the content of the book. The process of converting the book to slides started with Alistair making nearly 400 slides. We condensed that down to 166 slides, which is still a lot of content, but we felt it was manageable. We didn’t get through all of it, but that’s OK. The interaction with participants was great. We got some amazing, important questions and lots of insightful and meaningful feedback. So thank you to everyone that attended.

We’ve uploaded the presentation to Slideshare and included it below. Hopefully it helps those that attended with remembering what we talked about, and for those of you that weren’t there it’s going to give you a great sneak peek into the book.

As well, it’s now possible to pre-order the book from Amazon: Get Lean Analytics now!

When you pre-order the book it will be delivered to you as soon as the book is released (which should be in March/April 2013.) Thank you!

An Introduction to Lean Analytics Webinar

Recently, I did a webinar through O’Reilly on Lean Analytics. It was an introductory guide, covering a lot of what we’ll be including in the book. I got some great questions as well, which validated some of what Alistair and I have been thinking about content-wise for the book; where we need to include more information and address specific concerns people have. For example, I got asked a few times about how Lean Startup and Lean Analytics applies to enterprise startups and companies. I also got asked how to know if you’ve hit product-market fit.

The presentation is now on Slideshare and embedded below:

As well, you can get a recording of the event here: http://w.on24.com/r.htm?e=526656&s=1&k=EA3D95B4759017B2C9F02235B18CEBE0 (requires registration) The webinar follows the basic structure of the book (although at a much higher level.)

Incidentally, you can get the Early Release here: http://shop.oreilly.com/product/0636920026334.do.

The Early Release does get updated as we write more content and submit it to O’Reilly. Right now it’s quite short, but you can expect an update that has ~60% of the book (unedited) soon. So if you get the Early Release, you will get updates and more content and you’ll have a pretty good idea of what the final version will look like. Having said that, Alistair and I are still writing tons, editing and iterating.

Get an Early Release of Lean Analytics Book!

lean analytics early release book coverLast week we released a sneak peek of Lean Analytics, sharing a few chapters.

Today you can buy an Early Release copy of the book!

Buy it now >>>

The Early Release copy in digital format means you’ll get updated versions as we write the book. The content will be fairly raw and unedited, but you’ll get access to it way before the final release. As we write content and post it into the Early Release, you get it.

Ebooks from oreilly.com are DRM-free. You get free lifetime access, multiple file formats, free updates. And now you get Dropbox syncing too.

You can also pre-order the final copy in paper format.

This is a big step for us as authors–we can now see if the early interest from people translates into early purchases. And we can push the digital copy on a constant basis to people and get ongoing feedback.

Take a look at the Early Release and … feel free to buy *smile*

Get a Sneak Peek at Lean Analytics Book

To-date, Alistair and I have written a total of 43,964 words for the book. We’ve actually written a lot more, but we’ve cut a bunch of stuff, and put a bunch of stuff on the blog too. But the book right now stands at that word count. It’s a decent number of words. We’re about 50% of the way there –at least with an overall draft– and there’s going to be a lot of editing and refining along the way.

We’ve always wanted to share with people, as much as we can, throughout the writing process. With that in mind, we’re releasing a draft of our first few chapters for everyone to read (and hopefully enjoy!) Please note: this is largely unedited. Alistair and I have edited the work but it hasn’t gone through any serious review process. And you can expect considerable change when the book is finally launched, but we want your feedback right away. We’re eager to hear from you on what you think of the approach, the tone, the ideas and the overall content.

All you have to do is click the link below:

Download the sneak peek of Lean Analytics book (pdf)

If you haven’t signed up for updates from us, we hope you’ll do that. Just drop your email address into the sidebar. But it’s not required to check out the sneak peek.

All feedback can go through our site: http://leananalyticsbook.com/contact-us/.

Please share this with your friends and colleagues. You are free to distribute the sneak peek to anyone you’d like; the more distribution the better. Sharing this blog post with others is greatly appreciated as well … now we’re ready to start ramping up our marketing efforts. And one last thing, pre-orders of the book will be available soon.

Thank you!

1,000 (How we’re validating the opportunity for Lean Analytics book)

When Alistair and I first started promoting Lean Analytics, we set ourselves a target of adding 1,000 subscribers to our mailing list by the end of August. At this stage of promoting the book, the “Number of subscribers” is our One Metric That Matters, and one thousand is our line in the sand.

Some might consider this a vanity metric, but it’s not—for us, it’s a measurement of interest in the book and its subject matter. It’s also an indication of our ability to contact a number of people with survey questions, additional content, and more so that the book meets the expectations of our intended audience.

Sure, the number goes up continuously (unless we fail miserably and more people unsubscribe than sign-up), and it’s not a ratio or rate (which we’ve said is important for a good metric.) But ultimately it was a good measure for testing our initial MVP (which is basically the website + the content we’ve produced on the site and through the mailing list.)

So why 1,000?

Like any line in the sand, it’s a combination of guesswork, aspirations, and judgement around what is challenging to reach, but attainable at the same time. We also spoke with other authors about how well they did building up their mailing lists, and we have our own experience doing this for other businesses, so we had some rough benchmarks to compare against. It’s also an indicator of things like survey completion and pre-orders that could help us break even.

Drawing a line in the sand is really important. Without that line, you can’t tell if you’re making adequate progress at a fast enough pace to meet your goals. The line also tells you how much effort to invest in your current course and direction, whether to double down, or whether to try something else.

If we had hit 100,000 subscribers in a couple of months, we would have known we were onto something. We’d be able to convince our families that we really needed to skip important vacations in order to write; or start scheduling book tours in advance; or charging more for speaking engagements. Similarly, if we had hit only 100 subscribers, we would have known it was a failure.

Most startups fall into the murky middle—the metrics they’re tracking are neither home runs nor dismal failures. Picking a target helps define success and failure, and gives you the opportunity to be honest with yourself. Are things going well or not? Should I continue or not?

This is a big part of succeeding with Lean Analytics. Draw a line in the sand (and remember, it’s in the sand, so it’s moveable), run some experiments, measure your results, and learn from them. Focus on actionable goals that derive from the metrics that you are tracking.

So how did we do?

Unfortunately, we missed our target. By the end of August we had 906 subscribers. We were off by about 10%. Not bad, but disappointing nonetheless. In evaluating the result versus our target, Alistair and I asked a pretty important question, “Do we think there’s enough interest in Lean Analytics to make the book a success?” And if there is enough interest, why didn’t we hit our target or surpass it? The number alone isn’t enough to give us the answers; but it’s a good starting point for the discussion.

Alistair and I concluded the following:

  • The qualitative feedback we’ve received since announcing the book has been very strong. A lot of it has come from our peers (which you have to discount a bit), but a lot of it has come from strangers too. That means (a) we’re reaching into new audiences we haven’t tapped into yet; and (b) there’s unbiased interest there that’s meaningful.
  • Some of the attention has involved speaking engagements we’ve been invited to, which suggests that we’ll be able to spread the world through those organizations as well.
  • Our survey response rate was amazing. People who signed up really cared, and took the time to give us their thoughts. Response rates of 75% are an excellent sign of engagement.
  • We reviewed the 750+ survey responses we got (the survey pops up after you sign up for updates) and did a bit of lightweight quantitative analysis on people’s interest in Lean Analytics. This gave us good insight into what people care about (validating a number of our hypotheses.) It also helped us trim some things we didn’t want to do.
  • We could have done more to juice our subscription numbers, but we didn’t have the time. For example, we had plans for experimenting with paid advertising, which we didn’t get to (but probably will at some point.) And we also had to spend a lot of time writing the book (which takes away from marketing the book.)
  • Both of us were fairly confident we could have hit our target 1,000 subscribers had we put more effort in. At the same time, that effort was better spent on other things: writing the book (and in many ways changing what we had written and planned to write based on good qualitative feedback from existing subscribers), debating ideas, planning future marketing campaigns, etc.
  • The number we did hit is pretty close to the target we set out. Had we only hit a few hundred, panic would have sunk in. But we were close enough, considering the effort we put in, to justify continuing with the project.

As you can see, we’ve got a mix of qualitative and quantitative analysis going on. There’s a bit of guesswork, gut and intellectual honesty thrown into the mix and that’s common when evaluating almost any metric, particularly at an early stage. You’re not dealing solely in fact, but without any facts at all, you’re not giving yourself the right framework for learning and adapting quickly enough.

Side note: As of this moment, we’re at 945 subscribers. We’ve added 39 subscribers in September so far, which is ~3.25 subscribers per day. In August we had 142 subscribers, which is ~4.6 subscribers per day. Our rate of subscribers is dropping. That’s not surprising, but it’s a number that we’re keeping an eye on. September will provide a decent benchmark that we can then use to compare for future months.

How To Score Problem Interviews During the Lean Startup Process

If you’re a fan of Lean Startup, you know about Problem Interviews. They’re the part of a startup where you first go out and speak to people you think might be customers, and try to determine if they have the problem you want to solve.

This post isn’t about the interviews themselves—there’s a ton of good thinking on how to conduct them already out there, and for many entrepreneurs they’re a rite of passage where you realize that your worldview is radically different from the market reality, and (hopefully) adjust accordingly.

But I do want to float a somewhat controversial idea, and get some feedback.

Yes. This is where YOU participate.

I want to talk about scoring interviews. It’s something we’re working on for the book; and it’s surprisingly thorny and controversial. Consider this a sneak peek, but also an opportunity to help us run an experiment. Let’s start with the idea first.

How to score interviews

http://www.flickr.com/photos/sven_kindler/3987690653/Problem Interviews are designed to collect qualitative data. They’re meant to indicate strongly (or not) that the problem(s) you’re looking to solve are worth pursuing. They’re hard to do well, and take lots of practice and discipline to master. If you do it right, you’re left with a ton of insight into your customers’ needs and thoughts.

Unfortunately, those reams and reams of notes are messy. Interpreting and sharing qualitative data is hard, and often subjective.

So we want to try and score them. Scoring interviews is designed to help you quantify your results, without getting overly scientific.

The challenge here is that you can’t beat a forest of qualitative data into a carefully manicured lawn of  quantitative data. We’re not even going to try that. And we’re also not proposing that you go overboard with this method: if you’re not good at collecting and interpreting qualitative data, it’s going to be difficult to get very far at all (through the Lean process or through any startup.) But our hope is that this method helps coalesce things a bit more, giving you some clarity when analyzing the results of your efforts.

During the Problem Interviews, there are a few critical pieces of information that you should be collecting. I’ll go through those below and show you how to score them.

1. Did the interviewee successfully rank the problems you presented?

Yes 10 points
Sort of 5 points
No 0 points

During a Problem Interview you should be presenting multiple problems to the interviewee—let’s say 3 for the purposes of this post—and asking them to rank those problems in order of severity.

  • If they did so with a strong interest in the problems (irrespective of the ranking) that’s a good sign. Score 10 points.
  • If they couldn’t decide which problem was really painful, but they were still really interested in the problems, that’s OK but you’d rather see more definitive clarity. Score 5 points.
  • If they struggled with this, or they spent more time talking about other problems they have, that’s a bad sign. Score 0 points.

It’s important to note that during the interview process, you’re very likely to discover different problems that interest interviewees. That’s the whole point of doing these interviews, after all. That will mean a poor score (for the problem you thought you were going to solve), but not a poor interview. You may end up discovering a problem worth solving that you’d never thought about, so stay open-minded throughout the process.

2. Is the interviewee actively trying to solve the problems, or have they done so in the past?

Yes 10 points
Sort of 5 points
No 0 points

The more effort the interviewee has put into trying to solve the problems you’re discussing, the better.

  • If they’re trying to solve the problem with Excel and fax machines, you may have just hit on the Holy Grail. Score 10 points.
  • If they spend a bit of time fixing the problem, but just consider it the price of doing their job, they’re not trying to fix it. Score 5 points.
  • If they don’t really spend time tackling the problem, and are okay with the status quo, it’s not a big problem. Score 0 points.

3. Was the interviewee engaged and focused throughout the interview?

Yes 8 points
Sort of 4 points
No 0 points

Ideally your interviewees were completely engaged in the process; listening, talking (being animated is a good thing), leaning forward, and so on. After enough interviews you’ll know the difference between someone that’s focused and engaged, and someone that is not.

  • If they were hanging on your every word, finishing your sentences, and ignoring their smartphone, score 8 points.
  • If they were interested, but showed distraction or didn’t contribute comments unless you actively solicited them, score 4 points.
  • If they tuned out, looked at their phone, cut the meeting short, or generally seemed entirely detached—like they were doing you a favor by meeting with you—score 0 points.

4. Did the interviewee refer others to you for interviews?

Yes, without being asked 4 points
Yes, when you asked them to 2 points
No 0 points

At the end of every interview, you should be asking all of your subjects for others you should talk with. They have contacts within their market, and can give you more data points and potential customers. There’s a good chance the people they recommend are similar in demographics and share the same problems.

Perhaps more importantly at this stage, you want to see if they’re willing to help out further by referring people in their network.  This is a clear indicator that they don’t feel sheepish about introducing you, and that they think you’ll make them look smarter. If they found you annoying, they likely won’t suggest others you might speak with.

  • If they actively suggested people you should talk to without being asked, score 4 points.
  • If they suggested others at the end, in response to your question, score 2 points.
  • If they couldn’t recommend people you should speak with, score 0 points (and ask yourself some hard questions about whether you can reach the market at scale.)

5. Did the interviewee offer to pay you immediately for the solution?

Yes, without being asked 4 points
Yes, when asked 2 points
No 0 points

Although having someone ask to pay or throw money at you is more likely during the Solution Interviews (when you’re actually walking through the solution with people), this is still a good “gut check” moment. And certainly it’s a bonus if people are reaching for their wallets.

  • If they offered to pay you for the product without being asked, and named a price, score 4 points.
  • If they offered to pay you for the product, score 2 points.
  • If they didn’t offer to buy and use it, score 0 points.

Calculating the scores

http://www.flickr.com/photos/jekert/2282064498/A score of 25 or higher is a good score. Anything under is not. Try scoring all the interviews, and see how many have a good score. This is a decent indication of whether you’re onto something or not with the problems you want to solve. Then ask yourself what makes the good score interviews different from the bad score ones. Maybe you’ve identified a market segment; maybe you have better results when you dress well; maybe you shouldn’t do interviews in a coffee shop. Everything is an experiment you can learn from.

You can also sum up the rankings for the problems that you presented. If you presented three problems, which one had the most first place rankings? That’s where you’ll want to dig in further and start proposing solutions (during Solution Interviews.)

The best-case scenario is very high interview scores within a subsection of interviewees where those interviewees all had the same (or very similar) rankings of the problems. That should give you more confidence that you’ve found the right problem and the right market.

The One Metric That Matters

We’ve talked about the One Metric That Matters before and it’s important to think about it even at this early stage in the Lean Startup process. The OMTM at this point is pain—specifically, the pain your interviewees feel related to the problems you’ve presented. It’s largely qualitative, but scoring interviews may put things into perspective in a more analytical way, allowing you to step back and not get lost in or fooled by all the interviews.

So are you ready to help us?

Here’s the thing: we’d would love to speak with people that are currently in the middle of doing Problem Interviews, and have them try out our scoring methodology. We need feedback here to iterate and improve the concept for the book.

So if you’d like to help please contact us or reply in the comment thread below.

Doing a Workshop at Lean Startup Conference, December 4th in San Francisco

Lean Startup Conference

Alistair and I are very excited to be doing a workshop at this year’s Lean Startup Conference in San Francisco.

The conference itself is on December 3rd. Workshops are on the 4th. There are a bunch of events prior to the conference starting as well. You can check out the site for more details. There are a bunch of other interesting workshops too. Each one is a half-day (I believe), so you’ll have to pick which ones you go to. Pick ours. Just sayin’.

Our workshop will be about using analytics within a Lean Startup to get to product-market fit before the money runs out. We’ll cover many of the highlights from our upcoming book on Lean Analytics. It will be great to put our ideas further to the test with a group of Lean Startup practitioners. You can see some of those ideas in presentation-style here: the One Metric That Matters, which is a high-level primer on Lean Analytics.

Hope to see you there!

Lean Analytics and the One Metric That Matters: A Presentation for Acceleprise

Alistair and I are starting to take our proverbial show on the road. We’re “getting out of the building” to share some of our material, test it, and collect feedback. Build ➔ Measure ➔ Learn applies to books, too, and to presentations.

Yesterday I did a presentation for Acceleprise, a Washington, DC-based accelerator that’s focused on B2B startups. The presentation is embedded below.

There are a couple useful references/resources in the presentation:

You will start to see the evolution of the material we’re working on for the book: how Lean Analytics fits into Lean Startup, the One Metric That Matters, how to find the right metrics, how to balance gut and analytics, and so on.

Parts of the presentation are very visual, so some of the meaning may be lost, but hopefully it’s helpful and interesting enough that you send us feedback, ask questions, and (if you haven’t already!) sign up for future updates on the book.

Share your Lean Analytics story with us!

We want to make sure our book is chock full of case studies and stories from people that are using analytics and Lean Startup on a daily basis. We want to provide real numbers –where possible– so you can benchmark yourself and your startup or project against others. The more people share with us, the more we can as a group help each other and improve practices around Lean Startup and Lean Analytics.

With that being said, if you have a story, case study, anecdote or number you want to share, please do so!

Here’s our brief survey: http://bit.ly/Mo2ZNh

It should take ~10 minutes to complete. In return, we’ll buy you a beer (or another drink of your choice.) That’s assuming of course that we get to your city or meet you in-person somewhere (we’re not sending beers through the mail or sending out “free beer” coupons … sorry!)

A few things to remember:

  • We’re happy to keep things anonymous, so just tell us you want it to be that way in the survey.
  • The more specific you are, the better. If you can share real numbers with us – KPIs, benchmarks, targets, etc. – that’s fantastic. Again, if you don’t want us to share your name or company name publicly, we can hide that.
  • We may publish your stuff in our book. And if you don’t want it to be anonymous, we’ll thank you several times over publicly. Fame and fortune … here we come!
  • You’re genuinely going to help a lot of people. That’s a big deal.
Share Your Lean Analytics Story >>