1,000 (How we’re validating the opportunity for Lean Analytics book)

---

When Alistair and I first started promoting Lean Analytics, we set ourselves a target of adding 1,000 subscribers to our mailing list by the end of August. At this stage of promoting the book, the “Number of subscribers” is our One Metric That Matters, and one thousand is our line in the sand.

Some might consider this a vanity metric, but it’s not—for us, it’s a measurement of interest in the book and its subject matter. It’s also an indication of our ability to contact a number of people with survey questions, additional content, and more so that the book meets the expectations of our intended audience.

Sure, the number goes up continuously (unless we fail miserably and more people unsubscribe than sign-up), and it’s not a ratio or rate (which we’ve said is important for a good metric.) But ultimately it was a good measure for testing our initial MVP (which is basically the website + the content we’ve produced on the site and through the mailing list.)

So why 1,000?

Like any line in the sand, it’s a combination of guesswork, aspirations, and judgement around what is challenging to reach, but attainable at the same time. We also spoke with other authors about how well they did building up their mailing lists, and we have our own experience doing this for other businesses, so we had some rough benchmarks to compare against. It’s also an indicator of things like survey completion and pre-orders that could help us break even.

Drawing a line in the sand is really important. Without that line, you can’t tell if you’re making adequate progress at a fast enough pace to meet your goals. The line also tells you how much effort to invest in your current course and direction, whether to double down, or whether to try something else.

If we had hit 100,000 subscribers in a couple of months, we would have known we were onto something. We’d be able to convince our families that we really needed to skip important vacations in order to write; or start scheduling book tours in advance; or charging more for speaking engagements. Similarly, if we had hit only 100 subscribers, we would have known it was a failure.

Most startups fall into the murky middle—the metrics they’re tracking are neither home runs nor dismal failures. Picking a target helps define success and failure, and gives you the opportunity to be honest with yourself. Are things going well or not? Should I continue or not?

This is a big part of succeeding with Lean Analytics. Draw a line in the sand (and remember, it’s in the sand, so it’s moveable), run some experiments, measure your results, and learn from them. Focus on actionable goals that derive from the metrics that you are tracking.

So how did we do?

Unfortunately, we missed our target. By the end of August we had 906 subscribers. We were off by about 10%. Not bad, but disappointing nonetheless. In evaluating the result versus our target, Alistair and I asked a pretty important question, “Do we think there’s enough interest in Lean Analytics to make the book a success?” And if there is enough interest, why didn’t we hit our target or surpass it? The number alone isn’t enough to give us the answers; but it’s a good starting point for the discussion.

Alistair and I concluded the following:

• The qualitative feedback we’ve received since announcing the book has been very strong. A lot of it has come from our peers (which you have to discount a bit), but a lot of it has come from strangers too. That means (a) we’re reaching into new audiences we haven’t tapped into yet; and (b) there’s unbiased interest there that’s meaningful.
• Some of the attention has involved speaking engagements we’ve been invited to, which suggests that we’ll be able to spread the world through those organizations as well.
• Our survey response rate was amazing. People who signed up really cared, and took the time to give us their thoughts. Response rates of 75% are an excellent sign of engagement.
• We reviewed the 750+ survey responses we got (the survey pops up after you sign up for updates) and did a bit of lightweight quantitative analysis on people’s interest in Lean Analytics. This gave us good insight into what people care about (validating a number of our hypotheses.) It also helped us trim some things we didn’t want to do.
• We could have done more to juice our subscription numbers, but we didn’t have the time. For example, we had plans for experimenting with paid advertising, which we didn’t get to (but probably will at some point.) And we also had to spend a lot of time writing the book (which takes away from marketing the book.)
• Both of us were fairly confident we could have hit our target 1,000 subscribers had we put more effort in. At the same time, that effort was better spent on other things: writing the book (and in many ways changing what we had written and planned to write based on good qualitative feedback from existing subscribers), debating ideas, planning future marketing campaigns, etc.
• The number we did hit is pretty close to the target we set out. Had we only hit a few hundred, panic would have sunk in. But we were close enough, considering the effort we put in, to justify continuing with the project.

As you can see, we’ve got a mix of qualitative and quantitative analysis going on. There’s a bit of guesswork, gut and intellectual honesty thrown into the mix and that’s common when evaluating almost any metric, particularly at an early stage. You’re not dealing solely in fact, but without any facts at all, you’re not giving yourself the right framework for learning and adapting quickly enough.

Side note: As of this moment, we’re at 945 subscribers. We’ve added 39 subscribers in September so far, which is ~3.25 subscribers per day. In August we had 142 subscribers, which is ~4.6 subscribers per day. Our rate of subscribers is dropping. That’s not surprising, but it’s a number that we’re keeping an eye on. September will provide a decent benchmark that we can then use to compare for future months.