Seven onboarding mistakes you don’t want to make

Bayram Annakov Recently, Bayram Annakov of Appintheair wrote to us about how he’d used analytics and Lean approaches to improve his user onboarding, with some pretty dramatic results. He was kind enough to outline them here for all of us.

We build great apps, we solve critical problems, and we help our users achieve their goals. But you know what the real problem is? Despite all that, sometimes users simply don’t use our product—because we failed to get them on board. All that hard work goes to waste.

Here’s my hit-list of top mistakes that app developers make in onboarding:

Requiring that users register too early

Don’t force the user to register. Delay until it is absolutely necessary, or you risk losing them entirely. Allow them to skip registration, but ask for it when user takes some action. For example,  iTunes invites you to register when you click “Buy.”


And when you do ask them, be sure to explain the benefits of registration, such as backing up their data or syncing between their devices.

Explaining the obvious

Get out of the user’s way. Literally. Don’t force them to watch ten slides just to explain how to use a calculator! Instead, show the interface and make it intuitive.

Grabbing for push notification or address book permission

I know getting permission to contact users is very important for retention. I know you want to spam notify users to make them come back to your app. But hey—if you really want the user to give you their permission, don’t ask them on the app’s first screen. Explain why you need it using a custom dialogue (iOS does not allow customizing permission text—that’s why you need to implement your own screen).

Or better yet, make the user perform an action that signals her desire to give you permission. Banking apps are a great example of this: You really want to get notified when any transaction is done with your credit card, so they likely have a very high notification permission acceptance rate at this stage.

This is what we do in App in the Air: we ask for push notification permission only after a user demonstrates his/her desire to receive flight status alerts.

Look how Snapchat prepares the user for address book permission. First, they explain why they need access:


Then, with this context, they hand things over to iOS to get the authorization:Bayram-snapchat-2

Displaying all instructions at once

Some apps have a bad habit of displaying all their instructions and hints on a single, overwhelming screen. It’s far too much information, so the user remembers none of it—and gets the impression that the app is complex or hard to use. Instead, reveal yourself as they use the app. Teach one thing at a time and let the user learn by doing.


Above is an example of how one feature-rich task management application achieves this.

Not giving them an option to skip

Users sometimes re-install the app, whether they’re purchasing a new gold iPhone 6, or just recovering from a backup. Whatever the case, you want to avoid annoying an expert user with your hundred-page tutorial. Let them skip. Save them time and they’ll thank you.

Displaying empty screens

Please don’t ever display empty screens to the user. Josh Elman, former product lead for growth at Twitter, calls this the the “totally awesome blank screen of death.” Provide instructions, test thoroughly with edge cases, and make sure you avoid the kinds of empty screens that alienate and frustrate users.

Not measuring onboarding

This is perhaps the most important step, and the one most closely tied to Lean Analytics. It’s not enough to measure clickthroughs, or calls to action, or downloads. Your job isn’t done until your user has reached a point in their engagement process where they’re using the application as you intended.

Measure the number of users who successfully pass onboarding. Investigate why users drop, and tirelessly optimize the experience as much as possible. Remember, if your user can’t make it through onboarding, she won’t understand the power and functionality of your application. She definitely won’t use it, and you’ll miss the key leverage in growing your app to millions of users.

Our experience

For AppInTheAir, fixing these mistakes helped us move the bottom of our funnel—onboarding conversion rate—from 50%  to roughly 95%.

Josh Elman agrees—he thinks Twitter’s new onboarding process, which he covers in this video, is the secret behind Twitter’s growth from 10M users to 100M+ daily users: They taught users how to use Twitter without annoying or alienating them along the way.

Here’s some extra homework: check out this great compilation to see how popular web and mobile apps handle their sign-up experiences. Snapchat, in particular, works hard to make signing up not just smooth, but fun.

Lean Lunch with Move The Needle

Last year, Ben and I presented a workshop at the International Startup Festival with Brant Cooper and Patrick Vlaskovitz, the co-authors of The Lean Entrepreneur. It was a highlight of the festival for us, and we realized the four of us have a lot in common—and a lot to learn from one another.

Fast-forward a year, and a couple of weeks ago, I got on a Google Hangout with the team from Move The Needle. The brainchild of Brant and Aaron Eden, MTN helps companies implement Lean strategies and customer development. Since I’ve spent much of the last year talking with large organizations about innovation myself, it felt like it was time to catch up with them about the past few months.

Here’s the video of the event.

Quantitative Interviews: the importance of scoring customer feedback

thfs1cI1_400x400.jpeg (JPEG Image, 400 × 400 pixels) 2014-05-14 14-32-06 2014-05-14 14-32-16Back in December, Roger Huffstetler of Zillabyte contacted us to say he’d applied some of Lean Analytics to his startup, and wanted to fill us in on what happened. At the time, by his own admission, he was “Up to my ass in alligators” But now, a few months later, is his story.

I had just completed my 30th demo of our product, and I remember leaving the feedback session on cloud nine. As I wrapped up showing our API to the potential customer, he suggested what we were building was “amazing.” Yes, I thought giddily, this is going to work.

It took me a few minutes to come down from the clouds and realize as a founder, I’d been here before…and while it felt good, it could also be—unfortunately—an indication of nothing.

This phenomenon of euphoria is well documented: business founders see what they want to see, or as Ben & Alistair of Lean Analytics might say, “small lies are essential to company founders.” These ‘lies’ are what keep you going and believing through the toughest times.

The difficulty, of course, is in facing reality, discerning hard truth from praise and fluff. In particular, when you’re in the in the midst of customer feedback interviews—hearing that your product is “great,” “helpful,” or “amazing”—identifying reality can be incredibly difficult. Qualitative feedback, while often complementary, can be especially confusing.

It was at this point—post interview, but realizing I’d been through this cycle before— when I stumbled upon the Pain Index part of Lean Analytics (Page 170). With Ben’s & Alistair’s scoring system, you’re turning customer feedback into a quantitative process. You begin with a key customer development question: how do I know if the problem (we’re trying to solve) is really painful enough? Think of this method as the referee on the interview field, someone from the sideline to keep you honest. [We wrote this up in a blog post in 2012—AC&BY]

After Ben & Alistair introduce this question, they provide you with a set of six additional questions and a suggested scoring framework, as follows:

  1. Did the interviewees successfully rank the problems you presented? [Yes (6), Sort of (5), or No (0)]
  2. Is the interviewee actively trying to solve the problems, or has he done so in the past? [Yes (10), Sort of (5), or No (0)]
  3. Was the interviewee engaged and focused throughout the interview? [Yes (8), Sort of (4), or No (0)]
  4. Did the interviewee agree to a follow-up meeting/interview? [Yes (8), Yes, when asked (4), or No (0)]
  5. Did the interviewee offer to refer others to you for interviews? [Yes (4), Yes, when asked (2), or No (0)]
  6. Did the interviewee offer to pay you immediately for the solution? [Yes (3), Yes, when asked (1), or No (0)]

On their system, a score of 31 or higher is a good score. Anything below that, you haven’t really nailed a painful problem. But here’s where the real insight comes. Within all your interviews, look for a subset of scores that spike; that’s your early-adopter customer segment.

This process worked like a charm for us where qualitative feedback had failed to move us forward. When I scored the interviewees and sorted from highest to lowest, a pattern quickly emerged: the more technical the interviewee, the higher the score. This insight, that our customers were developers, was deeply buried in the morass of qualitative interviews, but scoring and sorting surfaced it at exactly the right time.

You may quibble with the specifics (so change them!), but the overall process is genius. This forcing function made a tremendous difference for us, as we worked to get through customer development.

The Lean Enterprise Experiment Canvas

20130809_eric-150x150This is a guest post from Eric Klaassen of Bloom, a consulting firm that helps companies grow online. We first met Bloom late last year in South Africa, and they’ve been pushing the envelope of applying Lean Startup concepts to big, established companies.

The success of the lean start-up methodology is increasingly resonating in large enterprises. Companies like Intuit, Amazon, GE and many others are implementing key principles of the lean start-up in order to deal with the complexity of their markets and the increasing speed of disruptive innovation.

Our company, BLOOM, has the main objective to grow companies online. While doing so, we see companies achieve great results using the lean start-up principles and tools. However, a challenge that especially the larger companies face, is translating the lean start-up theory to an enterprise setting. As the table below depicts, many differences between start-up and enterprise are at the root of this challenge.

Differences enterprise startup

One method that has proven to be very applicable in both start-ups and large enterprises that are among our clients is the Lean Analytics Cycle. This process provides the much-needed rigour in translating fundamental business problems into metrics that matter, creating hypotheses in order to test them, and driving change in the business from everything you learn.

The fact that the lean analytics cycle can also be applied effectively in large enterprises triggered us to translate it into a tool, based on Javelin’s Experiment Board.

Lean Enterprise Experiment Canvas

Although the canvas is designed to be self-explanatory, it could be helpful to have guide to use the canvas in your team. The text below explains how to use the lean enterprise experiment canvas.

Bloom table 2 - Lean Enterprise Experiment Canvas - 720px

1. Define most important metric and draw a line in the sand

Your most important metric is the one that allows you to track how changes in your products and services impact your business goals. For a start-up this means finding product-market fit, and a sustainable business model.

For a large company, the most important metric differs between departments. Customer service might focus on customer retention, where the IT department steers on number of roadmap items delivered. What is the one metric that helps your department contribute to the overarching goal of the company?

Three criteria help choose the one metric that matters: the business you are in, the growth stage of a company and the audience.

Once you have decided the metric to focus on, it is important to define the current value of the metric before you start experimenting (=set the baseline). If you don’t know what the current value it is, go find out. If you can’t find out, develop the instrumentation to do so.

With the metric and its baseline in mind, you need to set a target value for this metric in order to manage expectations across the team—in other words, you draw a line in the sand for everyone to see. Make sure the target is ambitious, edging on the uncomfortable. Better set high goals and not fully reach them than to aim low. This does not mean that small achievements (e.g. completed an iteration) shouldn’t be celebrated. Celebrate quick wins to boost team morale, but never lose sight of the work that’s still ahead. And if the goal proves impossible: remember that the line is set in sand—not stone. If achieving it proves too easy or too hard, you can change it in a later stage.

Additionally, we suggest you define a control variable to keep track of. This is advisable as experimentation comes with a certain level of risk. This way we ensure we are not improving one KPI at the cost of business as usual. For example: removing a complaint form will bring complaints down to zero, but it surely won’t improve customer satisfaction.

2. Identify and prioritize issues from your customer’s perspective

Once you start analyzing your product or service, you will often identify different problems. This number grows exponentially with the added complexity of having multiple business units, product, market segments and customers. Therefore it is important to keep in mind that not every problem is equally important.

A problem is phrased from customer’s perspective, not only forcing the team to clarify exactly the problem at hand, but also making it easier to share with other departments.

Before a singular observation is deemed an actual systemic problem, it should first be supported by patterns in data, customer feedback or additional anecdotal evidence. When support is found, it can be identified as a (validated) problem.

As to not waste scarce (development) time, efforts should be directed at problems with both high potential and high importance. High potential means that there is a lot that can still me improved; high importance means that the problem has a large business impact. Prioritising problems this way has a beneficial side effect: because you are able to show how important a certain problem is, it is much easier to obtain buy-in from stakeholders.

3. Define possible solutions

Defining the solutions should be done in cross-functional teams. All employees are able to provide valuable insights against the backdrop of why a certain problem exists and how it might be solved. The CEO has a strategic high-level overview, the customer service representative understands the most important complaints, and a developer might know how to solve a problem from a technical point of view. Including more than one function in defining the solution, limits the chance that the genius to your local problem has a negative impact on the business as a whole.

It is important to keep in mind that solutions can be of a more incremental or more radical nature. The low-risk incremental experiments often receive more support from the organization and its leadership. This is a common trap: by experimenting with incremental solutions, you can only climb towards a local optimum. To also allow identification of the radically more effective solutions, experiments for solutions should be a mix of iterative improvements and larger leaps. The iterative improvements help you climb to the top of the mountain that you are on, and the leaps ensure you find the highest mountains.

4. Decide on the test method which allows for maximum learning with minimal amount of resources

A significant part of waste prevention lies in the determination of the minimal effort needed to validate a solution. For start-ups this is often relatively easy; they can move fast and break stuff.

In contrast, large enterprises need to be more careful as there is more at stake (e.g. their reputation). When building an MVP as a test method, keep in mind that the minimal version of a feature should actually solve the original problem.

In addition, the test is only useful if it enables you to act on the results. Relevant options for minimum viable features for start-ups, could also be useful in the enterprise. In some cases, more creativity and care is required.

5. Define success before actually running the test

There is a strong cognitive bias to look for positive results in a test. Make sure to define success criteria upfront in order to prevent yourself from being overly optimistic after the test. Try to phrase your success criterion as: “During the test, I expect strong signal from at least X% of visitors/customers”.

6. Get out of the building

As indicated before, experimentation in start-ups and enterprises serves two different purposes. Start-ups are looking for a sustainable business model, and experiment to find product-market fit. Enterprises already have a customer base and execute a repeatable and scalable business model.

For an enterprise, the goal of getting out of the building changes towards finding out what provides the most additional value to your existing customers. Methods to do so in enterprises include:

7. Analyze the results and check if you moved the needle

This is a critical step in the process. After you have run the test it is time to see if you actually moved the needle. Based on the test results you have three options

  1. Pivot, try to solve a different problem
  2. Persevere, go to the next column and try again
  3. Declare a success and implement full solution

It would be great if all tests are an amazing success, but in reality tests will be invalidated. This is not a bad thing at all. At the very least you prevented a lot of time wasted fully implementing a solution without validating it first. If a solution is invalidated we can choose to focus on a different problem (pivot/give up), or we can think of a different solution. Because we defined the problem as both important and with high potential, you probably want to do the latter.

If a test is successful it is time to scale up and create a lasting solution involving a larger team and tighter integration with the existing business.

Additional resources

The Bloom team has created a Slideshare presentation that annotates the steps described above. We’ve embedded that presentation below so you can see how they move through the Lean Experiment Canvas at each stage.

The enterprise experiment canvas was developed based on our experience applying lean in large enterprises. By publishing our tool we hope you can bring the approach to your organization as well, so feel free to download and use the template.

If you have any questions regarding the content, or if you want to share your experiences from applying these tools in practice, use the comment form or reach out directly at any time

Lean Analytics en français !

We’ve seen copies of the Polish and Korean versions of Lean Analytics in the wild, and spoken with a few of the other translators. We’re excited to see the book reach so many new readers. In the meantime, we’ve been doing a bit of translation of our own!

Photo of the WAQ stage from way up high, by Andréanne Beaulieu

Photo of the WAQ stage from way up high, by Andréanne Beaulieu

Last week, I spoke at Web A Québec, a conference on web technologies that happens in Québec City. I speak French (but far from perfectly) and figured this was a good time to see if I could present some of the ideas behind the book, as well as some lessons about how larger organizations are putting it to work, in French. I translated the slides (and then got some tweaks from the organizers.)

Anyway, here’s the deck, translated. So now you know your Empathie, Fidélité, Viralité, Revenu, and Échelle, whether you are a Marché biface or contenu généré par usagers.

Lean Analytics for Intrapreneurs

Later today, I’ll be speaking at the Lean Startup conference in San Francisco. It seems like only yesterday that Ben and I first taught a workshop on Lean Analytics, prior to the book’s launch. Since that time, we’ve visited a dozen countries, spoken with hundreds of founders, and found out that it’s being translated into eight languages. To say we were surprised by the progress is an understatement.

Rather than repeat last year’s content—which is widely available on Slideshare, on Udemy, and in classrooms by now—Ben and I have spent the past six months talking to Intrapreneurs. These are people within larger organizations, trying to create innovation. It’s a hard, often thankless job. After all, if a startup is an organization designed to search for a sustainable, repeatable business model, then a corporation is designed to perpetuate a business model. And in environments of rapid change and high uncertainty, perpetuating business models is deadly.

The most fundamental truth of intrapreneurship is that the difference between a special operative and a rogue agent is permission.

windmillIf you’re trying to change things, but don’t have organizational backing, you can do it—but you have to be subversive, and pick your battles. You’re tilting at windmills, battling on a pitched field against large, slow, old-fashioned incumbents. On the other hand, if your organization has a deliberate portfolio of innovation, then you’re going to need a program to find, test, incubate, and integrate new ideas.

Batch size changes everything

One of the biggest changes affecting companies of all shapes and sizes is the ability to do things in small batches. Once, scale mattered a lot, because it the only way to get the incremental cost of a customer down was to amortize fixed costs across many sales. This worked well, for a while, and gave us everything from mass production to broadcast media.

But that model is crumbling: tools like social media and on-demand printing and automation are reducing the economic order quantity. Part of this is because software is eating the world, optimizing the back-office and supplanting other channels as the dominant means of communication and delivery on the front end. Consider how quantity, cost, lead time, self-service, and customization vary across different production models.

EOQ of one

How customer value changes as technology moves production to an economic order quantity of one.

Is it any wonder that we’re seeing most innovation in digital sectors?

Big companies are taking notice. We’ve spoken with around 30 large multinationals—GE, DHL, SCA, Motorola, Google, Time, VMWare, Metlife and so on—in the last six months. Each of them has a slightly different take on innovation:

  • Some prefer to incubate new products in-house; others favor acquisition.
  • Some source ideas through hackathons and crowdfunding; others work closely with early-adopter customers.
  • Some believe innovators need isolation, in a kind of Skunk Works model; others want innovation to live within the business units that will ultimately reap the benefits of invention.
  • All seem to differentiate between three kinds of innovation. Some use the Three Horizons model; others distinguish between core, adjacent, and transformational projects.

Despite these differences, they all agree on one thing: that innovation must happen, and that to survive, companies have to constantly reframe the business they’re in and disrupt themselves.

We’re myopic about how

For us, the real lesson of the last six months’ research is that companies spend too much time worrying about adjacent markets (who they sell to) or adjacent products (what they sell them) while ignoring how they sell. It seems that while everyone knows about product/market fit, there’s a myopia around method.

Igor Ansoff’s product/market matrix is common wisdom for business strategists. We’re taught it in business school, and it’s the basis for most strategic discussions of diversification. And yet it’s only two-dimensional, focusing on product and market. There’s no method. It assumes the how. This is undoing of many incumbents. One of the things we’ll propose today is that intrapreneurship is about product/market/method iteration, and that innovation involves changing one (or more) of those three dimensions.

method venn diagram

The three dimensions of innovation that Intrapreneurs can pursue.

The more dimensions you’re changing, the less your metrics resemble typical business cases and the more they focus on de-risking your assumptions through validated learning.

I should point out that a marketing purist would argue that “product” includes the pricing, distribution, and promotion as well as the product itself, and therefore encompasses the “how.” I should know; I’m a marketing purist. But it’s worth breaking out how as a separate thing, because it’s far too often overlooked. Call it go-to-market strategy, or unfair advantage, or method—it’s still the thing people forget, and yet it’s the thing that drives most of the successful innovation we’ve seen.

Amazon, at its core, sold books to readers. Neither the market nor the product was new; it was the method (e-commerce, with recommendations and a focus on logistics) that was new. Later, they were able to diversify the product (kitchenware) and the market (people with poor eyesight who could read Kindle books with large typefaces.) Amazon gets this, which is why it experiments with cloud computing and drones.

Indeed, there may never have been a company as good at iterating on how as Amazon. This is the core reason why its stock price is high despite its score on traditional dimensions like profit and margin. Amazon is really good at cycle time, and while accountants don’t have a good way of measuring “how fast you learn and experiment”, capital markets do.

There’s no evidence about the future

When companies “assume the how,” they reinforce the processes, IP, and organizational structure that makes them really good at the way they work today. For decades, management theorists have urged us to grow and standardize, believing that tomorrow is the same as today, only more so. As a result, sustainable competitive advantage came from those who achieved scale and predictability.

But here’s the thing: there’s no evidence about the future.

Rita Gunther McGrath, author of The End of Competitive Advantage, says that sustainable competitive advantage allows for inertia and power to build up along the lines of an existing business model, which will soon die. Instead, she says, we should seek transient competitive advantage. And this is why Lean methods are so relevant to big, entrenched organizations.

Back to analytics

Our discussions with Intrapreneurs, CTOs, and managers of innovation labs have taken us far afield from the original scope of the book. We haven’t spent as much time talking about analytics, in part because what you measure depends on the change you’re trying to produce. In many markets—particularly those without direct customer instrumentation—intrapreneurs have to use proxy data to estimate things like virality, engagement, and conversion rate. This is hard, and full of errors.

Intrapreneur proxy metrics

Intrapreneurs sometimes need to settle for proxy metrics to measure their business.

There’s also a lack of comparative data across incubators and innovation programs, although this is gradually being addressed as companies formalize their programs and communicate among one another.

Today’s workshop will be our Minimum Viable Presentation. It’s the first time most of the 265 slides have seen the light of day, and I’m eager to see what works and what sucks so I can get to the next iteration.

Freemium and the feature/business tension

Ben pointed out this week in a great post that the key to success for many businesses is to convince users to adopt a tiny, new, addictive behavior. This is essential for the stickiness phase of a company, because it’s what keeps people coming back, and using your product, from which you can then learn and experiment.

But as startups transition from stickiness (what users want) to revenue (what the startup wants) they often face a fundamental tension between usability/addictiveness and money/onerousness. That’s because keeping the money flowing sometimes means not making the users happy. While this might seem obvious, it’s behind much of the hand-wringing and analysis-paralysis of growth. It’s a really hard balance to strike, and it’s one of the reasons startups with good stickiness and virality fail to make it to revenue.

For free B2C applications, if you’re not paying for the product, you are the product. This is certainly the case with social platforms. Facebook pays for all your Likes with advertising revenues; Twitter sells your stream of Tweets as a firehose for others to analyze.

You often can’t do what you want on a free platform. Try mailing someone a link from Facebook on a mobile app. First, you have to open the link in an external browser to get the URL. Sharing is a critical KPI for Facebook, and this is one way to make it harder to do elsewhere.

Facebook has a number of ways to limit your ability to share. For example, it can block content in subtle, even insidious, ways to prevent it spreading.


Is Tumblr really spammy or unsafe, as Facebook suggests? Certainly, the company would prefer that sharing happens on their platform.

This kind of functional jerrymandering isn’t just limited to free B2C applications. Even paying for something is no guarantee of a good experience and you likely still are the product. Getting the functional balance right is a fundamental challenge for startups that use the Freemium model of customer acquisition.

Consider expense reporting service Expensify. I love it. I rely on it. I recommend it to people all the time. But the UI by which you upload, crop, and rotate receipts you’ve scanned is atrocious. So the product metric that you’d care about in the stickiness stage—usability—is bad. If you’re curious, here’s a quick video of me trying to rotate and crop a receipt.

On the other hand, Expensify makes money from a scanning service where, rather than you having to do it yourself, they charge you a small amount to do so.

A strong business metric (revenue from expense report scanning) is encouraged by a weak product metric (frustrating usability).

If you’re a product manager, that sucks.

This is a natural tension. It happens in all product categories. For example, the LinkedIn app for iPad is beautiful. But when you’re reading a story in it, you can’t copy a link to the iPad clipboard. This aggravates me, but it’s a price I’ll pay to play within LinkedIn’s walled garden. For LinkedIn, this is a way of keeping sharing and interaction within their application, encouraging shares within LinkedIn itself rather than, say, on Twitter or email.

A strong business metric (in-network interaction) is encouraged by a weak product metric (crippled sharing functionality).

Evernote is another example. The company wants users to bump up against its maximum free storage threshold so they’ll upgrade to a paid version of the service. That’s why they acquired screen-snapping tool Skitch. They removed a number of features from Skitch when they acquired it, changing how the tool ran. They also added synchronization with Evernote—on by default, of course.

Skitch sync preferences

This means that, towards the end of the month, you start getting warnings from Skitch saying it couldn’t sync its images with Evernote. Each of those warnings is a chance to upsell Evernote users—but it quickly becomes an annoyance, making users save images selectively rather than automatically.

Screen Shot 2013-04-29 at 9.10.50 PM

A strong business metric (upgrades to paid accounts) meets a weak product metric (nagging reminders and interrupted workflows).

In the Skitch case, what Evernote did prompted an Engadget post recommending the best screen-capture replacement to its readers.

LinkedIn, Expensify, and Skitch aren’t B2C applications. But they are freemium products, and they all need to balance product joy with occasional nudging. The next time you’re building an application, or using one someone else has built, take a minute to consider what constraints the designer is working with. Many of them are a function of the business model, rather than look, feel, or usability.

As we mention in the book, Freemium is a risky customer acquisition tactic for a number of reasons, and only works for certain business models and usage patterns. Before you decide Freemium is right for your startup, you need to understand the risks of intentionally frustrating your users.

Great products strike a balance. They make users eager to pay for the service in order to support it, rather than extorting money for features the users feel should be included. This is a real art: too onerous, and you alienate users; not tough enough, and you can’t force people to upgrade or pay in the way you hope, and your business can’t pay its bills.

Now it’s your turn: what metrics do you think a startup should employ to strike the right balance?

Guest post: Vanity Celebrations

brydonpicBrydon Gilliss founded the shared office space ThreeFortyNine in Guelph where he plays with Startupify.Me, Ontario Startup Train and 20 Skaters. A serial entrepreneur and fervent community builder, he’s also busy organizing a train-full of founders for this summer’s International Startup Festival.

One of the most often-repeated themes from Lean Analytics has been this: If a metric won’t change your behavior, it is a bad metric. So when Brydon mentioned to us that we’re all celebrating the wrong things, his comment made a lot of sense. Here are his thoughts on the subject, in our first guest post.

The moments we choose to celebrate say a lot about what we consider important. They’re a proxy for the metrics we value, because we’re signalling to others by their very celebration. And yet, I’ve always been of the belief that startups tend to celebrate the wrong things.

If that’s true, what signals are we sending? We celebrate product launches, government grant acceptance, fundraising, winning pitch contests, and so on. Too often, these are the vanity metrics of our startup ecosystem.

Of course, some of these events are worthy of celebration. A grant lets us live to fight another day; a winning pitch might drive sales or help us to hire a key employee. But they would be way down on my list, personally, if my goal was to build a real business. Let’s stop concentrating on celebrating events like taking on debt or winning what is often little more than a beauty contest—and focus instead on what we should celebrate but rarely do.

At ThreeFortyNine, we celebrate the achievements that matter to the business model. Consider, for example, the first time you sell something to a complete stranger. That’s worth celebrating because it’s the first sign your business might have legs of its own. In our Founder’s Club events, we celebrate selling our first train tickets to strangers; Foldigo celebrated its first-ever sale to a stranger. Our plan is to build up this list and move it into our monthly socials.

We’re building our Startupify.Me program around the concept that talented developers stepping into startup life need options. Incubators, accelerators and government grant programs funnel them into a single, traditional path, thereby discouraging experimentation. We want our cohort to have the option to create a lifestyle business or even a small, local business—if they choose. Of course, any of them can still try and swing for the fences, but the key is that they have all of the options available to them when they start.

“We didn’t get to where we are today thanks to policy makers – but thanks to the appetite for risks and errors of a certain class of people we need to encourage, protect, and respect”,

—Nassim Taleb

Only in recent years have books like Lean Analytics begun to draw out the real risks of obsessing over feel-good data that does little for the business—so-called “vanity metrics”. There’s a very real danger if a young entrepreneur believes that success comes in the form of taking on debt, winning a pitch contest and launching a product. Those may be required for some businesses but they shouldn’t be misconstrued as success.

Part of the challenge here is the proliferation of what I call success turnstiles in our ecosystem. These are entities whose prime motivation is to funnel as many businesses as possible through their turnstile. It’s a pure numbers game for them as they chase their success metrics. These entities tend to be government funded and these success metrics are defined by bureaucrats and can be tracked up the organizational hierarchy to a speech-writer’s desk.

We need to lead real conversations about what success is because it comes in many shapes and forms. Advocates of this more mindful form of celebration include Jason Cohen imploring founders to get 150 customers instead of 1000 fans and Rob Walling helping startups to start, and stay, small.

Here’s an initial list of milestones and accomplishments worth celebrating to get you started.

  • Performed 30 interviews with real potential users.
  • First customer acquired.
  • First customer acquired without knowing where they came from.
  • Covering your monthly personal costs.
  • Identifying the first product feature a potential customer will pay cash for.

Which vanity metrics need to stop being celebrated? Which do we need to celebrate more?

SXSW, bikes, and the Zen of finding things out

This is a really long post. It’s overly personal. It’s a bit of a travel triptych. But if you read through it, I hope you’ll find that there’s an important lesson in here for entrepreneurs.

I was in Austin for SXSW this year. It was my first time there—I’ve always had some other event to attend. I’ve been told it has long since jumped the shark, and many who’ve decried its demise say it’s proof that technology is mainstream. With that in mind, and not knowing anyone, I fully expected it to feel like crashing someone else’s high school reunion only to find that all the cool kids had already snuck out the back for a smoke.

I was presenting a workshop on social media measurement, playing the part of the curmudgeonly entrepreneur; and I was there for the launch of the book, and as a mentor for Eric Ries’ Lean Startup event.


I also managed to squeeze in an interesting afternoon with librarians at a guerilla “salon” they’d erected just beyond the confines of downtown Austin. Turns out I knew people, too: Blake Robinson of Analect took me under his wing, and my colleagues from Decibel showed up towards the end, ushering me into a fantastic set by Flying Lotus. I also had a bumpy, sweaty ride around the city on an RV that felt (and smelled) like a moving version of Spring Break. It says a lot about the festival that this RV had just performed engagement ceremonies for the Twitterati.

But all of these paled in comparison to my experience with bikes.

I need to be less sedentary, and I’m trying to find hacks to do so, as I’ve outlined elsewhere. When I started planning for SXSW, the closest hotel I could find was five miles North of the city. Wary of waiting in interminable shuttle lines or spending a lot on cabs, I thought that it might be good to have a bike.

A bike has other benefits, of course. It’s good exercise. It doesn’t burn gas. Bought from Walmart or Target, it costs less than six cab rides. And I can leave it to Goodwill, or a Boys & Girls Club, or some other association, so someone can benefit from a nearly-new bike.

Enamoured with this idea, I shopped around. Sure enough, a simple, one-speed road bike was $99 on I’d made up my mind: In a few weeks, I’d buy it, and ship it to a friend’s in Austin. I didn’t want it lying around in her house for too long, after all.

When the middle of February rolled around, I went back to Walmart, only to find the price had gone up by $50. Undaunted, I bought the bike, and shipped it. I knew I’d have to do some assembly, but this seemed a small price to pay.


Shortly after I’d drop-shipped the bike, I found out that another friend (who’s busy changing the world) wouldn’t be able to make it, and her partner had a spare room in downtown Austin at La Quinta, just a few blocks from the conference. The long journey I’d feared would, for the most part, not happen. I’d be walking distance from things for much of the conference.

(If you came here for Lean Analytics stuff, bear with me—we’re getting there.)

A few days later, I flew from Montreal to Newark, where, apparently, they don’t know how to deal with snow. As a result, I arrived in Austin late, missing a dinner and a Big Data meetup. I took a taxi to the Holiday Inn Express far North of the city, and crashed, surrounded by boxes of books rushed from the printer to my hotel.

I woke early the next morning, and my Austin friend came to pick me, my luggage, and my boxes of hot-off-the-press books up.

Books at the hotel

We drove to a Walmart near her house for supplies. This was my first real warning. In front of me were rows and rows of pre-assembled bikes, hanging from the rafters in an array of styles.

Prebuilt bikes at Wal-mart

Had I known I’d be here to buy a helmet and other things, I could simply have bought a bike that was ready to ride, and skipped the assembly, and not needed tools. Lesson learned.

Eventually we found our way to the bike supplies themselves. These were, to say the least, lacking. Most of them featured a licensed cartoon character of some sort, and few of the helmets fit my head.

Walmart bike gear

The locks were witheringly frail, but I rationalized that a cheap bike was unlikely to get stolen. In the end, I left with a helmet, lights, a bike lock, and some basic tools. I was also $97.22 lighter.

Walmart receipt

Then we went to her place, and I spent an hour or so assembling the bike. This was a sweaty, uncertain process. Fortunately, I’d found the right tools, but even then it was tricky in places, particularly in getting the brakes properly aligned. After an hour, I stood back to review my handiwork.

Assembled bike

Now, if you look closely at that picture, you’ll realize something that had only just occurred to me. Those tires aren’t inflated. Lesson learned. Having not thought this through, I now needed to find a bike pump. My friend drove me to my second hotel, downtown—already, I was missing the opening sessions of the Lean Startup workshop at the Hilton—and fortunately the parking staff had an air compressor on hand.

I thanked her for her help, changed, and biked off towards registration. I flew down a surprising number of hills on my way towards the river, regretting with each street I crossed the potential energy I was bleeding off, and the kinetic energy I’d have to give back on the return trip.

I need to pause, at this point, to say one important thing:

Having a bike at SXSW is fantastic.

The feeling of caroming around corners, hair in the wind, street rolling by, is wonderful. It’s exhilarating. I have no regrets. But if that sounds like foreshadowing, well, it is.

Badge and swag retrieved from the well-oiled machine that is SXSW, I hopped back on the bike—which handled fairly well—and pedalled a few blocks to the Lean Startup event. I spent some time there, and left to meet Blake at a party. I walked out of the Hilton with a spring in my step, ready to board my trusty steel steed and head off to the event …

And stopped. My front tire had a flat.

I texted Blake to tell him I couldn’t come to the party, because I had to fix my tire. Then this happened.


Apparently, SXSW provides. In this case, it provides a Chevy Volt party next to a bike repair shop.

Okay, if you’re here for the Lean Analytics stuff, this is where it will make sense. Because as I walked the ten blocks to Mellow Johnny’s, I had time to reflect. Normally, all of this would have made me furious—paying too much, realizing I’d made dumb mistakes, having a flat. But I was kind of enjoying it. My whole frame of mind had shifted from “I need a bike” to “I am running an experiment.”

As I’d been preparing for the trip, I’d thought to myself, “maybe someone should start a business for travellers who want to keep in shape.” For all the reasons I’ve mentioned, having a bike in a city is great. And giving it to goodwill—possibly with a tax receipt in the process—might make it a viable business model. But I hadn’t got out of the building.

Now here I was, decidedly out of the building, walking a dilapidated bike down a dusty street in a city I didn’t know. And I was loving it. I’d learned at least ten things about this hypothetical bike-in-a-city business in less than a day, and all of them were crucial to the business. Never mind that I had no intention of setting up such a business. Simply viewing it as an experiment turned every mishap into a triumph.

It’s hard to explain the Zen-like calm that had overcome me. But I didn’t have time to muse any more—I’d arrived at my destination.

Random bikeshop

Outside Mellow Johnny’s, it was SXSW as usual—dozens of people, all staring at their phones, oblivious to the other people they’d travelled hundreds of miles to see. I walked my bike into the store and introduced myself to one of their staff.

His first reaction was a compliment—he liked the bike. It seems the designers at Walmart had conceived a vehicle that, on initial inspection, looked solid and well designed. I explained the provenance of said bike, and the history, and he quickly changed his mind.

He agreed to repair the tire, and check the brakes for me. There was a small tear in the tube, so he’d replaced it. As it turns out, the tires on the bike weren’t standard, requiring not only a new tube but also an adapter for the valve. Lesson learned.

Johnnys receipt 2

$30 poorer, but delighted with the staff, service, and coffee at Mellow Johnny’s, I headed off. I was only slightly worried by his parting words: “I fixed those brakes as best I could, but I wouldn’t go down any hills if I were you.” After a few more events that evening—abetted by my ability to flit from venue to venue quickly—I returned to my hotel, locked up the bike, and went to sleep.

The following morning, I found a relatively unsullied coffee shop with good Wifi, did some work, grabbed a great brunch at an Exact Target party, and went to see friends at the Canadian tech pavillion. By now, I’d noticed that my bike had some wear and tear. The seat, for example, had started to split from the moisture and exposure of the overnight rain. Lesson learned.


Then I returned to my hotel for a board call, and headed back out to meet my friend and past co-author Sean. We decided not to wait in the line for Girl Talk, and instead headed to the Driskill.

Now, if you’d been out all night, you might be upset when you walk out of the bar and find …

… your back tire now has a flat.

Not me. My Zen-like attitude was unassailable. Joy! More data points for my test case! More lessons learned!

Sean clearly thought I was mad. But I was reminded of something Dave McClure said at Startupfest last year: The only thing worse than bad feedback is no feedback at all. Here I was, walking back to La Quinta with a flat tire once again, absolutely swimming in negative feedback!

The following morning I had to pack my bags and leave them at the downtown hotel. My respite from the long ride was over; I’d have to rely on my bike to get to and from the original, distant hotel. I walked back to Mellow Johnny’s and had the bike repaired a second time.

Johnnys receipt 3

Another $24.80 spent. I did learn, however, that renting a bike from Mellow Johnny’s would have cost me $25 a day. I took a moment to delight in this nugget of competitive pricing data, and another lesson learned.

I biked off for a quick chat with some of the folks from Soundcloud, then made my way across the bridge to a video interview I had scheduled with the folks at Software Advice (which I’ll post here sometime soon.)

Did I mention that having a bike in Austin is amazing?

The wind across the river, and the spring sun, were fantastic. I climbed a short hill, and turned down a smaller road towards the studio. And then I noticed that the wheels were making a strange noise.

Flat number three.

Again, at this point, I’d normally have thrown the bike off an embankment. But this was a lesson, to be consumed, considered, and shared, dear reader, with you. I walked my bike the remaining distance to the studio, and did the interview with Ashley Verrill, who handled a rather sweaty guest with poise and aplomb. Shortly afterwards, I was scheduled to speak with a bunch of librarians about Big Data, and they’d kindly offered to pick me up. So I asked them if they had room for a bike.

We squeezed the bike, and five passengers, into an SUV and crawled the few miles to the next venue. It was during this drive, shoehorned between seats and spokes, that I learned I was far from the first to consider getting bikes in each city I visited. No, David Byrne has been doing this a long time, and has even written a book—the Bicycle Diaries—about it.

The hypothetical-startup-founder in me celebrated: I’d unearthed a competitor! Another lesson learned.

When we finally reached our destination, my Austin native—who’d patiently waited while I assembled the bike in the first place—was there. She told me of a friend had recently lost everything, even her car, and badly needed a way to get around. So I gave her the bike, with $20 to fix the third flat.

Things have a funny way of working themselves out.

I’d spent $314.49 on my bike experiment. The rest of the conference was uneventful by comparison. There were some great parties, and the aforementioned RVIP. Reggie Watts helped us skip a line. And the workshop went really well, giving me insight into how social media marketers and agencies think about metrics—and into why vanity metrics won’t die. But that’s for another day.

I learned a lot from the experience. It reminded me just how vital it is to get out of the office and try something yourself. And it demonstrated what a difference it makes when you think of your experiences as experiments. They cease to be disappointments and become learnings. Even if you’re not starting a company right now, pick a crazy, hare-brained idea and see if it works. It’s refreshing.

Ultimately, when you’re trying something new, you’re not defined by your idea, your product, your plans, or your services. Rather, you’re defined by what they’ve taught you. This is true for founders, but it’s also true for humans. It’s a much more Zen way to look at your life.

And that’s why, for me, the best thing about SXSW 2013 was a shitty bike.