Lessons learned from a failed test

This is a continuation of my last post, The importance of knowing your customers.

FGrade-LGWhen we design tests for a new mobile app, we make sure our management team participates as well. Sure, we all have other (and more important) things to do than spending time with customers. Right?

WRONG! There is no possible way you can understand a situation better than living it firsthand. There is no way I would have devoted so much energy and time to fix a process unless I had felt the shame and agony that I experienced a few weeks ago.

Here’s the email I sent to my team after a failed test involving a mobile app for consumers:

(names have been change to protect the innocent)

Date:  September, 12, 2014
From: Leonardo Shapiro S.
To: Team
Subject: Lessons learned from a failed test


Hi team,

My apologies for a long email, please read it completely and thoughtfully. I will start with a small narrative of today’s test of our first-time user experience:

 First-time setup
  • We had a customer, Joshua, who quickly agreed to participate in our test and was using a Samsung Galaxy II smartphone.
  • The set-up process was very painful and took more than 45 minutes.
    • After trying various ways to install our app (FTP download from our site, USB, etc.), I had to send it to him via email
    • His phone’s connection speed was very slow (not our problem, but a reality for our target demographic)
  • To make a very long story short, after 50 minutes Joshua finally got the app working on his phone (by then, his girlfriend was pushing him to drop it, since “we have better things to do on a Friday night”)
Using our app
  • Even though we thought our app was easy to use, Joshua was having a lot of trouble. I tried to help but by then, he was way too irritated to listen to me.
  • Next, I told him to take his proof of participation to a bank (since we are giving a monetary incentive to our testers) and collect his money. Since there were three people already in queue, Joshua had to wait yet again. Not good.
What Joshua said to us after
  • “Did you just make me waste my Friday evening for this?”
  • “So, you guys built an app that helps people waste their time?”
  • He didn’t say much after that, but my mother called me a few minutes later saying that her ears were buzzing (for all of you not familiar with that popular saying, I’m trying to find a nice way to say our customer was basically cursing us)
Conclusion and next steps
We lost a few bucks and I had the worst 57 minutes I’ve experienced in a very long time. However, I like to say that good feedback only grows your ego, but bad feedback teaches you a lesson, so…this is what I think we should do:
  1. Our users need to be able to download our app in <2 minutes
  2. If users cannot easily complete our on-boarding steps, then we should bypass these steps for now
  3. Let’s re-design the first-time user experience to show the “magic” of our app

Thanks for listening and I’m counting on your full support.




After making these changes, the next day we started making bunnies appear out of hats, we created a delightful experience, our customers loved it and we all felt very proud of what we’ve accomplished and where are we heading as a company.

In retrospect, I would say that the shameful and embarrassing 57 minutes and the few bucks we spent are now repaying us 100X!

The hard lesson for us: no matter how important you think you or your duties are, there’s nothing more important than spending time with and empathizing with your customers.

Tienes algo que decir?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s