Our head of product, Ben Grohe, shares how we found a way to use multivariate testing during the app development process.
Soon after we started developing our first mobile app, we realized that we are now building “shipping software,” rather than a website or web services. There are fundamental differences between the two, which proved to be a big challenge for our team—particularly, given our desire to continue using best practices of agile and lean development.
Product development at TheLadders
Before I continue, allow me to quickly give you some background on how we develop products at TheLadders. We live and breathe agile and lean development day in and day out. If you haven’t read it already, my colleague Michelle wrote a great article about how we work in cross-functional teams in our company. If I had to sum it up, our mantras are:
- Communication and working software over documentation
- Actual data and user feedback over assumptions
- Product iterations over extensive planning
In true agile and lean-development fashion, we are committed to the idea of solving large problems by breaking them down into smaller, more edible problem statements. Based on that, we create hypotheses, which we turn into user stories. Armed with those, we then start developing actual software. Once it comes to software development, we aim for short iterations to get real user feedback and actual usage data as fast as possible because this guides our decisions as to what we build next. This process is designed to maximize speed of learning about our users and our ability to solve their problems. In a way, you can say that we “test” our way into a great product; a word that captures the essence of this process pretty well is “MVP” or Minimum Viable Product. In other words, we aim to build iterations of smaller products to learn along the way, rather than one, over-thought, or over-engineered, solution.
This iterative approach to software development works really well when you develop web products, because in a web world you can release software quickly and globally for all your users. You can also easily test features with subsets of your users by using multivariate testing. To put it more bluntly: it doesn’t matter if you screw up, because this is part of the learning process and you can recover from it quickly.
Our mobile "app-roach"
With this understanding of our product development philosophy, you might be able to understand why we first struggled when we started to work on our first mobile app. Our ability to “test” our way into a great solution didn’t seem to be a viable option because a mobile app is software that your users download. And unlike a website, you can’t make quick changes to this software once it is in your users’ hands (quite literally). Consequently, critical bugs or feature flaws have a much higher risk associated with them. Releasing an unfinished MVP version of an app into the app store wasn’t an option because we didn’t want to receive 1 star ratings in the event that the app is buggy or users didn’t care for what we built. What’s worse from a product management side is that you are not able to test different features and experiences easily. In other words, the product that you provide to your users for download must solve the majority, if not all, of their needs. We call this the MDP, the “Minimum Desirable Product.”
Finding a solution
The root of the challenge lies in getting constant user feedback and usage data while we are still working on the app. For reasons stated, we couldn’t accomplish this with a “live” product, but there are other tools and ways to solve that problem. We devised a diary study with 11 participants from our existing user base that fit our demographic criteria to represent the majority of our total user base. After screening hundreds of interested participants we picked our final 11 and used them as a proxy for user feedback and app usage. After initial set-up, we frequently sent them new versions of our app via TestFlight – a tool that allows you to provision apps on iPhones without going through Apple’s App Store. After every version release, we interviewed each user for 30 minutes each week and studied changes to user behavior. This process was tremendously helpful to prioritize our existing backlog of ideas and features, and helped us to uncover new problems and devise new solutions.
The decision to “test” our app along the way with a well-selected, handful of users was one of the best decisions we made early on. In true agile and lean-fashion, we listened to the user, looked at data and then changed direction accordingly. To prove the success of this approach, one of our participants recently mentioned that “this app feels like it was made for me.” So, to come full circle and answer the question I asked at the beginning of the article: Yes, we believe you can be agile when you “ship software.”
UPDATE: The app is here! Download our free app, Job Search by TheLadders, today for your iPhone or Android. Visit theladders.com/mobile
Want to work with people like Ben at TheLadders? Check out our open positions.
Benjamin Grohé is the Head of Product at TheLadders, the premier mobile and online job-matching service for career-driven professionals. When he is not coming up with innovative ideas to delight our customers, he is celebrating his European heritage by cruising the streets of New York City on his Vespa or playing football (the REAL football).