Some teams have a perverse pride that comes from "we test in production" but that can be a path to a failed product. As an alternative, any release process should consider a Customer Zero program so that before your first customer gets access to your delivery, you've done some integration testing on the adoptability of your new feature or product by having somebody act the part and test from the perspective of a key persona.
Product Success
We often imagine adoption as a workflow like this:
The reality is often that features get thrown over the wall, early adopters run into bugs and crashes if they can figure out how to use it at all, and the experience sours some users from ever giving you a second chance.
The key is that product engineering teams need a full end-to-end running water through the system type of test plan. It isn't just evaluating an API in isolation, but the sign-up process, finding usage information from a dashboard, verifying the documentation is clear, using the API within an application interacting with other dependencies, etc.
Running a Customer Zero Program
I did not invent the term Customer Zero, but the concept is that before your first customer touches a new release, you need to find customer[0]. The spirit is that anybody at a company who did not directly contribute to the creation of a release is a good candidate to verify the release is ready for the first customer.
For example, if you are at a company of sufficient size or have pizza box sized teams, somebody from a different group may have a little bit of context but not a lot of detail about your project. That is the ideal candidate, somebody who can look at things with a fresh pair of eyes and a vested interest in helping share their opinions in making things better.
Staffing a Customer Zero Program
A really great balance for a Customer Zero program would be to recruit:
- A developer advocate from the Developer Relations team
- An engineer from a different team or product line
- A pre-sales or solutions engineer
- A customer success or technical support agent
Typically these folks will need to onboard and ramp-up with elements of a release anyway, so this can give you the added benefit of verification in a coordinated way. That coordination is to assign each person with a representative task, not just an open-ended "go try this".
Example assignments might be:
1) Follow the Quick Start or Getting Started guide and see if everything works.
2) The engineering team tested things within a react environment, go try to use Svelte and see if you can get an equivalent hello world up and running.
3) If the product aims to solve a problem for a particular industry, try to create a prototype that could be used by one of those industries.
4) Write a blog post that demonstrates use of this feature along with a new popular library.
When done well, not only do you get a list of valuable feedback but some additional outputs like blog posts, sample apps, documentation, etc. that can improve the overall experience.
DX Testing Service
If you need help performing a developer
experience audit, reach out for more details: audit@developerxp.com
Keys to a Successful Customer Zero Program
A successful program starts with recruiting a variety of participants and providing them with small projects to complete.
✅ Remove curse of knowledge by inviting people who don't work directly on the product
✅ Mirror real-world scenarios
✅ Try unexpected use cases or integrations
✅ Create learning resources as a by-product of the effort
Given the right people and tasks, what exactly do they do?
Friction Logging
The goal of friction logging is to have a process for capturing the end-to-end experience when onboarding with a new feature, product, or API.
Friction can be subjective, but usually falls into categories such as:
- anything that causes frustration
- things that are complex or feel unnecessary
- whatever cannot be easily understood
- difficulty in figuring out what next step to take
Friction logging is often done by capturing error messages, screenshots, and notes that can be annotated at the end of the testing session. This may be familiar to anybody with User-Centered Design and similar usability testing experience.
The point is that teams can use these journals, identify the significant pain points as well as usability issues and come out with a prioritized list of actions to take.
Friction Recordings
Using video conferencing tools has proven to be a successful alternative to a written journal.
Schedule calendar time for customer zero participation.
This can be helpful for the participants for a few reasons. It's easy to de-prioritize usability when other deadlines demand attention. By blocking time out on the calendar we can increase commitment and reserve time to gather feedback without delaying a release. Scheduling can also help timebox how much of a time commitment is needed from somebody on customer zero to alleviate concerns over a large amount of effort.
If participants need more than one or two hours to learn how to use a new feature or product, that in itself is probably a good indication some work needs to be done to improve usability in the developer experience.
For examples of what this looks like, head over to one of these channels and subscribe to watch a future session:
Keys to Capturing Customer Zero Feedback
A word of caution. Feedback is important and you should listen and accept it, but not all feedback is good feedback or actionable feedback. A well planned customer zero program will have multiple people with varying backgrounds try things out. If everybody in the group runs into similar friction points, that my friend is a trend worth acting upon. Everything else still needs to be prioritized by frequency, risk, workaround availability, etc.
✅ Have new members of your team during onboarding try things out
✅ Regularly audit the health of your onboarding experience end-to-end
✅ Learn from friction experienced and prioritize that work as important
✅ Try recording sessions to get deeper insights while making it easier for people to participate
Developer Experience Consultation
If you need help improving your developer experience, reach out for more details and let's chat: devex@developerxp.com
Top comments (0)