Avoid common pitfalls when testing new software with your users.

Software pilots are a crucial step in the software development lifecycle. They allow IT teams to test the functionality, usability, and performance of new software with a subset of end users before rolling it out to the entire organization.

However, software pilots are not without their challenges. IT teams frequently fall victim to mistakes that jeopardize the success of the pilot and the ultimate adoption and success of the software at the organization. Let’s talk about seven deadly sins of software pilots and how to avoid them.

Blog1Sin 1 – Software for software’s sake

You are piloting software without a clear vision of its value to the organization.

Sometimes, software is being tested in an organization for reasons that are not aligned to any meaningful business objectives. Maybe it’s the favorite app of a specific leader, a way to appease a demanding sales rep, or something else. But the only software that an organization should be trying out is software that will fix real issues or help achieve the goals that move the business forward.

Instead, align all software to measurable value for the organization.

To combat software sprawl (and waste precious time and resources testing ill-fated software), take time to make sure you can clearly articulate (and measure) the impact the software should have. Make sure that desired impact is aligned with the goals and objectives that are important to your organization (and leadership). At BrainStorm, we call this exercise “building the ladder” and identifying the steps or links that connect this project with important organizational objectives.

Blog2Sin 2 – Missing or incomplete success criteria

You have not identified metrics or you’re running a pilot whose success will be measured only by sentiment.

Without a clear vision of what a pilot is trying to achieve (and how to test and measure it), software pilots often become directionless and meaningless exercises. In the end, IT teams might try to save the pilot by relying on a user satisfaction survey to make the final decision. This means that the decision to move forward or not could be made completely subjectively - without consideration of the factors that will actually determine its value to users and the broader organization.

Instead, identify KPIs or metrics that assess the software’s success at achieving expected outcomes.

A pilot is an opportunity to gather valuable insights and feedback on new software, but only if you have:

    1. A clear plan for what data you want to collect and,
    2. A defined path for measuring the results.

You should define the key performance indicators (KPIs) and success criteria for the pilot before it starts and track them throughout the duration of the pilot. This will help you evaluate the effectiveness, usability, and satisfaction of the new software, as well as identify any gaps or areas for improvement.

Blog3Sin 3 – Not selecting the right pilot users

You are piloting with only IT users or the wrong sampling of users.

Pilot users are the ones who will provide the most valuable feedback and insights on the new software. They should be a true representative of the target user group, have the relevant skills and experience, and be willing and able to participate in the pilot. However, IT teams often select pilot users based on convenience, availability, or seniority, rather than suitability, diversity, or interest.

Don’t mix up test groups and pilot groups. Instead, target users in IT as a test group, but build a pilot group that’s more representative of the intended audience (and large enough to give you meaningful data).

IT teams should carefully select the pilot users based on pre-determined criteria. Your pilot group users should:

    • Represent the target user group in terms of demographics, roles, functions, and needs.
    • Possess skills and experience sufficient to use the new software and provide constructive feedback.
    • Be willing and able to participate in the pilot, follow the instructions, and complete the tasks and surveys.
    • Be a large enough group to give you representative and statistically significant data.
    • Match the demographics requirements of the software being piloted (e.g., enterprise social software requires a critical mass of users to provide a representative experience so your group should be large, collaboration software should be piloted by people that collaborate together, etc.)

Blog4Sin 4 – Not giving the pilot group clarity and purpose

Because you’re so close to it, it’s easy to fall into the trap of assuming pilot group users understand the pilot and will prioritize it.

A common pilot introduction email starts with something like “Hi Sarah, you’ve been volunteered to test this software. It’s coming in XX days. We’ll send you a survey after XX days.” Then the pilot comms mostly stop for the duration of the effort. This tactic leaves users in the dark as to why the software is being piloted, why it’s important to be testing it, what they are expected to test and try, how much effort they should be putting in, and other important details. This leads to underwhelming pilot participation and unmeaningful results.

Instead, clearly articulate what is expected of the pilot group and communicate with them frequently about those expectations.

From day one – and many more times after that –clearly outline for users the following information:

    • What is being tested
    • Why it’s important that this take place
    • Why they are a part of the test
    • What is expected of them throughout the pilot (including specific actions if appropriate)
    • How they should think about prioritizing their involvement in the pilot
    • Who is running the pilot and who can support
    • What resources they have and should use to help them pilot the software (see Sin 5)

Blog5Sin 5 – Not providing the pilot group with adequate support

You leave users to sink or swim with new software, believing that is the most accurate way to measure how good a software product is.

These days it takes more than just great product design to get user adoption. It takes outside help to support your time-constrained users (whose regular job is probably not technology testing) figure out why they should use a new tool, and where to go to get started or move forward if they get stuck.

Instead, give users the initial and ongoing support they need to be onboarded to the new software and meaningfully pilot the tool.

Provide the following to pilot users:

    1. Adoption and training resources that are available to users when they need them to learn more about the tool. Include information on the ‘why’ and the ‘how’.
    2. A forum for feedback, questions, and updates from the pilot team.

The BrainStorm platform was purpose-built to provide tools that help you drive and measure adoption. It’s great for pilots and general deployments.

Blog6Sin 6 – Running a pilot for the wrong amount of time

Your pilot is either too short and won’t give you enough data to make an informed decision – or it drags on for too long, creating fatigue for users and a reduced focus on value assessment.

If you rush your pilot, you’re not likely to get enough data on user behavior to make the best decision. Conversely, if your pilot runs too long, users might endear themselves to a product you ultimately don’t move forward with or disengage when new responsibilities come up. Additionally, pilots that last too long can make the business feel like IT is dragging their feet.

Instead, take time to assess the relevant variables and set a timeline that makes sense for the project; not the calendar.

It’s a classic Goldilocks predicament: you’re tasked to design a pilot that’s not too long, and not too short, but “just right”. But what’s the right amount of time to pilot is a product? Many factors, including the size and complexity of the software, the size, skill, and involvement of the pilot group, your resources to communicate and gather data, and more come into play.

Don’t limit your pilot to arbitrary standard business timelines (e.g., 30 days, the month of June, 3 months, etc.) and fight the temptation to give in to deadlines that aren’t in the best interest of your project (e.g., finish by Q3).

Blog7Sin 7 – Not celebrating and thanking pilot users

You’re taking your pilot group’s efforts for granted and moving forward immediately after closing the pilot.

Your pilot users are the early adopters and champions of your new software. They deserve recognition and appreciation for their participation and contribution.

Instead, visibly recognize the contributions of the pilot group in this work and if the software moves forward, share their successes to promote early adoption.

Don’t just take your results and run, Be deliberate about celebrating and rewarding pilot users for their efforts, achievements, and feedback. The rest of the organization will benefit from the pilot group’s knowledge, and you should encourage them to share their experiences and recommendations with their peers. By celebrating and rewarding pilot users, you can boost their morale, motivation, and loyalty, and foster a positive and supportive culture for change and innovation. They may also be more willing to participate the next time too.

Remember:

A pilot is not a one-time event, but a continuous process of learning and improvement. You should not treat the pilot as a final test, but as an experiment that allows you to test your assumptions, validate your hypotheses, and learn from your mistakes. You should also be open to making changes and adjustments based on the data and feedback you collect during the pilot. You should not be afraid to fail fast and pivot, if necessary. By learning from the pilot and adapting accordingly, you can improve the quality and outcomes of the new software, and increase the chances of its successful adoption and implementation.

BrainStorm can help you with pilots and rollouts

BrainStorm is the software adoption platform that takes a human approach to user adoption. By incorporating change management best practices, relying on learning & behavior science, and being firmly rooted in data, BrainStorm makes it easy on IT to manage their change and adoption efforts. If you want to learn more or connect with BrainStorm to tackle your next pilot or software rollout, let us know here.