Recommendations: Part III, Conducting a Pilot Study

tfons

July 20, 2012

¶ 1 Leave a comment on paragraph 1 0 The Value of Piloting

¶ 2 Leave a comment on paragraph 2 0 A pilot study, sometimes referred to in the business literature as a feasibility study, is used to gather data and test the technical or commercial feasibility of the project option or options identified in the case-development phase (Turner 2005a). Upon completing the business case, including the implementation plan, the library planning team should determine how and when to test, or pilot, the options before proceeding with a full rollout” (Davenport 2009). While conducting a pilot does not guarantee successful full-scale deployment, it does increase its likelihood by identifying problems that can be corrected or unexpected consequences to be avoided, or, when confronted with a worst-case scenario, by helping to determine that the project should not be undertaken at all (Van Teijilngen and Hundley 2001; Thabane et al. 2010; Pal, Sengupta, and Bose 2008). Politically, a pilot can also help decision makers justify the cost of a full-scale implementation. As Pal, Sengupta, and Bose (2008) note, “while a pilot study may or may not be able to capture all intricate details of the actual project, with careful planning and design it can address many potential issues which would not be possible to consider without a pilot.”

¶ 3 Leave a comment on paragraph 3 0 According to Turner (2005a; 2000b), in addition to testing the application of the chosen methodology, the data the pilot team collects assists in the selection of an appropriate risk-mitigation strategy. The piloting process should also be designed to engage stakeholders, communicate and promote the project within the library, academic institution, or wider consortial setting, and enable the planning team to begin to build capacity on each of these levels in preparation for full deployment should the pilot prove successful (Siegal and Smith, 2006).

¶ 4 Leave a comment on paragraph 4 0 Step 1: Write the project brief

¶ 5 Leave a comment on paragraph 5 0 Project management best practices should be followed in designing and managing the pilot.  Turner (2005a) recommends the process begin with the planning team’s creation of a project brief, a document of from one to two pages for a relatively modest pilot to over ten pages for a complex study. The project brief should include the following components (Turner 2005a):

• ¶ 6 Leave a comment on paragraph 6 0
• A background or introductory statement, briefly describing the context of the study and what will be done.
• A description of the purpose of the pilot, or why it is being undertaken.
• A statement of objectives or outcomes, along with the desired results or deliverables.
• The scope of the pilot, including how the data are to be gathered and analyzed, and any  anticipated challenges or constraints.
• The detailed work plan of how the pilot is to be carried out. This should include a timeline and either milestones to be accomplished or, for less complex projects, a simple task or activities list.
• A responsibility chart defining the roles and responsibilities of everyone involved.
• A list of stakeholders identified as having an interest in or as being affected by the project, and any plans to deal with possible adverse responses.
• A list of quality standards to be met are listed and defined.
• A cost estimate and schedule.
• A list of acceptance criteria or standards that are to be met before the project will be  considered for full implementation.
• Identification and description of any known risks, along with mitigation strategies to address them.

¶ 7 Leave a comment on paragraph 7 0 If the library feels it necessary, especially when dealing with complex pilots, to hire a consultant or draw on internal specialists possessing needed expertise, these participants should be included in the responsibility chart as members of the project team (Pal, Sengupta, and Bose 2008). However, it is recommended that the pilot team be kept small and tight, with a maximum of eight to twelve members who are invested in the pilot’s success and can work to aggressive deadlines (Workplace Competence International Limited 2002). Moreover, in a highly technical project, not all members of the planning team may be directly involved at the pilot stage, although planning team members not included should routinely be kept informed of its progress through pilot team reports.

¶ 8 Leave a comment on paragraph 8 0 Step 2: Execute the pilot

¶ 9 Leave a comment on paragraph 9 0 The project team carries out the pilot, or “realization phase,” following the specifications outlined in their brief (Boscherini et al. 2010). The associated timetable is critical, and enough time must be scheduled for adequate assessment without dragging the process out until momentum and enthusiasm are lost and the decision on whether to move ahead with a complete implementation is unnecessarily delayed. Turner (2005a) notes that pilots, by their very nature, are characterized by “short time horizons” and the “need to see quick results.” In business and technology, pilot projects are typically kept to less than a year given the competitive environment and rapid rate of innovation and product development (Workplace Competence International Limited 2002; Pal, Sengupta, and Bose 2008).

¶ 10 Leave a comment on paragraph 10 0 It should be emphasized that pilots are managed differently from other projects, as they “have as their primary objective the creation of the ‘first version’ of something . . . and [are] undertaken for the very purpose of acquiring the type of experience needed to set or estimate” conventional project management parameters if the project is continued (Workplace Competence International Limited 2002). Because the testing involves a new product or service, the work plan cannot be structured based on past experience; therefore, teamwork is usually broken down by stages, often into three-month time periods. Each work step should normally conclude with a deliverable, although this might not be in its final form (Workplace Competence International Limited 2002).

¶ 11 Leave a comment on paragraph 11 0 The pilot team must be given a “license to innovate” as part of their brief, with the pilot process thought of as a “collective investigation” and “strategic experiment,” rather than merely an initial implementation of a preconceived outcome. As such, pilot teams should be given broad boundaries and scope for creativity (Siegal and Smith 2006). Library management should give them as much room for maneuver as possible in order to facilitate a successful experiment, with “success” defined to include a determination of failure and project termination (Naslund 2010). Team members, and especially their leader, must be able to work collaboratively and be comfortable with ambiguity, having the capacity to “handle the stops, starts, fall backs and leaps forward which will characterize their work” (Workplace Competence International Limited 2002). As they proceed, they must also be able to recognize that while “one thing might not work, something else may” (Naslund 2010).

¶ 12 Leave a comment on paragraph 12 0 Step 3: Communicate with stakeholders

¶ 13 Leave a comment on paragraph 13 0 This step overlaps the pilot-execution phase. The pilot team must produce regular status reports to give assurance that the project remains on track and on time, and to make certain that stakeholders remain engaged and interested in the outcomes. Naslund (2010) recommends weekly or biweekly updates to upper management, depending on the length of the pilot. Also needed are appropriate communication venues enabling interested stakeholder groups to review and comment on the pilot’s progress, including public presentations, library or university newsletter updates, and chat forums on the library’s website (Siegal and Smith 2006). Naslund (2010) stresses utilizing every means of capturing feedback, including blogs, wikis, and other social networking opportunities.

¶ 14 Leave a comment on paragraph 14 0 Step 4: Evaluate the results

¶ 15 Leave a comment on paragraph 15 0 Based on the list of acceptance criteria set out in the brief, the pilot team needs to measure the results of the project. Pal, Sengupta, and Bose (2008) suggest a number of metrics to measure value generated by technology pilots, such as revenue gains, cost savings, time to market, on-time delivery, and market share gains, and identify four major approaches, either alone or in combination, to analyze project results:

0. ¶ 16 Leave a comment on paragraph 16 0
1. Economic (using financial measures such as ROI, or cost/benefit approaches focusing on tangibles, usually monetary gains or losses).
2. Strategic (determining the degree of alignment with the library’s, university’s, or consortium’s overall purpose, goals, or current initiatives; this may include focusing on intangibles, such as faculty and student satisfaction, institutional prestige, or benefits to the research community).
3. Analytical (risk analysis and scoring).
4. Integrated (utilizing such techniques as the Balanced Scorecard).

¶ 17 Leave a comment on paragraph 17 0 Especially for more intangible or qualitative measures, libraries should carefully consider how to capture feedback from stakeholders; how, particularly if using an integrated approach, they will weight the relative importance of these measures in forming their decision or acceptance criteria; and how detailed their evaluation process must be to satisfy the needs of the planning team and senior library management.

¶ 18 Leave a comment on paragraph 18 0 Libraries should also consider a separate evaluation team to monitor and evaluate results presented by the pilot group (Workplace Competence International Limited 2002). This team can be formed with planning group members not directly involved with the pilot, who can be augmented, if necessary, with members of senior leadership. A library’s investment of time and of operating and human resources in a pilot should be considered a “risk investment” that can lead to nothing. An evaluation team, with the charge of minimizing investment losses via timely review of the pilot, is in a position to stop a nonproductive effort early on (Workplace Competence International Limited 2002).

¶ 19 Leave a comment on paragraph 19 0 It is for senior management, deciding on the most effective or acceptable approach for their local situation, to determine whether a library should rely on the planning and pilot teams to evaluate results and render recommendations, or whether a separate evaluation team should be formed.

¶ 20 Leave a comment on paragraph 20 0 Step 5: Make a go/no-go decision

¶ 21 Leave a comment on paragraph 21 0 As indicated in Step 4, should the pilot not show promise during the execution phase, it can be terminated, although it is most likely that the process will continue until its planned completion date. After the results are analyzed, either the pilot or evaluation team will make their recommendation to the planning team, who will ultimately decide whether to proceed with full deployment. One of four decisions generally results:

• ¶ 22 Leave a comment on paragraph 22 0
• End the project and do not transition to a full implementation (project not feasible).
• Determine partial success and incorporate aspects into a revised pilot or full implementation (project feasible with some modifications).
• Determine success, but extend pilot and monitor closely (project feasible but more information or experience is desired).
• Determine success and roll out full implementation (project feasible as is) (Thabane et al. 2010).

¶ 23 Leave a comment on paragraph 23 0 Even in cases where the project is deemed unfeasible, the pilot can still be considered valuable to organizational learning. Turner (2005a) outlines several positive side effects that libraries can benefit from as they reflect on the outcomes of their pilot:

• ¶ 24 Leave a comment on paragraph 24 0
• The process may generate better ideas of how to mitigate risk.
• It may result in better ways to reduce uncertainty in project processes.
• It can create an awareness of what will work or not work in new product or service design that can lead to better subsequent pilot development.
• It can test the efficacy of instruments or processes used to collect and evaluate data in order to improve their usefulness in future pilots.

¶ 25 Leave a comment on paragraph 25 0 Moreover, conducting new product or service pilots can build the library or university’s capacity for strategic change (Turner 2005b), and foster a climate of innovation (Boscherini et al. 2010).

¶ 26 Leave a comment on paragraph 26 0 Step 6: Present the findings

¶ 27 Leave a comment on paragraph 27 0 Once the results are finalized and accepted, the planning group should disseminate them in summary form among all stakeholder groups. These reports should be honest, straightforward, and jargon-free (Naslund 2010). Some broad points for discussion might include:

• ¶ 28 Leave a comment on paragraph 28 0
• If the pilot was deemed a success, in what ways was it successful and which factors contributed to this positive outcome?
• If the pilot was deemed unsuccessful, in what ways and to what extent was it unsuccessful? Were there any aspects that did perform favorably? What should have been done differently to achieve a better result overall? Were the problems or undesirable factors encountered political, structural, procedural, etc.?

¶ 29 Leave a comment on paragraph 29 0 Dissemination of results should not stop with written reports. Other appropriate channels useful in informing stakeholders and in maintaining their support—in addition to presentations in various venues and formats, informal discussion, and the other means mentioned in Step 3 above—depend on the accepted options and room for innovation provided by a particular institution’s culture. As always, think flexibly, no less in promoting your progress than in marketing your potential. The next step may well be full implementation, and the understanding, approval, and, no doubt, patience of staff and stakeholders will be needed for it to proceed as smoothly as possible.

¶ 30 Leave a comment on paragraph 30 0 Step 7: Transition to full implementation if a go decision is reached in Step 5

¶ 31 Leave a comment on paragraph 31 0 Finally, the planning team should give careful consideration to the results of the pilot and to any necessary revisions of the original implementation plan. Even if the pilot was a total success, piloting is an active, adaptive process, and it is likely that unanticipated conditions had to be clarified and glitches overcome prior to the achievement of that success. As a result, slight modifications in organization or procedures before moving to a full implementation are not uncommon (Naslund 2010).