Article by Prinicipal Consultant Thomas Hornbæk Svendsen, NNITSelecting a Commercial Off The Shelf (COTS) software product is usually a demanding process. Not only does it require clear objectives and goals, but also a dedicated group of key stakeholders with substantial business process insight. Moreover, to ensure the process is successfully planned and completed, it’s necessary to use a transparent method to document decisions and capture learnings and decisions.The organisation you work for probably has a more or less well-defined process for acquiring COTS software products. Normally, this will include a policy for how to engage in such a process, and a governance-structure for budgeting and decisions. The supporting process-descriptions will vary from organisation to organisation – you might find a detailed and transparent process – or the opposite, that the process is only defined in broad terms, leaving it up to you to define the concept and approach.In case of the latter – how do you get the process right? It’s up to you – perhaps together with a group of colleagues – to ensure that the financial requirements are adequately balanced against the requirements for business process support, that the organisation’s business imperatives are met, and that all relevant stakeholders are adequately involved. Furthermore, if the COTS software product is for a GxP-validated setup, you must ensure that the process contains the steps necessary to uncover whether regulatory requirements can be met.This article outlines 10 critical steps that we advise you follow when screening the market for a COTS software product. The advice applies regardless of whether you’re evaluating the maturity of a number of COTS software products on the market, or whether you are in the actual process of identifying one COTS software product and intend to buy it. The steps are organised into three phases – prepare, conduct and report. Figure 1 provides a summary of the steps involved, and the phases to which they belong:
The prepare phase involves establishing a transparent and controlled process, defining the normative element against which the vendors and products are to be measured, establishing an evaluation model with weighted measurements, and finally identifying the vendors and their products.01: Initiate a transparent and controlled process
Before commencing, it is important to acknowledge that the process of selecting a COTS software product must be transparent and controlled. This means that adequate time must be set aside, a relevant assessment method must be established and agreed upon, and key stakeholders must have adequate time for involvement. It is paramount that the right measurement-points are used in the assessment method, and that the measurementpoints are prioritised according to their relevance. At this stage, it is also critical to clarify whether the implementation must adhere to GxP regulations for example. If you don’t, you risk wasting valuable time with vendors that can’t deliver what you’re looking for. And if you rush the process, you risk conducting unnecessarily and ill-coordinated activities and ultimately selecting the wrong product.
02: Establish the normative element
The normative element is critical to a successful process. Without a clear understanding of what the product must support in terms of business processes and regulations, it is impossible to carry out a vendor and product evaluation that is aligned with business goals.
When evaluating a well-defined product, such as an Enterprise Resource Planning (ERP) system or an Enterprise Content Management (ECM) system, the product definition and the business processes can be used to establish the normative element.
When evaluating a product with no de facto definition, you will have to establish the normative element yourself. As figure 2 illustrates, a number of key processes and relationships must be defined. Once you have established the normative element, use it as a common frame of reference in the evaluation process – this will make it easier to manage the process and communicate the results. You can also use the normative element as a unique vehicle for communicating with the vendors. Make sure that all key stakeholders agree on the defined business processes – eventually these processes will be those supported by the selected product.
03: Get your priorities straight
When planning vendor and product assessments, it is important to understand the priorities. Consider grouping measurement-points into priority-groups, such as business imperatives, financials, industry focus, support for regulatory compliance, reputation/ awareness, business process support and technical architecture. For each priority-group, the measurement-points should be prioritised according to rank, with the most important priorities first. Figure 3 provides an example of business critical areas:
Once the business critical areas have been prioritised, the final assessment calculation will reflect what is most important to the organisation. Failing to use a weighted method will make the assessment imbalanced, as everything will have equal importance to the organisation, which will never be the case. Remember that there might be other areas that are of interest – consider establishing priority-groups for IT, performance and vendor presence.
04: Identify vendors/products
Now it’s time to prepare a list of potential candidates. Discuss the list with key stakeholders and the group that you’re working with. To identify potential candidates, use your network in other organisations to learn about the type of software they use, search for information on the internet, and consider purchasing reports from analyst companies – they might save you a lot of time. Measure the candidates against your priorities and select the ones that look attractive. If the COTS software product must work within a regulated environment, ask for permission to carry out a vendor audit. If the vendor is reluctant to participate, then it should probably be removed from the list. In general, it might be difficult to reduce the list, but remember, the longer the list of candidates, the more work it will take to select the right one. For later reference, it might be a good idea to document why candidates were selected, or deselected for assessment.
The conduct phase requires vendors’ active involvement – they must complete a questionnaire and participate in Conference Room Pilot sessions.
05: Vendor/product questionnaire
Based on the priorities, prepare an initial vendor/ product questionnaire to gain a first impression of the vendor’s behaviour and the product’s capabilities.
As well as an insight into the vendor’s responsiveness and professionalism, the questionnaire can also help reveal information about product references, product strategy and upcoming product releases. This might also be a good time to ask for Quality Management System documentation, a Software Development Lifecycle and audit reports.
Remember to keep in mind what you’re trying to achieve – if you’re conducting an informal vendor and product screening, tell the vendors. If you’re trying to select a product, tell the vendors. Furthermore, to help vendors focus their effort, highlight the key selection criteria. Prepare a document template for the vendors and ask them to submit their responses according to a specific numbering scheme. This makes processing and comparing their replies much easier. Also, define a limit to the number of pages the vendors can submit, such as five pages. At this stage, you will not need all the details – you need to get an impression of the vendors and their products, and information that will help you select the vendors who should participate in the Conference Room Pilots.
Also inform the vendors that their involvement is voluntary and does not obligate your organisation to award a contract, nor to pay cost incurred in the preparation of the responses.
06: Conference Room Pilots
Once you’ve evaluated the questionnaires, you should be in a position to invite qualified vendors to attend a Conference Room Pilot (CRP). Allocate a few days for this process, and make sure that the product presentations are relatively close together. This will make it easier to remember product details and make comparisons. Prepare the material that the vendors must use during the product presentation – and focus on the business processes. CRP’s are a great tool for testing the product’s ability to support your business process requirements, and to get an initial impression of the products ‘look and feel’. And use the CRP’s to get a feeling about the vendor – do they understand the domain, do they understand potential regulatory requirements, are they easy to communicate with?
Prepare an evaluation form for each participant. Make sure that all participants understand the assessment method, and what they’re asked to assess. When the product presentation is finished, facilitate an open discussion with the aim of reaching a common conclusion for each vendor and product. Remember that the participants can have different insights and priorities, which might lead to significant differences between the evaluations (this might indicate that participants have different understandings of a particular measurement point). ’Real’ differences should be noted as additional information which might prove valuable when narrowing down the number of vendors, and when preparing the final recommendation to your management
07: Final evaluation
Consider developing a spreadsheet-based tool to capture the final vendor and product assessments. And to facilitate overview, consider constructing one or more XY-graphs, to illustrate the final assessment of each vendor/product, and the relative position to each other. Also consider using bar-diagrams to summarise assessments in each element, again to facilitate overview. The spreadsheet should also include comments on the assessments to make it easier to remember details and arguments about a particular assessment. Optimally, the spreadsheet should be self-contained.
The report phase involves finalising the report, including the method used and the conclusions agreed upon within your group. The report phase might also include additional work together with the vendor(s) shortlisted.
08: Preparing the report
Although spreadsheets can be very useful, management will typically require a Word or PowerPointbased report. Not surprisingly, the report should be organised into an executive summary and a number of detailed chapters. You probably have a variety of supporting documentation that you have received from the vendors during the assessment period, or documentation that you and your group have prepared during the process.
Add these documents as appendices so that they are available for stakeholders who require detailed information. Collect all observations and findings and include it as reference material, rather than scattered around on file shares. In a few years, only this approach will facilitate overview. And remember to include a description of the method and the objectives.
09: Get close to a few vendors – or even just one
Once the initial screening is complete, you should have a few – or perhaps only one – vendor to continue working with. It is now time to change the approach from observant to hands-on. The only way you can test the product’s ability to succeed in real life is to make it available for end users. A common mistake is to let the vendor install the product in a sandbox-environment, and then encourage a team of selected users to work with the system when they have the time. Typically, users will never find the time, and they will end up checking out the system the day before the final assessment is due.
Instead, you should revisit the list of business processes used for the initial assessment, and carefully check that the product supports them. If relevant, consider asking a user with extensive regulatory understanding to test compliance features, such as how the product supports the electronic signature process and makes audit trail entries available for review.
Remember to establish a well-defined method for the assessment so that pilot users can work within a common and understandable framework. And make sure that the pilot users understand the business processes in detail – the new employee who happens to have the time is probably not the right person for the job.
10: Let the evaluation method inspire the contract, user requirements specification and project follow-up
Eventually – but not always – the product assessment ends in concrete negotiations with a selected vendor about a particular product. Remember this when working with the assessment material. Done wisely, the documentation can act as valuable input when preparing the contract negotiations, and planning the product implementation. The process descriptions could be used as an early version of the actual requirements specification, and technical details might be valuable input to documents, such as the functional and technical design specifications. Often, governance structures require that Key Performance Indicators (KPI’s) for benefit-realisation are defined. Keep this in mind when defining what the vendor and product should help you realise in terms of business goals. Done properly, this will be a valuable guiding light throughout the process, and help you create transparency.