Taste Test: Offering a Vendor Review Service Here's the bottom line. You can propose a vendor review service to your business decision-maker that answers these questions and more. A review involves the use of business and technical skills to check out the vendor's software ingredients and the vendor's chefs. It can improve your organization's ability to digest the software by identifying and mitigating risks pro-actively at the time of package selection. It's a value-added service you can market to your decision-maker that helps them shop wisely. Here are the major steps in a vendor review:

Step One: Plan the Review
Planning a review involves deciding with your customer when and where to conduct the review, what to review, review team staff, vendor notification of your intention to review and preparation.

When to Review
Remember that reviews take time and cost money. You don't need to review every vendor and product you research. Wait until you've narrowed the field to one to three vendors, then review these serious contenders. Decide with your business decision-maker which vendors to review.

What to Review
Talk with your the business decision-maker. Find out what her concerns are about buying a package. Find out what she likes about this particular vendor and software and what puzzles or concerns her then share what you like and what puzzles and concerns you. Agree on what you want to accomplish with a vendor review. For example, there might be two vendors she's considering and she wants to review both. Or she might have settled on one vendor but wants to know how they test customizations. You might recommend specific technical areas to review that she would not request but would value. Choose areas with your decision-maker that fit your situation; don't try to review everything. Here's a high level list of review areas to consider.
(See article sidebar for more details in each of these areas.)

• General Company Soundness

• Data Conversion and Fit

• Technical Quality and Fit

• Management Practices

Notify the Chefs
Now you should notify the vendor of your desire to conduct a review. At this point, you're clear about the objectives for the review and you can communicate these to the vendor. The best person to inform the vendor of your desire to conduct a review is the business decision-maker. The vendor takes the request seriously when it comes from this person rather than an IT person. Marketing people are not used to such thoroughness by potential buyers, so this request might scare them. If they refuse your request to conduct a review, this tells you all you need to know about this vendor. Once the vendor consents to a review, you can work with the vendor to plan the details.

Where to Review
The best place to conduct the review is on site in the vendor's kitchen – on-site in his own development shop. The key people you want to meet are there: project managers, developers, testers, database and customer support staff, technical writers, trainers, etc. Because you recognize you are buying not only the product but also the vendor's technical development shop, you want to meet these people. The artifacts you'd like to see are available there: sample project plans, requirements and design documents, source code, data models, test plans, scripts, defect reports, testing tools and environments, the customer help desk, technical and user documentation, and standards documents. This means your review team will need to travel to the vendor's site. It's amazing what people will show you and tell you when you can meet them face to face. If they can't show you a document you think they should have, that tells you something also. You can see their work with your own eyes and hear what goes on in their shop. You can get a feel for what their company is like by immersing yourself, even for a short time, in their environment. (And if it's a well-run organization with good ideas, take notes! Visiting someone else's shop can expand your own sense of what works and what doesn't.)

Staff the Review Team
Once you agree on the review, you can staff the review team with appropriately skilled people. A facilitator should be appointed to lead the review. This person's responsibilities are to lead the team in planning and preparing for the review, act as the liaison with the vendor before and during the review and oversee the writing of the review report and delivery of the review report to the business decision-maker. The rest of the team should be comprised of people skilled in the areas they are assigned to review. For instance, a quality assurance person would be assigned to meet with the vendor's QA and testing staff and review their test plans, defect reports, test results, etc. Try to keep the team size to a reasonable number (no more than five or six). This will require one person to cover several areas, for instance, a test expert might review user and technical documentation as well.

A business expert user is the best person to review business functionality of the software. I recommend this expert user perform hands-on verification of the software's functions with guidance by the vendor, using the written set of business functions as a checklist. This is not a system test but more of a high-level look for major mismatches. This kind of verification can uncover missing functions. It can also identify functions that the system performs very differently than your company performs them. These kinds of mismatches often dictate where software customization is required. This same business expert is an excellent person to review the soundness of the vendor from a business perspective. This ensures that the review is a partnership between our organization's technical and business concerns.

Prepare for the Review
The facilitator leads the preparation. The review team meets at least once prior to the review to go over their roles, agree on the schedule, discuss the process of conducting the review and decide who will write and deliver the review report. Each team member prepares by reviewing related documents, identifying who they want to meet and preparing questions to ask and coming up with a list of documents to see during the review.

Step Two: Conduct the Review
The review starts with a general company overview for the review team (limit this to one hour). Then the team splits up with each team member meeting with the corresponding vendor staff by review area. During these meetings, the reviewers note on 5x8 cards specific observations about strengths and concerns, one item to a card. They write just enough to remember what the card means to minimize the distraction of note taking. The review itself can take one to three days. By the end of the review, each reviewer will have a stack of 5x8 cards containing his individual observations.

Step Three: Write and Deliver the Review Report
Back at their office, the review team meets to consolidate their cards into an outline for the report. Cards are posted on the wall in groups according to the report outline. They discuss the cards enough that everyone understands and agrees on the strengths and issues collected. They remove duplicate cards and identify the seriousness of each card that contains a concern. At this point, the decision-maker can be given an oral briefing of review results. Designated reviewers write the report based on consolidated strengths and issues and deliver the written report to the business-maker.
The entire process can be completed within one week with concentrated effort. It provides a way for your organization to chew before they swallow a software package. Chewing prepares you to swallow and makes digestion easier. The contents of the report can be used to choose a vendor. Risks identified as show-stoppers should be addressed before signing a contract. The other identified risks form the start of a risk management plan for managing the project.

A Balanced Meal: Technical and Business Success
Vendor reviews are a wonderful technique to taste before you swallow packaged software. They're also a great way to build a partnership with your business decision-makers on packaged software projects, instead of being brought in late or left out completely. It demonstrates your sincere desire to satisfy their business needs in a quality manner. And most importantly, it gives a way for your voice to be heard - and your value added early enough - to be acted upon in a constructive way. If you're a QA manager or lead, try marketing vendor reviews to receptive business decision-makers. If you're a QA technical professional, take this idea to the person in your organization who can make reviews a reality. What have you got to lose? The worse that can happen may be what's already happening. What might you gain? What you want - to be involved early in the selection of software packages. Then you might find yourself grousing in Australian slang where it means excellent, great and wonderful.

Bon Appetit! STQE