By John Wilson

Introduction

Often, the selection of commercial off-the-shelf (COTS) products-or services-in a project is done without considering the factors that can determine the success of the project. It is important to recognize that there are many factors and that they occur throughout the project lifecycle. For instance, a COTS product that meets an initial need for an early application deployment may be poorly suited for future growth requirements. A factor such as this can be further exacerbated if a project involves multiple COTS software tools and several vendors.

Factors that lead to successful COTS usage can be grouped in several areas-products, vendors, existing infrastructure, users, organization, and many others. This article considers specific factors having to do with product and vendor viability.

Product Assessment

Assessment factors that can be examined when selecting a COTS package include functionality, performance, platforms, security, reliability, usability, openness, upgrades, documentation, and cost. Naturally, the weighting of the factors will vary from project to project. Nonetheless, they all should be evaluated when selecting a COTS product to gain insight into risk and cost. A few of the criteria are discussed here to illustrate the considerations in selecting products.

The functionality of the product is an obvious selection factor. In fact, cost and features often drive the decision to purchase a particular product, while other factors are not given ample consideration. The challenge in this factor is that product releases have accelerated during the past several years, particularly with the distribution of products and updates over the Internet. Rarely will a product provide an exact match in functionality to meet project requirements. There will always be missing functionality and/or superfluous functionality. It is important that the selected product provides a close match to the critical needs of the application. To the extent the product requires tailoring or workarounds for your application, this will tend to increase interdependencies with other parts of the system and increase risk on subsequent product releases. The upgrade forecast for the product should also be examined against future project needs.

The full performance considerations of a product are often overlooked. While a product may work well in a lab or experimental environment, it needs to be evaluated in terms of its ability to scale up and meet future growth needs. There are numerous examples of projects that failed because performance parameters were not adequately defined, resulting in the selection of COTS products that were not sufficiently robust for the application demands. This is particularly true in distributed, networked applications, where system performance may have extremely demanding peak loads.

A product's security features can be a critical factor for applications using COTS, particularly for government clients. Many COTS products have been implemented, at least in part, outside the United States. It is common in software product development in the United States to use teams from other countries. To the customer, a COTS product is essentially a black box, in the sense that the implementation of the software is most often hidden. As a result, specific security relating to project requirements may not be guaranteed as the security of the COTS implementation cannot be ascertained- sometimes COTS tools have back-door features or unexpected capabilities. In a project with particular security requirements, COTS may pose an unacceptable risk.

Reliability of a COTS package is a factor that can best be assessed by understanding the range of other client applications and installations, and the vendor's track record in building reliable COTS products. Moreover, the project's reliability requirements need to be assessed with respect to the cost incurred to provide certain levels of reliability. For certain applications, occasional errors and downtime may be acceptable. For other applications, the requirements may specify a Mean Time Between Failures and Mean Time To Repair that are very demanding-resulting in higher project cost. Also, the role of the COTS tool in system reliability must be understood. While overall system reliability may be important, a less reliable COTS tool may be permitted if it is satisfying a seldom-used or low-priority system function. Today, with the rush to bring many products to market, COTS products are notoriously error prone. One must recognize that a new product in a hot market segment will have problems, some potentially crippling to a system's reliability.

The openness of a COTS product is crucial in applications where many COTS products are being integrated. Openness, in general, refers to the ability of a package to interface with other packages, make its data available, and provide a programming interface. The customer must consider the possibilities when integrating COTS products or custom software. Facilities for exchanging data, such as multiple file formats and Extensible Markup Language are important so that different COTS products can integrate and work together in a system. Moreover, the existence of an application programming interface (API) affords system developers flexibility in controlling a COTS product, so that it can best match the system's needs.

Finally, external reviews on products and vendors are often readily available on the Internet and in industry publications. These reviews should be studied to provide additional information in making a selection. It is important to note, however, that many reviews of products are based on quick examinations of the products. Documented experience with the product in a demanding application that requires integration and performance is seldom available in popular reviews. Speaking with experienced users of the product is always the best source of information.

Vendor Assessment

Assessment factors that can be examined when selecting a COTS vendor include organization and background, market position, market volatility, external reviews, partnerships, financial, and access and visibility. Product and vendor assessments often go hand in hand. That is, one would typically not expect a high quality product from a vendor that has financial or organizational troubles. However, in an era with multiple partnerships and rapid product changes, it is prudent to understand the position a vendor has in its market. Moreover, attempting to understand the direction of the market over the coming 12-month period is also important. A vendor may have a solid reputation, but it may be moving its resources from a product you are considering to a new product with a different focus.

The first factors to evaluate are the background and overall strength of the vendor. If the vendor is an established company, then it will have some level of financial and staffing stability. This is important so that the vendor of the product in question will be in business a year from now. Also, it suggests that it has the resources to provide support and continued product development. There are many examples of interesting products introduced by companies with limited operating histories. Under such circumstances, there are many variables that will determine the company's long-term viability, including ability to attract more customers, execute product development activities, anticipate and adapt to its market, respond to actions taken by competitors, and attract and retain key technical personnel. While a product may show great promise, failure to execute in these areas can quickly render a product obsolete.

In many areas, the market volatility will be high because it is a new market. For example, companies that provide business-to-business e-Commerce suites are in a market area that is not well established. In two years, the market will be very different than it is today. In such a market, selecting any vendor is going to be risky. Rapid technological change, frequent new product introductions, changes in customer needs, and evolving industry standards characterize many segments of the COTS industry. One must be attuned to partnerships and acquisitions that make some vendors more viable than others. Mergers and acquisitions can strengthen market positions or cause problems if the perceived benefits of the merger are not achieved as rapidly as expected by industry analysts and investors.

External reviews on products and vendors are often readily available on the Internet and in industry publications. These reviews should be studied to provide additional information in making a selection. It is important to note, however, that many reviews of products are based on quick examinations of the products. Documented experience with the product in a demanding application that requires integration and performance is seldom available in popular reviews.

Often, successfully deploying a COTS tool in a system requires quality support from the vendor. This support continues after the sale and helps the organization using the product to adapt the product usage to changing needs. Difficulty in obtaining technical support can delay a project in which the COTS product plays a crucial role. If access to support is important, the vendor's Web site should be examined for readily available information. The existence of moderated news groups, where personnel from the vendor field questions is another positive indicator. For major purchases, check with other companies that have used the COTS tool in a manner similar to your application. Visibility into the vendor also includes information regarding future product releases. This is important so that you can plan upgrades appropriately.

Conclusion

This article has presented a number of assessment factors associated with selecting COTS products from particular vendors. Each project has a unique set of circumstances that make some factors more important than other factors. It is easy to look only at the obvious factors that drive initial selection-functionality, ease of deployment, and cost. In fact, considerations such as performance, openness, and vendor partnerships often play major roles over the lifecycle of the project, but may not seem important at the outset. Further complicating matters is the fact that most applications today are developed to use multiple COTS tools.

For more information, please contact John Wilson using the employee directory.