About Us Our Work Employment News & Events
MITRE Remote Access for MITRE Staff and Partners Site Map
edge top

March 2001
Volume 2
Number 1

Home > News & Events > MITRE Publications > The Edge >
The Edge Perspectives

Commercial Off-the-Shelf Software – Benefits and Burdens

By Anita King

confused man pictureThe chief information officer’s (CIO) job has never been tougher. Demand for information technology workers is at an all-time high and far exceeds the supply of skilled people. Retention is problematic, with attrition as high as 25-40 percent in many IT companies. Users are demanding ever-increasing quantities of business applications—the pursuit of on-time, on-budget, and bug-free projects remains the only acceptable goal.

How should you approach these application projects? There are many options: buy a commercial off-the-shelf (COTS) product from a vendor and integrate it into your environment, build your own application, or try to minimize your involvement and just use an Application Service Provider (ASP) (a company that offers individuals or enterprises access over the Internet to applications and related services that would otherwise have to be located in their own personal or enterprise computers). Selecting a COTS product seems to be the easy answer. You avoid the lengthy development cycle associated with building an application yourself and the inherent security risks of using an ASP. Why build an application if you can just go out and buy it? It will certainly save time and money. The product will be well tested for you. Integration should be a snap. It solves the users’ problem and it’s bound to be less expensive too!

So, where do you begin? Well, first you have to find products that address your needs. A Web search can yield quite a few leads. Talking to CIOs of similar companies can be helpful too. So can attending a conference dedicated to solving the business problem you are facing. In most cases, within a few days you have a myriad of options from vendors that range from “garage shops” to multibillion dollar companies. The price range and the sufficiently different product features make selection nontrivial. Contacting vendors, gathering product literature, and arranging for product demonstrations takes time. At this point it can seem like it would have been easier to just start writing code.

Next, you have to pick the “right” product from the list of possible solutions. (See article, “Selecting COTS Products or Services.”) As part of that process, identify the technical capabilities that are “must haves” versus those that are “nice to have.” This will help focus discussions with the vendor community and allow you to cut through the marketing hype. It is helpful at this time to try and get user participation in the project.

Unfortunately, in many cases the CIO has excellent software application developers on staff, but they may not have the right system engineering skills to lead a product selection. The first step is to eliminate products that don’t really exist yet (vaporware)—in most cases it is desirable to acquire a product that has been in production for a while. This allows you to assess how dynamic a product’s Application Programming Interface (API) is by reviewing past changes and determining if the vendor has a backward compatibility policy. Next, eliminate products that don’t run on your supported platforms and those that clearly don’t meet the essential users’ requirements, or are out of your price range. This should reduce the number of options to a manageable number.

Now it is time to do some serious work. First, conduct a gap analysis of differences between user procedures and product adaptability. It is reasonable to consider re-engineering processes and eliminating less urgent requirements in order to use a COTS product.

In addition to comparing product functionality, also talk to customers using the product under consideration. Assess how well the product met their needs, as well as how well the vendor performed. Was the vendor responsive to their problems? Was the documentation adequate? Was the product easy to use? If the number of products is down to just a couple, you might even want to bring the products in-house to run them through a scenario-based trial. This allows you to see how well the product works in your environment and how customization may affect your implementation.

Let’s assume that your decision-making process leads to a proper decision. You have selected a product that comes closest to meeting your needs from a vendor you believe will be responsive to you during the integration and deployment phases. Now you need to focus on getting the product installed and integrated with the rest of your information systems. Most sophisticated applications require at least some data exchange with your other application systems. For example, you might need to pull current user data from your human resources system or valid project data from your financial system. This data must be kept synchronized for the entire system to behave properly. The simplest case to deal with is one where the new application requires a “downstream feed” from one or more of your other information systems. In this case, the users interact with the new application and no data needs to be returned. In more complex situations, the applications must operate together, concurrently using the same resources and jointly meeting the performance constraints.

Unfortunately, there is no silver bullet. No two integration efforts are exactly alike. The one thing you can count on is that integration costs both time and money. In many cases internal development may be unavoidable, as custom software is required to “glue” the various COTS components together. This phase of the project requires intense system-engineering skills. Lack of planning and proper risk mitigation (see article “Understanding Risks”) has resulted in many COTS integration efforts being late, over budget, or judged by users to be a failure.

So now you’ve got the product integrated from a technical standpoint. Don’t make the mistake of thinking that you are off the hook when it comes to testing. Vendors are responsible for testing their products, and sometimes they don’t do that very well. You have the responsibility to make sure that the system works as desired. Debugging is made more difficult because each COTS system is in essence a black box. You can only make inferences about the product by observing component behavior. Even with vendor support to address compatibility issues, when things go awry, vendors tend to “blame” each other.

Assume that you made it through the system test phase. It is now time to plan the product rollout and training for the user base. You have had some users involved in both requirements definition and testing, but even so, this phase will present its own challenges. Planning an enterprise rollout is never simple. Unless we are talking about an extremely simple application, users require training. It will be difficult to get them together for training, and in most cases they will have difficulty with the change. No one likes change. No matter how carefully you tested the application, plan on encountering product bugs. Sometimes the product has more functionality than you need, which makes the application seem more complex. Sometimes, under a full user load, product performance declines. At best, your users will likely request some product customization. Only time will tell if the product really suits your users’ needs and business processes.

You’ve got the product installed and tested; your users are trained and they are using the product. Unfortunately, it’s still not clear that your project is a success. You have to deal with the application lifecycle. Sometimes the vendor you pick may go out of business or lose market share or be unresponsive to you as an individual customer. While you were worried about integration, the vendor was busy working on its next release. At some point, maybe very soon after your rollout, you have to deal with a COTS upgrade. If the interfaces are not stable (i.e., written to a robust API), you may have to rewrite them with every upgrade. Upgrades of other products in your information systems environment require you to test all the interfaces. You will have to deal with license renewal and figure out when to replace the product.

So, did the CIO make the right choice in choosing COTS? It’s hard to say. It certainly was not as easy as he thought it was going to be. The cost of integration, licenses, upgrades, customizations, testing, training, and rollout are all variable costs that must be compared to the “custom” alternative for a true picture. In truth, COTS integration efforts are less understood than software development efforts and can fail in more than one phase:

Selection—Did I pick the right product?

Integration—Is the “fit” right? Do I have the skills to get the job done?

User acceptance—Can they get their job done? Is it easier than the alternatives?

Factors that determine how successful integration will be include the following:

Skill in “integrating” vs. development

Ability to test “systems” vs. products

Determining alternatives

Establishing reasonable expectations with regard to cost and schedule

Weighing the benefits versus the burdens when deciding whether or not to make the leap to COTS requires decision makers to make hard choices. Choose wisely.


For more information, please contact Anita King using the employee directory.


Homeland Security Center Center for Enterprise Modernization Command, Control, Communications and Intelligence Center Center for Advanced Aviation System Development

 
 
 

Solutions That Make a Difference.®
Copyright © 1997-2013, The MITRE Corporation. All rights reserved.
MITRE is a registered trademark of The MITRE Corporation.
Material on this site may be copied and distributed with permission only.

IDG's Computerworld Names MITRE a "Best Place to Work in IT" for Eighth Straight Year The Boston Globe Ranks MITRE Number 6 Top Place to Work Fast Company Names MITRE One of the "World's 50 Most Innovative Companies"
 

Privacy Policy | Contact Us