Systems Engineering Strategies for Uncertainty and Complexity


Definition: Uncertainty comprises both external and internal elements. External uncertainty includes changes in the market, the operating environments, business processes, and threats. Other elements include emerging requirements/expectations, changes in priorities, emerging competitors (including users) and introducing new technologies. Internal uncertainties include program/project execution as well as design, implementation, and performance challenges. [1]

Complexity can be characterized as the interactions and interdependencies among people, organizations, technologies, tools, techniques, procedures, and economics that cause the emergence of patterns which transcend any particular goals of any particular group. Complex interactions can result in resilience and robustness but can also result in cascading failures. [2,3]

Keywords: adaptability, agility, complex systems, complexity, ecosystem, emergent behavior, fitness, flows, interactions, interdependency, robustness, selection, uncertainty, variety

MITRE SE Roles & Expectations: MITRE systems engineers are expected to understand the nature and sources of uncertainty, lack of effective control, [4] and complexity [5] in their environment and then select and apply appropriate strategies for managing or mitigating their effects.

Introduction

The complexity we are seeing in the enterprises and systems that MITRE helps engineer requires a spectrum of systems engineering techniques. When a system is bounded with relatively static, well-understood requirements, the classical methods of systems engineering are applicable and powerful. At the other end of the spectrum, when systems are networked and each is individually reacting to technology and mission changes, the environment for any given system becomes essentially unpredictable.

The metaphor of the watchmaker and gardener is sometimes used to describe the differences between engineering in the two types of environments. [6] Classical systems engineering is like watchmaking. Its processes, techniques, and tools are applicable to difficult problems that are essentially deterministic or reductionist in nature. Like gardening, engineering for the enterprise draws on the fundamental principles of evolution, ecology, and adaptation. It uses techniques to increase the likelihood of desirable or favorable outcomes in complex environments that are characterized by uncertainty and that may change in unpredictable ways. Engineering for the enterprise is not a replacement for classical systems engineering. Increasingly, both disciplines must be used in combination to achieve success.

This article begins with a discussion of ecosystems and includes a large number of footnotes and references for the interested reader. This will strike some as an odd place to start. But in many ways the point of view required to understand ecology is analogous to the one needed to comprehend the complex environment in which MITRE systems engineers find themselves today. In fact, a number of the emerging best practices and lessons learned discussed later in this article draw on ecology or evolutionary biology for their inspiration. Lastly, the best practices and lessons learned are organized around important conundrums, needs, or issues that systems engineers face in complex environments.

Because engineering in complex environments is an emerging and rapidly changing field, MITRE systems engineers and others are developing its processes, techniques, and tools as they execute their program responsibilities. As a result, in many cases, there is an inherent uncertainty about the right wisdom to recommend. But pointing them out has value even if we don't yet know exactly the right wisdom to convey about solving them. When it has been possible to suggest at least potential approaches to dealing with a problem, the article does so.

Background

People exist within an ecosystem. We have a sense of what that means and understand the natural world around us as elements in an ecosystem. Our technical systems exist within ecosystems as well. We need to unwrap what it means to be systems engineers within an ecosystem; and, thus, understand the nature and sources of uncertainty, lack of control, and complexity in our environment. [7]

Most people have a keen sense of what a natural ecosystem consists of, and how it morphs over time in response to changes of (and to) its constituent members. Ecosystems are dynamic. Their aggregate state is arrived at through various interactions among the elements present (some connected directly, others indirectly through some transitive path), and how the elements push and pull all the time along many dimensions. Any apparent stability is a dynamic stability which, when one of the interacting elements is altered, changes the stability point of the aggregate—and the changes ripple through the connected pieces (sometimes rather rapidly—and unexpectedly) until another dynamic stability point is found.

Ecosystems are distributed and also have no leader; no one's "in charge." This is nothing that needs to be "fixed"—in fact, it can't be fixed. [8]

All of the systems we work on at MITRE have always existed in this type of ecosystem and have been subject to this type of push-and-pull. But three things have changed:

  1. In the past we have used people as the "impedance matching" element between the artificial elements (traditionally, fully formed, and conceived systems), now artificial elements are connecting to other artificial elements (machine to machine)
  2. The wide potential interconnections we now accept (and demand) among the artificial elements (composition on demand [9])
  3. The engineering we are now expected to perform at large scopes and scales (enterprise engineering).

We now find our systems to be primary elements of the ecosystems in which they reside, rather than augmentations to the primary elements (i.e. the people using them), and we must factor that into our requirements, analyses, and designs. [10] They must be able to respond to changes in the context they find themselves within, rather than relying on people to be the elements that change in response to context changes (i.e., the environment).

Note also that this environment is changing at rapid and unpredictable rates, and in places we didn't necessarily predict. The technology itself is also changing at unprecedented rates. Thus, we are finding that agility is most desired. The systems themselves must be agile; not just the users of the systems. [11,12] Most important, isolation (or attempted isolation) doesn't work.

Having made the argument for variety and interaction, it is important to add the guiding factor: selection. Arbitrary and random change merely leads to chaos. However, the environment guides or channels change by selecting the winners and the losers among those present. Those chosen are said to be "more fit" for the environment. Thus fitness, and its measurement, might be something to pursue. [13]

Given multiple interdependencies, rippling change, an unknown (and possibly unknowable) future, and selection among choices, then, clearly, we can expect uncertainty and therefore agility is a top need. But agility of what?

  • Agility of the aggregate:
    "Systems" and "Systems of Systems" are nothing more than collections of technical elements, people, and processes that perform useful work. It is in the aggregate where this useful work is realized. To be agile, the aggregates must be able to adapt (and possibly, to be assembled) near the point of need, and in a timeframe that allows the potential solution (the aggregate) to be applied to the need in a timely way.
  • Agility of the elements:
    Each element within an aggregate must itself be able to evolve. The rate of its evolution—or its ability to change rapidly with the emergence of a new need—may well define its value to the enterprise. Thus, adaptability becomes a strong design aspect.

It is within this environment, and with these challenges, that MITRE systems engineers are expected to perform. One needs to have this understanding and mindset. It is within this mindset that we can see users as arbiters of "fitness" and the greater stakeholder community as the environment. [14]

Government Interest and Use

The government has a direct interest in seeing that systems built are agile and composable in order to meet the changing ecosystem in which our government customers live. Examples of capabilities MITRE has built this way are in the SEG article Special Considerations for Conditions of Uncertainty: Prototyping and Experimentation. Being agile and composable satisfies the ability to change quickly as conditions, technologies, missions, and procedures change. It also suggests that we may be able to achieve more (re)usability and thus more effectively manage cost. Uncertainty becomes less of a problem if agility is possible. It allows rapid reaction to current conditions rather than prediction of future conditions followed by subsequent reaction/change. Best practices and lessons learned fall along these lines. [15, 16, 17, 18, 19]

Best Practices and Lessons Learned

Technology:

  • Given an unknown future, designing for options: [20]
    Expectations: MITRE systems engineers are expected to consider and recommend the value of building options into designs. They are expected to envision possible system or enterprise extensions in advance, the likelihood of whether and when they would be needed, and the cost of extending the design versus creating a replacement.
    • Partition design by both functionality and time differences of change
      • Traditional design tends to partition primarily by function. However, partitioning also by rate-of-change allows the isolation of elements that change quickly (or might change quickly) from those elements that are more stable and will change slowly. [21]
      • Encapsulate change. A basic tenet of design that has weathered the test of time is to isolate things that change. Key to this is the use of interfaces as a method to isolate the partitions from each other, yet allow interaction.
    • Carefully choose "Bow Ties" [22]
      • Identify and codify in the design, those key decoupling points that divide designs into coherent layers. These should be small in number and ruthlessly enforced. It is the essence of workable architectures. A small number of connection/decoupling points of very low variety (i.e., goal of one) allows high variety on each side of these strategic points.
      • The key decoupling points should use well-known and popular protocols and methods to ensure they have "legs."
    • Building an enterprise element while building a local system: Understanding your offering to the enterprise
      • What does it do (the single thing that provides value to the enterprise)?
      • How do others interact with it?
      • Where/how is it available?
      • How do others find it?
    • Design rules
      • Refactoring for the enterprise
        • Once local elements are discovered and used by the enterprise (i.e., by consumers outside of those originally anticipated by the program originators), refactoring their appearance and presentation to the enterprise is likely warranted. This could mean:
          • Splitting a system into two or more (allowing each part to change at its own rate, or permitting access and interaction to only a piece of the original whole)
          • Substituting one element for another (allowing a new element to perform a role previously provided by another; this allows evolution and change and is the fundamental idea behind interface implementer substitution
          • Augmenting a system with new elements (adding on new elements may allow new roles for the system)
          • Inverting element dependencies to alter business/political considerations (consider the different political/business dynamics resulting from using a design pattern such as subclass/inheritance vice containment/delegation)
      • The actions in the previous bullets have been argued to be design primitives. [23]
    • Flows [24] and their emergence
      • Information flows are the essence of command and control systems. Often, we used defined flows in the past within our designs to decide what elements in a system need to connect together to realize a system's behavior. To achieve agility, however, we need to create designs that allow technical elements to join and leave existing flows dynamically, and which will enable the creation of new flows.

Structure and Organization:

  • Strategies for early and continuous discovery
    Expectations: MITRE systems engineers are expected to consider, recommend, and apply systems engineering strategies such as early prototyping, exploratory integration test-beds, field trials, and experiments to support early and continuous discovery activities in situations in which the required behavior of the deployed system(s) is difficult to predict.
    • Development Networks—mimic the real world as much as possible
      • Providing vetted access to online-available software services that are also found in the fielded system allows third parties to learn about and use aspects of the system-of-record that would otherwise need to be guessed at
      • Third party developers who use the resources available on the development network will require less integration, hand-holding, and rework, thus speeding fielding and holding costs under control.
    • Developmental Spirals
      • Since the future is difficult to predict, using spirals (smaller scope, shorter duration) to sharpen the focus on future requirements lowers uncertainty and risk.
    • Modeling and Simulation
      • People are poor at predicting patterns formed from the interactions of elements (e.g., rules, computing artifacts, etc.). The only way that one may fairly, and without introducing additional bias, elicit patterns (other than the choices and assumptions that go into a model – which should be explicit) is to use modeling and simulation to explore the interactions (be they operational, technical, systemic).
  • Piloting integration strategies
    Expectations: MITRE systems engineers are expected to consider, recommend, and implement integration strategy pilots to explore terminology, operational patterns, technology, and desired features when interoperating systems cross multiple seams and lack a history of effectively working together.
    • Using "technical intimacy"—from casual relations to deep commitment
      • One is most likely to use (and depend upon) an external element when it:
        • already exists
        • is available
        • is likely to remain available
        • is understandable
        • makes small demands on my resources
        • requires few unique agreements
        • appears to be part of the environment
    • Replaceability vs. reusability:
      • Focus on designs that offer the ability to replace elements with other (similar) elements as experience is gained with a system, and/or as requirements change, rather than seeking or designing elements that purport to include all future needs. One can start with small sets of known functionality, then grow it. This lowers risk greatly.
    • Partnerships: [25]
      • Forming partnerships among both consumers and producers of services builds trust. Activity taking place on a development network can provide pointers to potential partnering opportunities that may not have been obvious.

Business and Economic: [26, 27, 28]

  • Reducing uncertainty [29]
    Expectations: MITRE systems engineers are expected to understand the elements that may drive uncertainty in the tasks they're supporting. Uncertainty may come from requirements and/or technologies and MITRE engineers must help customers understand this environment and help mitigate the uncertainty.
    • Where a project is stable in both requirements and technologies, one is able to plan ahead, then execute the plan
    • Where the project is dominated by new or emerging technologies, one should consider a strategy of a "portfolio of small bets"
    • Where a project is dominated by evolving requirements, one should consider a strategy of "staged commitments"
    • A hybrid strategy is needed where all characteristics exist.
  • Reducing uncertainty in cost estimation
    Expectations: MITRE systems engineers are expected to understand the principles underlying good cost estimation and be able to recommend and implement techniques to mitigate cost uncertainty, to include developing design alternatives as bases for cost. MITRE's CASA organization has many methods to help MITRE engineers with cost estimating and associated decision analysis.

    There are two "truths" in conflict: 1) One needs to know what to build before building it, and 2) things always change. Thus, the idea that requirements must be known before building is desired, but the requirements themselves may be changing; so, if things always change, knowing what to build may be fuzzy. But "what to build" needs to be known to estimate well.

    If it's fuzzy, then tighten it up, either in time or scope. Can one define what will be done this year? This month? This week? Find a time slice where this is clear, outcomes are definite, and the method to achieve them is known. Where things becomes fuzzy, this may well be a point where there's a logical branching of possibilities, and a perfect opportunities for "Real Options" [30] to be developed. This is good for interfaces in which details can be deferred.

With respect to estimation:

    • The smaller it is, the easier it is to estimate
    • The simpler it is, the easier it is to estimate
    • The more mature the technology is, the easier it is to estimate
    • The more that is supplied by others, the less needs to be done (i.e., the smaller it is).

There are many approaches one can take for an estimation methodology. They all share one key characteristic: none are able to satisfy all. This goes from agile and lean techniques, [31] which measure team velocity delivering story points, to function points, and the classic SLOC (source line of code) counts. Be very wary of whatever technique is chosen. Don't automatically accept it—always seek supporting AND refuting evidence on the estimates during execution.

    • Establishing baselines:
      • The baselines should be appropriate for the estimation method and the development measurement methods. For example, "done done" in agile methods should be ruthlessly watched. This fits well with defining earned value milestones (EVM). [32] A potential benefit of EVM is that it demands a crisp definition of a milestone, and provides early hints when the cost and schedule assumptions are being violated. This may provide a tool for knowing when to abandon one option and pick another.
    • A hidden problem with service-oriented approaches [33]:
      • Ironically, while service-oriented approaches offer the potential agility and composability desired, the manner in which we contract with developers may erect barriers to realizing the benefits. Consider the situation in which a program offers a service that delivers some of its information bundled in a collection. Suppose further that this is discovered and found useful by many outside the originally planned users and stakeholders. Under these circumstances, one might expect the use of the service to be greatly beyond the planned use. However, there may be transaction densities that exceed the design limits and degrade the performance. Whose problem is this, and how is it mitigated?
    • Contract types:
      • Consider using contract structures for which the default behavior is not continuation. One might do this using an indefinite delivery/indefinite quantity contract with a series of tasks that build on one another, yet where each has a clear ending.
      • Consider using "supplier" models in contrast with "developer" models:
        • Payout based on use.

There are many challenges to working the uncertainty and complexity of MITRE's customer environment. The need to manage these challenges has become more prevalent in the tasks MITRE takes on as we continue to work the strategic and hard problems of our customers and their enterprises. The practices listed can help work this critical area—as more experience is gained by MITRE staff, these practices will evolve as well in our uncertain and complex MITRE world.

References & Resources

  1. As characterized in Stevens, Renee, September 24, 2009, "Acquisition Strategies to Address Uncertainty: Acquisition Research Findings," from her MITRE-Sponsored Research, "Enterprise Systems Acquisition Using Venture Capital Concepts."
  2. Dorner, Dietrich, 1996, The Logic of Failure, Basic Books.
  3. Mitchell, Melanie, 2009, Complexity: A Guided Tour, Oxford.
  4. "Lack of control" includes many conditions and situations, but the most general sense of its use here is the inability to set the desired state, or vector, of an element under one's authority, and which one is expected to exercise control over. The "traditional" approach to ensuring control is isolation of a system—both in development and in use. With more interconnected and interdependent elements and systems, this presumption (and technique) is violated.
  5. "Complex" has become a term of art in engineering and science, and its meaning is slightly different than how it is used in the vernacular. At the risk of oversimplifying, "complicated" as a word means difficult to understand, while "complex" means stable collections and patterns arising from (or emerging) from simple interactions among component pieces. See almost any of the references for more description.
  6. Metzger, L.M., April 27, 2009, "MITRE Systems Engineering: So What's It All About?"
  7. Bar-Yam, Yaneer, 2004, Making Things Work: Solving Complex Problems in a Complex World, Knowledge Press.
  8. Johnson, Steven, 2001, Emergence: The Connected Lives of Ants, Brains, Cities, and Software, Scribner.
  9. Also see the SE Guide article Composable Capabilities on Demand in Engineering Information-Intensive Enterprises
  10. DeRosa, Joseph K., et al, 2008,"The Requirements Conundrum in Complex Systems Engineering," ICSSEA 2008, Paris.
  11. Watts, Duncan, 2003, Six Degrees: The Science of a Connected Age, Norton.
  12. Barabasi, Albert-Laszlo, 2002, Linked: The New Science of Networks, Perseus.
  13. Perhaps only the recognition of fitness as a process is sufficient, and the understanding and management of choices and "choice-spaces" is where we can make our engineering processes reflect fitness. If we were to be able to understand and quantify fitness, it might give us a predictive tool that is currently absent.
  14. There are many efforts that attempt to quantify fitness in engineering systems. In our own community, there are attempts to define measures of effectiveness focusing on operational metrics.
  15. Norman, Douglas O. and Brian E. White, 2008,"Asks the Chief Engineer: "So What Do I Go Do?!," SysCon 2008, IEEE International Systems Conference.
  16. Norman, Douglas O. and Michael Kuras, 2006,"Engineering Complex Systems," Complex Engineered Systems, Springer, Chapter 10.
  17. Norman, Douglas O., 2009, "Engineering Complex Systems: challenges in the theory and practice," Organizing for a Complex World, CSIS Press, Chapter 7.
  18. Friedman, Thomas L., 2006, The World Is Flat, Farrar, Straus, and Giroux.
  19. Ackoff, Russell L., Fred Emery, 2006, On Purposeful Systems, Transaction.
  20. Options provide variety. And, variety is absolutely necessary to promote evolution. This may seem counterintuitive to those who view variety as mere redundancies. It must be recognized that variety (also diversity) requires selection to lead towards effective evolution. Variety is explained nicely by Ross Ashby in "Law of Requisite Variety" in his book Introduction to Cybernetics, 1956, Chapman and Hall, London. Currently out of print, it can be found at here. Also see Diversity Collapse: Causes, Connections, and Consequences
  21. Think about automobiles. If one didn't allow for the removal and replacement of the wheels/tires, then one would need to wait for a different (redesigned) vehicle to operate the vehicle in a different environment—like loose sand rather than asphalt. By recognizing that the wheels/tires can be changed quicker than the vehicle is replaced, one allows change at that point, and the evolution of the whole can occur more rapidly.
  22. Also see the SE Guide articles Architectures Federation and Design Patterns in Engineering Information-Intensive Enterprises
  23. Baldwin, Carliss and Kim Clark, 2000, Design Rules: The Power of Modularity, Vol. 1, MIT Press.
  24. Holland, John, 1995, Hidden Order: How Adaptation Builds Complexity, Perseus.
  25. Moore, James F., 1996, The Death of Competition: Leadership & Strategy in the Age of Business Ecosystems, Harper Business.
  26. Beinhocker, Eric, 2006, The Origin of Wealth: Evolution, Complexity, and the Radical Remaking of Economics, HBS Press.
  27. Wheatley, Margaret J., 1999, Leadership and the New Science: Discovering Order in a Chaotic World, Berrett Koehler.
  28. Also see the SE Guide topic Acquisition Program Planning in Acquisition Systems Engineering
  29. Stevens, Renee, 24 September 2009, "Acquisition Strategies to Address Uncertainty: Acquisition Research Findings," from her MITRE-Sponsored Research "Enterprise Systems Acquisition Using Venture Capital Concepts."
  30. Real Options research at MITRE
  31. Shore, James and Shane Warden, 2008, The Art of Agile Development, O'Reilly.
  32. Fleming, Quentin and Joel Koppelman, 2005, Earned Value Project Management, Third Edition, Project Management Institute.
  33. Martin, James, 1995, The Great Transition, Macon.

Publications

Download the SEG

MITRE's Systems Engineering Guide

Download for EPUB
Download for Amazon Kindle
Download a PDF

Questions?
Contact the SEG Team