David G. Ullman E-mailUllman
Department of Mechanical Engineering
Oregon State University
Bruce D'Ambrosio
Department of Computer Science
Oregon State University
ABSTRACT
The design of even the simplest product requires thousands of decisions. Yet very few of these
decisions are supported with methods on paper or on computers. Is this because engineering
design decisions don't need support or is it because techniques have yet to be developed that are
usable on a wide basis? In considering this question a wide range of decision problem
characteristics need to be addressed. In engineering design some decisions are made by
individuals, others by teams - some are about the product and others about the processes that
support the product - some are based on complete, consistent, quantitative data and others on
sparse, conflicting, qualitative discussions. In order to address the reasons why so little support
is used and the characteristics of potentially useful decision support tools, a taxonomy of decision
characteristics is proposed(1). This taxonomy is used to classify current techniques and to define
the requirements for an ideal engineering design decision support system.
1. INTRODUCTION
There are thousands of decisions made in every design project. Many are made by individuals,
others made by teams or committees. In any case, few are supported by formal methods or tools.
This leads to one of two conclusions: Either there is no need for decision support (designers and
design teams do very well without aid), or tools and methods have not yet been developed that
give adequate decision support for the wide range of problems faced. Observation and limited
results of formal studies lead to the conclusion that individual designers and, more importantly,
teams of designers take too long to make decisions, frequently rehashing previous information,
drop issues and fail to document rationale for later use. Thus, there is a need to explore why
current decision support systems are not more frequently used by designers. In this research, a
classification scheme is developed to assist in understanding the existing methods and tools, and
to define specifications for future decision support. To further understand the rationale for this
paper, consider the following example.
The dialog below is a short segment of a design team meeting in which the discussion centers on
selecting a material to use to manufacture a part. This problem is taken from actual practice and
has been shortened and refined for this paper. It is typical of conversations that occur during the
design of even the simplest product. Here Dave is the mechanical designer, Bill is a
manufacturing engineer and Lucy is a sales/marketing representative. Lucy, as the manager of
this project, needs to decide what to do.
Dave: O.K. Our next problem has to do with the strike plate. This plate has to take the repeated impact of the latching mechanism. Currently I have it as part of the front panel which is made of 6065-T6 aluminum.
Bill: Isn't the impact going to really mess up the surface of this plate?
Dave: That's what I'm afraid of. So is Lucy.
Lucy: I told Dave that our customers wouldn't think much of our product if it looks like trash after only a short time in service. But, Dave tells me that it will add cost if the front panel and strike plate are two different parts. That's why we are having this meeting. Any ideas?
......Silence.....
Dave: Well.... here are the options as I see them: I either put some coating on the plate, replace the plate with a harder material or redesign the whole interface.
Bill: That sounds like a pretty complete list. I can be of help on the first two options. We can put a polyethylene, polypropolyene or nylon coat on the surface. As for replacing the Aluminum, I would suggest stainless steel.
Dave: Do you think we can get by with just a plastic coat on the surface? That sounds too easy.
Bill: I'm not sure, but I did it once before on a similar product and it worked well. A test piece could be made up and cycle tested.
Dave: Seems too soft to me?
Lucy: What about the stainless steel plate idea?
Bill: It should be hard enough to wear well.
Dave: I agree, but then I will have to find a way of connecting the plate to the rest of the aluminum panel. This presents some new problems such as transmitting the forces during impact and consideration for assembly. I think I should look at eliminating the plate by avoiding the impact.
Lucy: Can you do that?
Dave: I think so. I need to go back to some earlier decisions and see why we ended up with this problem in the first place. I am confident that, if given some time, I can avoid the problem altogether.
Lucy: Well guys, what do we do next to resolve this problem?
There are many important points to notice about this simple dialog and they form the basis for a
taxonomy that is refined throughout the remainder of the paper. The taxonomy's key
classifications are in bold text. First, the problem that the team is trying to solve has a
describable structure. It is characterized by completeness and quality of information describing
the design space (i.e. alternatives and criteria for judging them), and the details on how a choice
is made. Second, the problem focus changes from the product being designed to consideration of
the tasks (i.e. the process) needed to support design decisions. Third, the resolution of this issue
has a range of effects. A decision made on this issue may affect future issues and require
reconsideration of decisions made past issues. Fourth and finally, Lucy is seeking support for
deciding "what do we do next." The support she seeks can range from simply a good
representation of the problem at hand to a full decision analysis with clear guidance as to what to
do next. The four areas in bold will be refined and supported in Section 3, and used to classify
the example in Section 4 and common decision support methods in Section 5. First, a simple
decision problem model is developed as a basis for discussion.
2. THE STRUCTURE OF DECISION PROBLEMS
There are many types of problems solved during the design process. In reviewing the literature
for this paper the authors found publications on decision making in five different sections of the
library: operations research, optimization, artificial intelligence, psychology and business.
Further, there are over 160 computer based decision support systems on the market [2]. In order
to develop a common vocabulary which covers all these disciplines and tools is challenging.
Since this paper is focused on those requiring a decision about which alternative to choose. A
model based on IBIS (see Section 5) has been chosen. The terms in italics are key words based
on the IBIS model and used throughout this paper.
1. In the example a number of interdependent design issues are discussed. A Design Issue is any
question, problem or task identified by designers that needs to be resolved. The dialog begins
with a potential problem with the product - "the strike plate... has to take the repeated impact of
the latching mechanism" - and ends with a process or planning issue - "what do we do next." An
issue may be the need to fulfill design requirements (i.e. specifications or constraints), the need
to establish a value for some design object parameter, or the desire to determine the next part of
the design problem to work on. Some important points about issues are: (1) they are often
dependent on the resolution of other issues; (2) sub-issues arise (e.g. cycle testing the plastic
coating); and (3) they can be focused on the product (the object being designed) or on the design
process (e.g. what to do next) and these two types are often interwoven with each other as in the
example above.
The same issue, revisited after gathering more information, may yield a different decision. In the
example, an earlier set of decisions resulted in the current configuration. David even mentions
going "back to some earlier decisions" and the resolution of the current issue may alter or even
nullify some of these earlier decisions. Further, a process decision to reconvene in a couple of
days after more information about some of the alternatives and criteria is gathered will probably
be made. A refined decision can then be made at the next meeting. Thus, decision making is a
dynamic activity.
Each issue has a set of criteria (i.e. constraints, specifications, requirements, needs) for
evaluating alternatives that are potential solutions. These criteria may be a well developed set of
equality and inequality constraints. They may be refined requirements developed through a tool
like Quality Function Deployment. They may be ad hoc (developed during the discussion as
above) or they may be totally unstated.
2. For each issue a number of alternative solutions are proposed. An alternative (i.e. proposal,
idea) is an option generated by an individual or a team for the resolution of a particular design
issue. Alternatives range from all the possible values for a variable(s) to a set of discrete options
having no apparent common variables. Some important points about alternatives are: (1) they
are developed by various members of the design team; (2) they are often incompletely described
and/or understood; (3) they are often dependent on the resolution of other issues or alternatives;
(4) they will often introduce new sub-issues needed to develop sufficient information to evaluate
or refine it; (5) they may be inconsistent with other alternatives used to solve another issue; (6)
they may be at different levels of abstraction (e.g. some are vague ideas and others refined values
for a component) and in a different language (e.g. text, sketches, physical objects). In the
example Bill knows a lot about plastic coatings. This alternative is well known, but it is
dependent on a new issue - develop and cycle test a sample - proposed by Bill to improve his
knowledge. At another point Dave has only a rough, incomplete idea about the alternative for
eliminating the front plate. To learn more about this alternative Dave will need to return to the
earlier issues to reassess them and may need to address totally new issues.
Alternatives and criteria describe the decision space or, in terms of decision theory, the world. In
the decision support methods described in Section 5, the decision space ranges from a matrix of
the discrete alternatives versus the criteria for their evaluation (e.g. utility theory and Pugh's
method), to the continuous area of potential variable values bounded by constraints (e.g. formal
optimization).
3. In resolving an issue, arguments for and against the alternatives are given. Arguments
require that alternatives be evaluated against the criteria. Evaluation requires comparisons that
identify the relative merits or demerits of an alternative. These comparisons are made either
between alternatives relative to a specific criterion or between an alternative and a criterion (see
page 169 in [18] for a discussion). Some important points about arguments are: (1) they are
made with varying levels of confidence, knowledge and abstraction; (2) they are posted by
various members of the design team; (3) they may be inconsistent with each other; and (4) they
will often introduce new sub-issues needed to develop sufficient information to complete the
evaluation. In the example, some arguments given by team members were inconsistent (Dave
and Bill did not agree on the plastic coat). Some arguments were incomplete (not everyone
evaluated every alternative relative to every criterion) and they were made with various levels of
team confidence. To refine the arguments sufficiently to support making a decision, Lucy must
allocate scarce resources. Part of her decision is how much time and expertise to devote to
increasing the completeness, confidence and consistency of the arguments.
The argument structure gives the rationale [19] for either supporting or opposing a particular
alternative. This structure has two main parts, preference and belief, which are developed in the
Section 3.
4. The decision on "what to do next" is a process decision based on information about both the
product and the process. A Decision weighs the arguments supporting or opposing the
alternatives for a particular issue. The effect of the decision is to either accept, reject, or modify
each alternative, or to leave the issue pending while more information is gathered. The history of
the design process is the chronological sequence of decisions that relate progress on the design
project [8][9].
This model of the decision making, based on issues, criteria, alternatives, arguments and
decisions helps in understanding the example dialog. It fits the structure of engineering decision
making as shown in a number of studies [1][8][9][14][21][22]. Here it serves as a basic
vocabulary for a taxonomy that is useful for classifying decision making problems and tools.
3. A TAXONOMY OF ENGINEERING DESIGN DECISION PROBLEMS
Based on the points raised above, Table 1 shows a taxonomy that can be used to classify types of
problems, and the tools and techniques used to solve them.
|
STRUCTURE |
DECISION SPACE |
1. PROBLEM COMPLETENESS |
| 2. ABSTRACTION LEVEL | ||
| 3. DETERMINISM | ||
|
PREFERENCE MODEL |
4. OBJECTIVE FUNCTION | |
| 5. CONSISTENCY | ||
| 6. COMPARISON BASIS | ||
|
BELIEF MODEL |
7. DIMENSION | |
| 8. BELIEF COMPLETENESS | ||
| FOCUS | 9. PROBLEM FOCUS | |
| RANGE | 10. RANGE OF INDEPENDENCE | |
| SUPPORT | 11. LEVEL OF SUPPORT | |
Table 1. Decision problem taxonomy
The four key classifications, that were introduced after the example dialog, form the basis of this
taxonomy. The structure of the decision problem is further decomposed into three sub-classes.
The decision space is the set of all possible outcomes for the problem and the limits on them.
This world of possibilities is defined by the alternatives (i.e. the variables) that are the potential
solutions for the problem and the criteria or constraints that measure or limit the solution. In all
decision problems these two pieces of information are central to the problem definition.
The preference model and the belief model characterize the structure of the arguments used to
evaluate the alternatives. The preference model represents the goal of the problem or value of
each potential outcome. In other words, preference is a model of what the decision maker(s) care
about. This is often quantified by an objective, cost or utility function. The belief model is a
representation of the likelihood of success for a particular solution to the problem. The belief
model expresses what the decision maker(s) knows or accepts to be the basis for making the
decision. This can be measured by the knowledge about the alternatives and the confidence in
them satisfying the criteria.
The total taxonomy is composed of eleven numbered measures (see Table 1). Discussion of each
of these includes an itemization of the possible values for each measure (underlined text) and an
example from the dialog. Common decision support methods will be classified using these
measures in Section 5 of this paper.
Decision Space Structure: 1. Problem Completeness
In the decision space, information describing the alternatives and criteria may be complete or
incomplete. If all the alternatives are known and all the criteria for evaluation can be itemized
(i.e. fixed), then the problem is considered complete. In the example problem the alternatives
and the criteria for their evaluation evolve as the discussion progresses. There is no confirmation
that either is complete. The problem is open to new alternatives and criteria, as is the case in
many design problems. In contrast however, most of the formalized decision support methods
require complete information.
Decision Space Structure: 2. Abstraction
In the decision space, the alternatives and criteria may be refined or abstract. Generally, refined
information is quantitative, whereas abstract information is qualitative. In some decision
problems, the Abstraction level of the information is mixed. It is well recognized that much
early design information is qualitative and that the final manufacturing specifications are
quantitative. In the example, the team's discussion is qualitative. Even though quantitative
information exists about the hardness of materials, Dave and Bill do not bring it into the
discussion. Rather, they use terms like "too soft" as appropriate. As the issue is further studied,
quantitative information may become important (e.g. the results of Bill's cycle test).
Decision Space Structure: 3. Determinism
The information used in the problem may be deterministic or distributed. While most analysis is performed on point-valued variables it has long been recognized that all design parameters are actually distributions [5][12]. Often the exact distribution of information is unknown.
Preference Model Structure: 4. Objective Function
A key feature of all decision making methods is a mechanism for evaluating how well
alternatives meets the criteria. This evaluation is quantified through an objective, utility or cost
function, a measure of preference of one alternative relative to others. In many problems the
objective function is based on the values for parameters that characterize the alternatives and the
desire to maximize, minimize or find an internal optimum point for some parametric
characteristic as in formal optimization. In other problems, the objective function is a weighted
preference of criteria that call for judgement or satisficing rather than analysis. In the example,
there is clearly no parameter being optimized, rather the team's decision will be based on their
judgement of the information available.
Preference Model Structure: 5. Consistency
When a team of designers is making design problem decisions, there may be many different
viewpoints on the importance of criteria and the evaluation of alternatives, thus, preference
models may vary within the team. If the problem is being solved by a single individual, a team
with a unified view, or if a methodology can only represent a single viewpoint then the decision
making problem is considered consistent. If, as in our example, there are conflicting viewpoints,
then the preference model is inconsistent.
Preference Model Structure: 6. Comparison Basis
The comparison basis is a measure of the type of comparison made in the evaluation of an
alternative. An alternative is either compared absolutely to a criterion or relatively to another
alternative. Relative comparisons are made along dimensions set by criteria. In the example,
both types of comparisons are made. Dave, evaluating the concept of a plastic coat on the
surface, makes an absolute comparison to a hardness criteria. If the issue is further refined, the
choice between polyethylene, polypropolyene and nylon will be a relative comparison among the
hardness of the three materials plus a comparison along other measures set by other criteria.
Belief Model: 7. Dimension
Some decision support methods allow the decision maker to express his/her belief or certainty in
the information within the decision space. These methods generally use probability to quantify
knowledge of the alternatives and criteria. Probability is also used to quantify confidence in an
alternative's ability to meet the criteria. Knowledge is a measure of how much the evaluator
knows about the alternative/criteria space. During design activities knowledge is generally
increased by building prototypes, doing simulations (analytical and physical) or finding
additional sources of information (e.g. books, experts, consultants). Confidence is a measure of
how well the evaluator believes the alternative actually meets the criteria. Decision problems or
methods are classed by the number of dimensions of information involved: none; one
(confidence or knowledge); or two (knowledge and confidence). In the example, Bill's
knowledge of plastic coatings is greater than Dave's and his confidence that this alternative meets
the criteria is higher. In this case the team should have some confidence that plastics are a
feasible option. Additionally, in the discussion, both Bill and Dave propose tasks to improve
their knowledge about the alternatives. Dave's confidence in the alternative for "eliminating the
plate by avoiding the impact" is tempered by his lack of knowledge about the idea. This lack is
expressed in his response to Lucy's query with "I think so."
Belief Model: 8. Belief Completeness
In most engineering decision making problems all the alternatives are not evaluated against all
the criteria by everyone on the design team. This is especially true if the team is multi-disciplinary or if some team members have little knowledge about some of the criteria. Where
item 1 above measured the completeness of the information representing the problem, this
measure focuses on the completeness of the team evaluation of the information. In the example,
if everyone on the team had given an evaluation of each alternative relative to each criteria then
the belief model would be complete. Lucy gives no evaluation of the plastic coat idea, amongst
others, and so the belief model is incomplete.
Focus: 9. Problem Focus
The focus of each design issue is either on the product being designed or on one of the processes
that support the design, manufacture or distribution of the product. The issue addressed in the
dialog is focused on the product. A planning issue evolves during the dialog.
Range: 10. Range of Issue Independence
Each design issue may be independent or dependent on other issues. The level of dependence
affects the number of times an issue is reconsidered. There are three types of dependency
considered here:
Type I: Independent Issues. A Type I decision is based on a single, independent issue.
Arguments and alternatives related to the issue are based on unchanging information (i.e. a
decision on an issue is made with constant (i.e. static) information). The implicit underlying
model is that decision making proceeds monotonically. Each issue is resolved with a decision
that is never retracted. Dependent issues must wait for decisions to be made in parent issues.
Type I decisions can only occur in systems that can be decomposed into independent issues. This
is an ideal situation and not usually the case in engineering decision making.
Type II: Dependent Issues. An issue in a Type II system is dependent on the resolution of other issues. Information about arguments and alternatives are shared by more than one issue. In order to reach a decision, an issue may be suspended while other issues are addressed. Additionally, issue dependence is important if decisions are retracted or changed as these changes affect other, dependent issues. Issue interdependence and retraction of previously made decisions are characteristic of most engineering design situations and is evident in the example: Dave faces the possibility of returning to an earlier decision because of a new criteria to minimize face plate marring. Lucy faces the problem that if the decision is made to put more effort into one issue there will be less resources available for work on other issues.
In reaching decisions for Type II situations, the alternatives for each issue and the dependency of
arguments on those alternatives must be tracked. This is difficult for all but the simplest
problem. Often, during the design process, this information is lost or the problem artificially
simplified to make the problem Type I.
Type III: Interdependent Issues. Type III decision making concerns the composition and
decomposition of issues. Where Type I and Type II decisions are limited in their range to a
single issue, and simple "linear" relationships between issues, Type III focuses on the
decompositions of issues into sub-issues and the subsequent composition of information derived
in the sub-issues back into the higher level issue. In the example, the alternatives spawned new
sub-issues. As the problem progresses, alternate sets of sub-issues may arise. The resolution of
these sub-issues affects the resolution of the original issue. For Type III decisions, the
representation of the design process must be rich enough to not only share data about the design
object as in Type II tools, but to also represent the interdependencies of the issues. Specifically,
the representation must relate how the resolution of an issue affects super-issues and how
sub-issues are generated to resolve it. Another key feature of Type III decision making is that the
issue decomposition is itself evolving as the design evolves, an extremely dynamic situation.
Most design work is of Type III. In a video taped study of a design session the issues associated
with the development of a single physical part was explored [20]. Nineteen issues were visited
70 times, or 3.7 visits per issue. Each visit included new information which affected the
alternatives considered, the criteria used and the decision made. The new information was a
result of decisions made during the resolution of sub-issues and co-dependent issues.
Support: 11. Level of Support
The level of support offered to the decision makers can be classified as either: representation,
outcome determination, or decision analysis.
Level 1: Representation. The minimal amount of information to support decision making is the
representation of the issues, alternatives, arguments and constraints on which the decision is
based.
Level 2: Outcome Determination. In engineering, many decisions are made based on the result
of analysis or simulation. In order to compare an alternative to a criteria the behavior must be
determined either by mathematical analysis, development of a physical test or some other
simulation of system. Thus, outcome determination gives information to support the evaluation
needed to make a decision. Some decision support tools within commercial CAD programs are
analysis tools with a strong user interface allowing rapid change of variable values (i.e.
alternatives) so that iterative evaluations can be made easily.
Level 3: Decision Analysis. This level of support determines which alternative to choose in
order to satisfy an issue. This is an extension of Level 2 in that information is needed about
design preferences in order to make the decision. Level 3 tools are the most knowledge
intensive. In order to actually make such a recommendation, the tool must not only know the
distribution of possible outcomes, but also their utility.
4. CLASSIFYING THE EXAMPLE DECISION PROBLEM
The classification outlined in the section above can be applied to both design problems and to
decision support tools. In this section, as an example, the dialog in the introduction is measured
using the categories listed in Table 1. Table 2 shows the eleven major measures of the taxonomy
in the first column and values for classifying the example in the second column. In this table, as
in others that follow, values that are shaded are discussed.
| DECISION SPACE | 1. PROBLEM COMPLETENESS | Incomplete |
| 2. ABSTRACTION LEVEL | Qualitative | |
| 3. DETERMINISM | Undefined | |
| PREFERENCE MODEL | 4. OBJECTIVE FUNCTION | Judgement |
| 5. CONSISTENCY | Inconsistent | |
| 6. COMPARISON BASIS | Absolute | |
| BELIEF
MODEL |
7. DIMENSION | 2; Knowledge and Confidence |
| 8. BELIEF COMPLETENESS | Incomplete | |
| 9. PROBLEM FOCUS | Product | |
| 10. RANGE OF INDEPENDENCE | Type III:: Interdependent | |
| 11. LEVEL OF SUPPORT | Level 3: Decision Analysis |
Table 2. Example problem
The issue in the example is how to reduce the potential damage of the latching mechanism as it
impacts the strike plate. The alternatives and criteria for the team's evaluation evolve as the
dialog unfolds. Since more alternatives and criteria may evolve, the decision space is
incomplete. The discussion in the dialog is only about qualitative information. It is so abstract
that is not clearly deterministic nor stochastic and is thus rated as undefined. The objective
function is based on judgement. Since it is so informal, it is not clear if the basis of judgement is
consistent and so it is assumed inconsistent. Reading the dialog shows only absolute comparison
to the criteria. The belief model is not articulated well, yet it is clear that the team members
express both
knowledge and confidence on their evaluation of some of the alternatives. Lucy gives no
arguments and Dave and Bill give incomplete arguments on most alternatives. The dialog,
although focused on one issue, develops other interdependent issues. Finally, Lucy wants to
know "what do we do next" - a request for a decision analysis.
Certainly not all decisions need the level of support demonstrated in this example. As is shown
in the next section there are no tools currently available that can support Lucy in making a
decision on this problem. However, there are many that are useful in more limited situations.
5. CLASSIFYING EXISTING DECISION SUPPORT SYSTEMS
There are many types of decision support tools available for use in engineering design. In this
section a few of these tools will be classified using the taxonomy. This exercise will add insight
into the usefulness and limitations of the various tools.
IBIS: The IBIS representation of decision making has been shown to be a good model of design problem solving [1][8][9]. IBIS allows the representation of the interdependent nature of issues, alternatives and arguments regardless of the type of data, its completeness or its consistency. In a software development project at NCR Corporation, researchers used IBIS based tools to represent the organization of issues, alternatives, arguments and decisions on a computer [21][22]. The entire project addressed 2260 issues and is by far the largest study of its type. Some of the results from this study are:
* IBIS provided a shared memory for the design team. The history of decisions made and recorded in the tools was easily reviewed and utilized by members of the design team and by management.
* IBIS helped the team detect design issues that had "fallen through the cracks." It was estimated the resulting savings were 3-6 times greater than the cost of using the tools.
* IBIS helped the team to more quickly understand the problem they were trying to solve.
* IBIS helped structure the information (issues, positions, and arguments) and helped establish and focus the agenda. Team meetings seemed more productive.
* IBIS supported communication with other levels of the organization. People not at a design team meeting easily discerned what was discussed -- not just what the final decision was.
The IBIS model allows for the representation of incomplete, dynamic information about
interdependent issues associated with either the product or the process. Information is expressed
informally, so that the design space can include quantitative or qualitative as well as
deterministic or distributed data in support of the product or process. There is no formal
structure to support belief or preference, however, these can be expressed informally as part of an
argument. Although IBIS can model complex decision making information, it offers no
automated support beyond the representation. This classification is shown in Table 3.
| DECISION SPACE | 1. PROBLEM COMPLETENESS | Incomplete |
| 2. ABSTRACTION LEVEL | Quantitative or qualitative | |
| 3. DETERMINISM | Deterministic or distributed | |
| PREFERENCE MODEL | 4. OBJECTIVE FUNCTION | None |
| 5. CONSISTENCY | N/A | |
| 6. COMPARISON BASIS | N/A | |
| BELIEF
MODEL |
7. DIMENSION | None |
| 8. BELIEF COMPLETENESS | N/A | |
| 9. PROBLEM FOCUS | Product or Process | |
| 10. RANGE OF INDEPENDENCE | Type III: Interdependent | |
| 11. LEVEL OF SUPPORT | Level 1: Representation |
Table 3. IBIS
Probabilistic Design: Most engineering design analysis is deterministic. A single value
representing the mean is calculated as part of the evaluation process. An extension of this
traditional analysis is probabilistic design [5][12], a well refined technique for statistically
analyzing the design of machine components. In this method, the equations that relate the
variables (e.g. dimensions and material properties) in a problem deterministically are extended to
relate their true, uncertain nature. Thus, each variable is represented by its probabilistic
distribution.
As shown in Table 4, this method requires a complete and quantitative representation of the
information in the decision space. The information represents the distribution of the variables
that characterize the product. The distribution represents a form of confidence in the values of the
variables. This does not support any form of preference model. The method is generally applied
to the outcome determination of independent issues.
| DECISION SPACE | 1. PROBLEM COMPLETENESS | Complete |
| 2. ABSTRACTION LEVEL | Quantitative | |
| 3. DETERMINISM | Distributed | |
| PREFERENCE MODEL | 4. OBJECTIVE FUNCTION | None |
| 5. CONSISTENCY | N/A | |
| 6. COMPARISON BASIS | N/A | |
| BELIEF
MODEL |
7. DIMENSION | Confidence |
| 8. BELIEF COMPLETENESS | N/A | |
| 9. PROBLEM FOCUS | Product | |
| 10. RANGE OF INDEPENDENCE | Type I: Independent | |
| 11. LEVEL OF SUPPORT | Level 2: Outcome determination |
Table 4. Probabilistic design
Formal Optimization: Formal optimization requires the design problem (i.e. the issue) to be
expressed as a cost, objective or utility function to be minimized. Alternatives, characterized as
sets of values satisfying this function must meet equality and inequality constraints (i.e. criteria).
Arguments are given by the search algorithm and the decision is reaching the maximum or
minimum value. This type of decision support tool, for both single issue and coupled systems, is
classified in Table 5.
In design, most optimization is focused on variables that represent the behavior of the product
being designed. These variables are quantitative and generally deterministic. In order to develop
the cost function, a complete model of the system is necessary. The preference model searches
for an optimum based on the absolute comparison of the variable values to the criteria. There
must be a consistent view of this information and it must represent a single independent problem.
There is no mechanism to represent belief in this method. Finally, the goal of optimization is to
find the extreme value for the utility function, ergo there is decision analysis support.
Sobieski [13] and others have extended optimization across several interdependent issues. This
type of optimization allows the decisions to be made on a number of dependent issues. For
example, to optimize aero-space systems such as those with structures which need minimum
weight, maximum strength and specific aerodynamic characteristics - three dependent issues each
characterized by its own utility function must be satisfied.
| DECISION SPACE | 1. PROBLEM COMPLETENESS | Complete |
| 2. ABSTRACTION LEVEL | Quantitative | |
| 3. DETERMINISM | Deterministic (generally) | |
| PREFERENCE MODEL | 4. OBJECTIVE FUNCTION | Optimum |
| 5. CONSISTENCY | Consistent | |
| 6. COMPARISON BASIS | Absolute | |
| BELIEF
MODEL |
7. DIMENSION | None |
| 8. BELIEF COMPLETENESS | N/A | |
| 9. PROBLEM FOCUS | Product | |
| 10. RANGE OF INDEPENDENCE | Type I:: Independent
Type II: Dependent | |
| 11. LEVEL OF SUPPORT | Level 2: Decision Analysis |
Table 5. Formal optimization
CPM/PERT: CPM/PERT methods are used to order a set of known tasks (i.e., issues). For
each task, information about its dependency and precedence (i.e. constraints) is known with
certainty and consistency. In CPM the alternatives are simply the various ordering of the issues.
A similar method, called the precedence matrix method, exists for ordering sets of equations that
represent the function of a device. Both of these techniques are fully developed in Steward [16].
Additionally, the CPM method is often extended through the use of PERT, which adds
uncertainty to the CPM method by assuming a beta distribution for time estimates. The taxonomy
in Table 6 covers all these methods.
| DECISION SPACE | 1. PROBLEM COMPLETENESS | Complete |
| 2. ABSTRACTION LEVEL | Quantitative | |
| 3. DETERMINISM | Deterministic or Distributed | |
| PREFERENCE MODEL | 4. OBJECTIVE FUNCTION | Optimum |
| 5. CONSISTENCY | Consistent | |
| 6. COMPARISON BASIS | Absolute | |
| BELIEF
MODEL |
7. DIMENSION | None |
| 8. BELIEF COMPLETENESS | N/A | |
| 9. PROBLEM FOCUS | Process (generally) | |
| 10. RANGE OF INDEPENDENCE | Type II: Dependent | |
| 11. LEVEL OF SUPPORT | Level 3: Decision Analysis |
Table 6. CPM/PERT
CPM/PERT methods require complete, quantitative information in the decision space. Whereas
most methods of this type require deterministic information, PERT supports distributions. In all
cases, the goal is an optimum based on consistent data and is compared absolutely to the criteria
constraining the problem. There is no belief support system for these methods. They usually
support decision analysis of dependent issues (i.e. tasks) representing a process.
Decision Support Systems and Decision Trees: Decision Support Systems (DSS) and
decision trees are widely used in business. DSSs are the outgrowth of management information
systems (MIS) and are aimed at modeling the business processes. According to Hopple [6], they
are "therefore targeted at top managers and executive decision-makers." There are many
different DSS systems available and their development is still immature. Hopple lists 9 major
families and over 50 different methods used in modeling. The literature contains very little
information about DSSs for aiding decision makers dealing with ill-structured design problems.
Little is known about ill-structured problems and so far computer decision aids have not been any
more effective than delivery by traditional, paper and pencil means (e.g. Pugh's method)[3].
| DECISION SPACE | 1. PROBLEM COMPLETENESS | Complete |
| 2. ABSTRACTION LEVEL | Quantitative | |
| 3. DETERMINISM | Distributed | |
| PREFERENCE MODEL | 4. OBJECTIVE FUNCTION | Optimum |
| 5. CONSISTENCY | Consistent | |
| 6. COMPARISON BASIS | Absolute | |
| BELIEF
MODEL |
7. DIMENSION | 1, Knowledge |
| 8. BELIEF COMPLETENESS | Complete | |
| 9. PROBLEM FOCUS | Process(primarily) | |
| 10. RANGE OF INDEPENDENCE | Type II: Dependent | |
| 11. LEVEL OF SUPPORT | Level 2: Decision Analysis |
Table 7. Decision tree, Decision support system
Decision trees represent the relationships between issues and chance occurrences. From each
issue node there are branches representing the various alternatives for resolving the issue. At
each chance node there are branches representing the possible outcomes from events affecting the
system over which there is no control. The probability of any of the branches from a chance node
is a measure of the belief in the chance occurrence [7]. The decision space is defined by all the
possible outcomes, i.e. all the alternative/chance combinations. The values placed on each
outcome by the decision makers, i.e. the worth of the outcome, define the preference model.
Decision support systems and decision trees support a belief model through which the expression
of knowledge about chance occurrences affects the outcome of the system in dependent issues.
This expression must be complete at each chance node to calculate the outcomes.
Pugh's Method: Pugh's method is a refined form of the decision matrix method [10][18].
Selection among itemized alternatives is accomplished by relative comparison to a set of criteria
defined in the issue. Each alternative is weighed by its ability to meet each criteria. Belief is
expressed through the expression of confidence in each alternative's ability to meet each criteria.
This method is used to support judgements about qualitative information. It results in an abstract
satisfaction expectation for each alternative. Pugh's method supports an individual decision
maker and thus uses consistent information. The information must be deterministic and complete
for the method to be useful.
| DECISION SPACE | 1. PROBLEM COMPLETENESS | Complete |
| 2. ABSTRACTION LEVEL | Qualitative | |
| 3. DETERMINISM | Deterministic | |
| PREFERENCE MODEL | 4. OBJECTIVE FUNCTION | Judgement |
| 5. CONSISTENCY | Consistent | |
| 6. COMPARISON BASIS | Relative | |
| BELIEF
MODEL |
7. DIMENSION | 1, Confidence |
| 8. BELIEF COMPLETENESS | complete | |
| 9. PROBLEM FOCUS | Product or process | |
| 10. RANGE OF INDEPENDENCE | Type I: Independent | |
| 11. LEVEL OF SUPPORT | Level 2 Decision Analysis |
Table 8. Pugh's method
Utility Theory: Utility theory allows decision makers to give formalized preference to a space
defined by the alternatives and criteria. For example, in one method, each alternative/criteria pair
is given a score reflecting how well the alternative meets the criteria. The scores for each
alternative are combined with measures of each criterion's importance (i.e. weight) to give a total
utility for the alternative. Utility is a measure of preference for one alternative relative to
another. A good formalization of this method is given in Edwards [4].
Although utility theory allows a strong representation of preference, it has no model of belief.
Thus, methods using this approach focus on ways of modeling the objective or utility function,
have no formalized way to manage information concerning knowledge and confidence.
| DECISION SPACE | 1. PROBLEM COMPLETENESS | Complete |
| 2. ABSTRACTION LEVEL | Quantitative or qualitative | |
| 3. DETERMINISM | Deterministic | |
| PREFERENCE MODEL | 4. OBJECTIVE FUNCTION | Judgement |
| 5. CONSISTENCY | Inconsistent is possible | |
| 6. COMPARISON BASIS | Relative | |
| BELIEF
MODEL |
7. DIMENSION | None |
| 8. BELIEF COMPLETENESS | N/A | |
| 9. PROBLEM FOCUS | Product or Process | |
| 10. RANGE OF INDEPENDENCE | Type I: Independent | |
| 11. LEVEL OF SUPPORT | Level 2: Decision Analysis |
Table 9. Utility theory
6. CONCLUSIONS
There are thousands of decisions made in every design project. Few are supported by formal
methods or tools. This leads to one of two conclusions: Either there is no need for decision
support (designers and design teams do very well without aid), or tools and methods have not yet
been developed that give adequate decision support. From the results of the IBIS experiments
combined with anecdotal evidence it is believed that individual designers and, more importantly,
teams of designers do not make good design decisions. This is evidenced by the length of design
meetings, the rehashing of previous information, the dropped issues and the lack of documented
rationale for decisions. Thus, in this concluding section, the reasons for this lack of use and
research directions to improve the quality of design decisions will be summarized. This section
is organized parallel to the taxonomy, and reference to the example problem and Table 2 is
suggested.
In reviewing the decision making literature from the different disciplines it is clear that all are
solving problems that can be compared and contrasted using a structure composed of five
information types - issue, criterion, alternative, argument and decision. Thus, the uniqueness or
similarity of design problems and methods to those in other fields can be made with measures of
these five. The taxonomy is an attempt to add such measures.
The taxonomy was developed by synthesizing terminology from disciplines which already have
an established history of supporting decision making. Thus, the vocabulary in the taxonomy is
difficult. However, this effort at unifying the various view points serves as a basis for evaluating
problems and decision support tools. It is suggested that before any tool is used on a design
problem, the problem and the tool be evaluated against the eleven measures in the taxonomy.
In many design problems the decision space is incomplete - there are alternatives that have not
been developed and some criteria are unknown or are only weakly understood. This is especially
true in the early stages of the design process when critical decisions are made. Ideal design
decision support would point the designer toward solutions possibly better than those in the set
considered. It would also be able to assist criteria that are evolving, not fully refined or
conflicting. Further, in many design problems, the alternatives and criteria are not all
quantifiable. Thus, design decision support needs to accommodate information that is refined
along with that which is abstract - qualitative. Finally, all design information, whether
quantitative or qualitative, is distributed and not point-valued.
These characteristics of the decision space point to the need for tools and methods that can
accommodate alternatives and criteria that are not well known, point-value variables. Pugh's
method and IBIS move in that direction, but are almost too informal. Fuzzy methods attempt
this, but are not widely accepted. Extensions of utility theory, optimization and other accepted
methods to manage such information may be needed.
All decision support tools and methods are built around a preference model which relates how
the alternatives are compared to the criteria. All of the methods or tools reviewed require a
consistent preference model. However, most design problems are solved by teams of
individuals possibly representing different view points. Each team member may have a different
model of what is important and this preference needs to be easily reflected in any support tool.
All decisions are based on belief in how well the alternatives to meet criteria. Engineers are not
accustomed to dealing with belief in a formal manner and thus tools commonly used are weak in
support of any information that is not totally true. However, all decisions are made based on
less-than-perfect knowledge and less-than-absolute confidence in the information. As a design is
refined, tests are made and simulations run, knowledge increases and confidence in solutions
evolve. There is challenge to develop decision support methods and tools that can support belief
and its evolution.
To support engineering design there needs to be the realization that the product and process co-evolve. Thus support for decision making about the product must reflect the effect of the
decision on the process (i.e. what to do next, time, cost, personnel commitments). Conversely,
decisions about the process obviously affect the product.
Engineers decompose design problems into manageable sub-problems and work on these as
independently as possible realizing that the decisions made on one may have ramifications on
others. Managing this interdependence is difficult and often poorly done. The tools reviewed
were limited in their support and need extension to give sufficient aid.
Finally, experiments with IBIS showed that even representation of the decision making process
had benefit to designers. Thus, even the simplest aid in helping the design structure the decision
space, preferences and beliefs seems to be of help. Of course, systems that give outcome or full
decision analysis are even more supportive. It is suggested that systems which currently offer
representation or outcome analysis be pushed to offer more decision support. In these
developments, attention to the previous ten measures of decision support is essential.
Like any classification system, the one presented in this paper is transitory. As new methods are
developed and new insight gained, this scheme will change and evolve. As it is now, it adds
structure to a diverse field that is little used in day-to-day engineering work, and explains that
lack of use: existing tools do not address many of the core requirements for engineering decision
support.
References
[1] A Process- Based Approach to Computer- Supported Engineering Design, Blessing, L.,
Thesis, University of Twente, Enschede, the Netherlands, 1994.
[2] Computer Select, CD Rom database of available computer programs.
[3] "Decision Support Systems for ill-structured Problems: an empirical study" Cats-Baril W. L.
and G.P. Huber, Decision Sciences, Vol 18, 1987, pp350- 372.
[4] "SMARTS and SMARTER: Improved Simple Methods for multiattribute Utility
Measurement", Edwards W. and F. H. Barron, To be published in Organizational Behavior and
Human Decision Processes.
[5] Probabilistic Mechanical Design, Haugen E. B., Wiley Interscience, New York, 1980.
[6] The State of the Art in Decision Support Systems, Hopple, G.W., QED Knowledge Sciences,
1988.
[7] "Critical Decisions under Uncertainty: Representation and Structure" Kuipers B., A.J.
Moskowitz and J.P. Kassirer, Cognitive Science #12, 177-210, 1988, pp177-211.
[8] A Knowledge Base Data Representation for Collaborative Mechanical Design, Nagy R.,
Master Thesis, Department of Mechanical Engineering, Oregon State University, Nov 1990.
[9] "A Data Representation for Collaborative Mechanical Design," Nagy, R.L. and D.G. Ullman,
Research in Engineering Design, Vol. 3, No. 4, 1992, pp. 233-242.
[10] Total Design, Pugh S., Addison Wesley , 1990.
[11] "Dilemmas in a General Theory of Planning", Rittel H.W.J. and M.M. Webber, Policy
Sciences, #4 1973, pp155-169
[12] Probabilistic Engineering Design, Siddal J. N., Marcel Dekker, New York, 1983.
[13] "Sensitivity of complex, internally coupled systems", Sobieszczanski-Sobieski J., AIAA
Journal, 28(1), 1990.
[14] An Empirical Study on the Process of Mechanical Design, Stauffer, L., Doctoral
Dissertation, Department of Mechanical Engineering, Oregon State University, Sept 1987.
[15] "Protocol Analysis of Mechanical Engineering Design," Stauffer, L., D.G. Ullman, and T.G.
Dietterich, Proceedings of the 1987 International Conference on Engineering Design, WDK 13,
Boston, MA, August 1987, pp. 68-73.
[16] Systems Analysis and Management: Structure, Strategy and Design, Steward D.V.,
Petrocelli Books, 1981.
[17] "A Model of the Mechanical Design Process Based on Empirical Data", Ullman, D.G., T.G. Dietterich, and L. Stauffer, Artificial Intelligence in Engineering, Design, and Manufacturing. Vol 2 (1) 33-52. 1988.
[18] The Mechanical Design Process, Ullman D. G., McGraw Hill, 1992.
[19] "Issues Critical to the Development of Design History, Design Rationale and Design Intent
Systems", Ullman, D. G. and R. Paasch: DTM94, Minneapolis, Sept 1994, and in Journal
review.
[20] "Analysis of Protocol Data to Identify Product Information Evolution and Decision Making
Process", Ullman D.G., D. Herling and A. Stinton, Delft Workshop on Protocol Analysis, Sept.
1994, Delft, the Netherlands.
[21] "The Capture of Design Rationale on an Industrial Development Project: Preliminary
Report" Yakemovic K.C.B. and J. Conklin, MCC Technical Report Number STP-279-89, July
1989.
[22] "Report on a Development Project Use of an Issue-Based KNOWLEDGE System" Yakemovic K.C.B. and J. Conklin, MCC Technical Report Number STP-247-90, June 1990.
Tables:
Table1: Decision Problem Taxonomy
Table 2: Example Problem
Table 3: IBIS
Table 4: Probabilistic Design
Table 5: Formal Optimization
Table 6: CPM/PERT
Table 7: Decision Tree, Decision Support Systems
Table 8: Pugh's Method
Table 9: Utility Theory
Bruce D'Ambrosio
Dr. D'Ambrosio received his Phd in Artificial Intelligence from the
University of California at Berkeley in 1986. His current research is
focused in developing decision-theoretic models of real-time
monitoring, diagnosis, and assessment. Dr. D'Ambrosio is a member of
AAAI and the ACM, is on the editorial board of Knowledge Engineering
Review, the International Journal of Approximate Reasoning, and the
Journal of Heuristics. He has served as both program and general
chair for the Conference in Uncertainty in AI, and is organizer of the
first Advanced Summer Institute on Probability in AI.
David G. Ullman
Dr. Ullman is a Professor of Mechanical Engineering Design at Oregon State University and a Professional Engineer in Oregon and New York. He has worked as design manager for a medical products OEM; holds two patents for current products one of which is licensed to major corporation. He is author of The Mechanical Design Process and The Failure Analysis Method. He contributed to Concurrent Engineering: The Product Development Environment for the 1990s. Dr. Ullman is the founder and first chairman of the ASME Design Theory and Methodology Committee and was chairman of the 1989 and 1990 Design Theory and Methodology Conferences. He is a fellow of the ASME. He holds a Masters in Aerospace Engineering from the University of Cincinnati and a PhD. in Mechanical Engineering from Ohio State University. He is currently involved in research aimed at understanding the mechanical design process and the development of design aids.
1. This research has been supported by the National Science Foundation under grant DDM-9312996. The opinions in this paper are the authors' and do not reflect the position of the NSF or Oregon State University.