Tom A. Kuffner
David G. Ullman E-mailUllman
22 February 1990
Abstract:
Design documentation does not typically include all of the information sought by mechanical
design engineers. This paper reports on a study of practicing engineers making modifications to
existing designs. Particular attention is paid to the design information required to answer
questions about the design and verify or refute conjectures about the design. A taxonomy of the
questions asked by the designers in this study and the conjectures they formed is presented. It is
proposed that an intelligent CAD system be developed to capture, structure, and re-play this
information.
Introduction
Current design documentation consists of a complete set of blueprints, showing the physical structure of a design, along with specifications, showing the manufacturing process. Designers are encouraged to keep design notebooks as well. These are often maintained along with the more formal documentation. A design notebook is traditionally a bound notebook in which all of a mechanical engineer's work on a particular design is performed and recorded. The ideal design notebook contains every written or drawn artifact relating to a design, from concept through blueprint. The pages in such notebooks are permanently bound, numbered, and dated.
With a clear and comprehensive design notebook, one could follow the progression of a design from the original germ of an idea through its various iterations to the final, completed design. Design notebooks are held to be useful, even essential, during the initial design process (to record decisions) as well as in cases of patent law (claiming the originality of a design), liability litigation (proving the validity of a decision making process), and in subsequent design efforts. Subsequent efforts could include modification to the original design, using the original design as a model when designing a similar object, designing an adjacent component as in an assembly, or analysis of the design by management or in downstream efforts such as drafting or manufacturing. (Buckley 89, Pare 63, Weber 84)
The problem with the current state of design notebooks is that very few (possibly none) are maintained to the above ideal level of completeness. Sketches are made on cocktail napkins and the backs of envelopes, groups work out ideas on chalkboards, realizations are made in the shower and on the way to work, decisions are made on the shop floor in response to unforseen conflicts or opportunities. This work seldom makes it into even the most meticulous of design notebooks. Additionally, notes that make perfect sense to the original designer when written may be unintelligible to any other person and jumbled even to the original designer months later.
An intelligent computer aided design (CAD) tool could potentially maintain design notebooks automatically. Having a computer tool maintain the design notebook has the advantages of automatically capturing complete design information and structuring this information in a useful manner.
This intelligent CAD system should contain the information about a design that human designers are interested in. It should have a knowledge base capable of answering questions a designer might pose about a design and verify or refute any likely conjecture about the design.
This paper reports on research performed to identify the design information designers are
interested in, the questions designers ask which could be answered by intelligent CAD and the
assumptions designers make that could be verified by intelligent CAD. Research is on-going to
determine a format to store and replay for this information and to determine a method of
capturing this information.
Research Methods
Three professional Mechanical Design Engineers were used as subjects in the research. The subjects were given a complete set of blueprints and original specifications for completed designs and were audio and video taped while making modifications to these designs. Experiments of this sort are known as question asking protocols (Kato 86, Letovsky 86, Newell 72, Stauffer 87A, Ullman 87). As they worked, each subject sought certain information about the design. Much of this information was available in the documentation provided. To supplement the documentation, an examiner who was familiar with the designs was available as a design information resource. Any inquiry for design information was labelled a question and examined carefully in the analysis of these protocols.
Also of interest were the conjectures formed by the subjects. Conjectures are formed when a designer does not have enough information to know things with certainty but can make an informed guess. The information necessary to verify uncertain conjectures was also analyzed in this study.
The three protocols ranged in length from just over 1 hour to 2 hours 45 minutes. S10 was the first subject, the protocol being performed in February 1988. The re-design protocol problem for this subject was based on the design of a piece of manufacturing equipment that dips aluminum plates into a water bath coating them with a thin chemical layer. The original design was performed during a separate 16 hour video-taped protocol. (For complete specifications of the designs, see Stauffer 87B or Kuffner 90.) S10 was given blueprints of the finished design, the original specifications, and four proposed changes to these specifications. The S10 protocol was studied in some depth and the protocol technique refined before continuing with the S11 and S12 protocols in June 1989.
The S11 protocol was based on the same design as the S10 protocol. To streamline the process, however, S11 was given only two changes to make.
S12 worked on a different design: a plastic enclosure for three small batteries and the formed copper contacts for connecting these batteries in series. This was designed by yet a different protocol subject in a 12 hour protocol. As with the other subjects, S12 was given finished blueprints and the original problem specifications for this design and was given two changes to make to the design. A different design was chosen for S12 in an attempt to acquire more general results than would be obtained if all subjects worked on the same design.
The S11 and S12 protocols differed from the S10 protocol in one important aspect. All three
protocols were preceded by a brief warm-up session. This acted as an equipment check and got
the subjects accustomed to the verbal protocol process. S10 warmed-up by performing a simple
original design problem without any examiner intervention. The warm-up sessions for S11 and
S12 involved re-design tasks that were similar to those they worked on during the actual re-design protocols. In these sessions, however, instead of only answering direct questions about
the design, the examiner worked with the subjects by volunteering design information thought to
be helpful. The examiner thus tried to build a rapport with the subjects and worked to train the
subjects as to how stored design information could be used as a re-design tool. This different
approach resulted in the two later subjects asking more questions than S10. S11 used the
examiner's knowledge 2.3 times more than S10 and S12 used the examiner's knowledge 3.5
times more than S10 to answer questions and verify conjectures. The other functions of the
warm-up session, to ensure that the equipment was functioning properly and to make the subjects
feel more comfortable verbalizing their thoughts while working, were also achieved by this
procedure.
Analysis Techniques
The analysis of the protocols focused on the questions that the subjects asked and the conjectures
that the subjects formed, the hypothesis being that access to a complete design information would
answer all questions and eliminate the need for unsupported conjecture. The following
definitions are used in this research:
Question: Interrogation by the subject or discussion initiated by the subject about any uncertain
aspect of the design. These inquiries may be directed toward either the examiner, the designer's
notes, drawings, given specifications, or the subject's own memory.
Conjecture: Conclusion about the design inferred by the subject from incomplete information.
Interpretation, supposition, or assumption believed but not known for certain.
Transcripts of the three protocols were studied to find the questions and conjectures in each
protocol. Those questions and conjectures that related to the design artifact or its requirements
(as opposed to questions about the protocol process) were used to generate the taxonomy
presented below and then classified according to that taxonomy. The questions asked and the
conjectures formed by the subjects were studied to evaluate the classes of information that the
designers were interested in: information that should available from an intelligent CAD system.
Taxonomy of Questions and Conjectures
The analysis of the protocols focused on the questions that the subjects asked and the conjectures
that the subjects formed according to the above definitions. Those questions and conjectures that
related to the design artifact were classified according to the taxonomy shown in Table 1, below.
Following the table is a definition of each of the terms.
Table 1: Taxonomy of Questions and Conjectures
| CATEGORY: | NATURE: |
| Simple conjecture | Construction |
| Conjecture with verification | Location |
| Verification question | Operation |
| Open question | Purpose |
| TOPIC: | CONFIRMATION: |
| Assembly | Unconfirmed |
| Component | Confirmed by: |
| Interface | Examiner |
| Feature | Drawings |
| AGE OF TOPIC: | Specifications |
| Old | VALIDITY: |
| New | True |
| Specification | False |
| Unconfirmed | |
| No conjecture |
Category
Question and conjecture passages are classified as being either conjectures or questions
according to the above definitions. Conjectures are categorized as being either Simple
Conjectures or Conjectures with Verification; questions are similarly categorized as Open
Questions or Verification Questions according to the following:
Simple Conjecture: A conjecture formed with no apparent, immediate attempt at verification.
e.g. "I think this is for mounting."
"I think this is steel."
Conjecture with Verification: A conjecture immediately followed by a verification attempt.
e.g. "I think this is for mounting. Is that right?"
"I think this is steel. Is that right?"
The conjecture may or may not actually be verified by the examiner or other outside source, the passage is classified here by its format only, not by the response.
Verification Question: A question formed such that a simple answer is all that is required by way of response. These are primarily yes or no questions formed when the subject wants to verify a single, conjectured, plausible answer
e.g. "Is this for mounting?"
"Is this steel?"
Also in this class are disjunctive questions asked when the subject has conjectured two feasible answers.
e.g. "Is this for mounting or for strength?"
"Is this steel or aluminum?"
Note that questions of this form are classified as verification questions whether or not they are explicitly verified.
Open Question: A question asked requiring a detailed answer. Formed when the subject has no clear idea of what the answer might be.
e.g. "What is this for?"
"What is this made of?"
The number of questions and conjectures in each of the four categories formed by each of the
three subjects appear in Table 2, below.
Table 2: Category of Questions and Conjectures by Subject
| S10 | S11 | S12 | Combined | |
| Simple conj. | 116 (58%) | 32 (33%) | 18 (25%) | 166 (45%) |
| Conj w/verif. | 15 (7%) | 34 (35%) | 30 (42%) | 79 (21%) |
| Verif. question | 37 (18%) | 15 (15%) | 16 (22%) | 68 (18%) |
| Open question | 34 (17%) | 17 (17%) | 8 (11%) | 59 (16%) |
| 202 (100%) | 98 (100%) | 72 (100%) | 372 (100%) |
All three protocol subjects formed approximately two questions for every one conjecture. The
ratio of Verification Questions to Open Questions is fairly consistent for the subjects as well,
varying from about one to one to two to one. There is a striking difference, however, in the ratio
of Simple Conjectures to Conjectures with Verification among the subjects. S10 formed 7.7
Simple Conjectures for every Conjecture with Verification; for S11, this ratio is roughly one to
one; S12, however, formed more Conjectures with Verification than without, forming only 0.6
Simple Conjectures for every one Conjecture with Verification. In analyzing this, it must be
acknowledged that S10 worked longer on the protocol than the other two subjects changing more
and forming more new conjectures (see below). Since the examiner was present expressly for
purposes of helping with the old design, these new conjectures are more likely to be Simple
Conjectures rather than Conjectures with Verification or either type of question.
Topic
The topic of each passage is also identified. The topic is defined as the design object that the
question or conjecture focuses on. If the question or conjecture were in the form of a simple
sentence, the topic would be the noun or the subject of the sentence. All questions and
conjectures are classified as belonging in one of the following four categories (all examples are
from the protocols):
Assembly -- The topic of the question/conjecture is an assembly, either the complete assembly or a sub-assembly.
e.g. "What is this flipper dipper?" is a question about the entire assembly which is the focus of the re-design effort.
Component -- The topic of the question/conjecture is a single component of the whole structure.
e.g. "My clamp appears to be OK." is a conjecture about the clamp which is a single component of the design.
Interface -- The topic of the question/conjecture is the relationship or interface between two or more components or assemblies.
e.g. "How does this pivot arm seat in (the mounting brackets)?" is a question about the interface between two components of the design.
Feature -- The topic of the question/conjecture is some specific feature of some assembly, component, or interface.
e.g. "I've got 11 1/2 inches, it appears, on the interior of this frame." is a conjecture about a
dimension which is a feature of a component.
The number of questions and conjectures in each of the four topics by each of the three subjects
appear in Table 3, below.
Table 3: Topic of Questions and Conjectures by Subject
| S10 | S11 | S12 | Combined | |
| Assembly | 40 (20%) | 18 (18%) | 7 (10%) | 65 (17%) |
| Component | 74 (37%) | 37 (38%) | 12 (16%) | 123 (33%) |
| Interface | 31 (15%) | 18 (18%) | 10 (14%) | 59 (16%) |
| Feature | 57 (28%) | 25 (26%) | 43 (60%) | 125 (34%) |
| 202 (100%) | 98 (100%) | 72 (100%) | 372 (100%) |
The proportions were surprisingly consistent among the three protocol subjects with one
exception. S12 tended to focus more questions and conjectures on the features of the design than
the other subjects, information at the finest level of detail. This may be due to the different
character of the problem that S12 worked on.
Topic Age
The topic is further identified by its relative age according to the following classifications:
Old -- The topic of the question/conjecture is some aspect of the original design as it existed before the current re-design effort.
e.g. "Does the original flipper dipper work (well)?" is a question about some aspect of the old, or un-modified, design.
New -- The topic of the question/conjecture is some aspect of the design as modified during the current re-design effort.
e.g. "Would it matter where I mount this micro-switch?" is a question about some aspect of the new, or modified, design.
Specification -- The topic of the question/conjecture is some aspect of either the original specifications or changes to the specifications.
e.g. "These are what kind of plates, aluminum plates?" is a question about the design
specifications, in this case the original specifications.
As shown in Table 4, below, 13% of the questions and conjectures observed relate to the
specifications; this indicates that specification information should be available to designers. 51%
of the questions and conjectures had to do with old topics, topics that would be contained in any
information resource for an existing design. The remaining 36% of the passages related to the
changed design (in other words, new topics). A static design information tool would not address
new topics, but a design tool that recorded design histories as the design was in progress would.
Table 4: Topic Age of Questions and Conjectures by Subject
| S10 | S11 | S12 | Combined | |
| New | 100 (50%) | 29 (30%) | 5 (7%) | 134 (36%) |
| Old | 81 (40%) | 51 (52%) | 58 (81%) | 190 (51%) |
| Specification | 21 (10%) | 18 (18%) | 9 (12%) | 48 (13%) |
| 202 (100%) | 98 (100%) | 72 (100%) | 372 (100%) |
There are great differences between the subjects in this area, but all subjects referred to all ages
of design information.
Nature
In addition to identification of the topic, each question and conjecture is characterized according
to its nature. The nature is identified by the type of information that the subject either seeks (as
in a question) or presumes (as in a conjecture). While the topic, discussed above, indicates
which class of design object the question or conjecture is regarding, the nature indicates what
about that design object the subject is interested in. The four natures of questions and
conjectures are identified below.
Construction -- The question/conjecture concerns the physical structure of a design object, the manner in which a design object is made including material, shape, etc.
e.g. How is this built?
"I've got 11 1/2 inches, it appears, on the interior of this frame." is a conjecture about the construction of a component.
Location -- The question/conjecture concerns the position of a design object with respect to some reference. Where a design object is with respect to some other design object or in some reference frame.
e.g. Where is this?
"The plate comes within 1/8 inch from this edge. Right?" is a conjecture (with verification) about the location of one design object with respect to another.
Operation -- The question/conjecture concerns the behavior of a design object, the manner in which the design object performs its intended function.
e.g. What does this do?
"Does (the pivot arm) flip all the way out, or (are there) two positions?" is a question regarding the operation of the assembled mechanism.
Purpose -- The question/conjecture concerns the reason a design object is included in the design, the function a design object is to perform.
e.g. Why is this here?
"Why the two inch tubing?" is a question regarding the purpose of a feature of a component in
the design.
The number of questions and conjectures belonging to each of the four natures formed by the
three subjects appear in Table 5 below.
Table 5: Nature of Questions and Conjectures by Subject
| S10 | S11 | S12 | Combined | |
| Construction | 105 (52%) | 46 (47%) | 24 (33%) | 175 (47%) |
| Location | 37 (18%) | 19 (19%) | 26 (36%) | 82 (22%) |
| Operation | 47 (23%) | 22 (23%) | 5 (7%) | 74 (20%) |
| Purpose | 13 (7%) | 11 (11%) | 17 (24%) | 41 (11%) |
| 202 (100%) | 98 (100%) | 72 (100%) | 372 (100%) |
Between one-third to over one-half of the interest of each subject was in the construction of design objects. Construction information (as well as location information, which was also highly sought) should be contained in any complete design documentation. Operation and purpose information, on the other hand, is not included in standard design documentation yet is sought by designers when it is available. This information must be stored in intelligent CAD if it is to be available at all.
Confirmation
Whether or not the question or conjecture is confirmed and the source of the confirmation is also noted. Here confirm is used in a general sense,not just for confirming correct conjectures; for questions the term "answer" may be more appropriate, for mistaken conjectures, the term "refute" is more accurate.
Note that questions and conjectures confirmed by the subject's domain expertise are considered
unconfirmed for these purposes. The categories of confirmation are:
Unconfirmed -- No immediate confirmation or answer.
Examiner -- Confirmed by the examiner
Drawings -- Confirmed by drawings supplied to or generated by the subject.
Specifications -- Confirmed by specifications or changes of specifications provided to the
subject.
The number of questions and conjectures confirmed by each of these three sources along with the
number of unconfirmed questions and conjectures appear in Table 6, below.
Table 6:
Confirmation of Questions and Conjectures by Subject
| S10 | S11 | S12 | Combined | |
| Drawings | 29 (14%) | 9 (9%) | 2 (3%) | 40 (11%) |
| Examiner | 46 (23%) | 53 (54%) | 58 (82%) | 158 (42%) |
| Specification | 5 (3%) | 1 (1%) | 0 (0%) | 6 (2%) |
| Unconfirmed | 122 (60%) | 35 (36%) | 11 (15%) | 168 (45%) |
| 202 (100%) | 98 (100%) | 72 (100%) | 372 (100%) |
As mentioned in the section discussing category of questions and conjectures, S10 formed far
more Simple Conjectures and fewer Conjectures with Verification than the other two subjects.
This behavior is seen again in analyzing the confirmation of questions and conjectures. 60% of
S10's questions and conjectures went unconfirmed (or unanswered) compared to 36% for S11
and 15% for S12. One prime factor in this may be the additional training these two subjects
received using the examiner's design information during the warm-up sessions, as discussed
earlier. Because the examiner worked more closely with subjects S11 and S12 during these
warm-ups, they appeared to be more confident using the examiner's design knowledge. The fact
that the protocol subjects referred to the examiner's stored design knowledge at all indicates that
mechanical designers would use design information stored in any intelligent CAD tool if
available.
Validity of Conjecture
Validity is a measure of the accuracy of a conjecture. The validity of all confirmed conjectures --
including conjectures implicit in verification questions -- was determined. The validity of
unconfirmed conjectures and the validity of questions without an implicit conjecture was not
established. The validity of most unconfirmed conjectures (such as "I don't know which option is
better, but this one looks easier to solve") is impossible to measure with any certainty, while the
validity of an explicitly confirmed conjecture is readily determined. Open questions and
disjunctive verification questions do not contain a single conjecture so validity, in these cases
validity has no meaning. The four categories of validity therefore are:
True -- The conjecture formed by the subject is a valid conjecture.
False -- The conjecture formed by the subject is not a valid conjecture, it is incorrect.
Unconfirmed -- The question or conjecture was not immediately confirmed.
No Conjecture -- There is no clear single conjecture implicit in the question. The passage is a
confirmed open or disjunctive question.
Note that a listing of unconfirmed here corresponds directly to a listing of unconfirmed in the preceding confirmation category. If an open or disjunctive question is not immediately confirmed, it is listed as unconfirmed rather than no conjecture even though no clear single conjecture is present.
The number of questions and conjectures belonging to each of the four validity classes formed by
the three subjects appear in Table 7, below.
Table 7: Validity of Questions and Conjectures by Subject
| S10 | S11 | S12 | Combined | |
| True | 32 (16%) | 42 (43%) | 41 (57%) | 115 (31%) |
| False | 20 (10%) | 6 (6%) | 13 (18%) | 39 (11%) |
| Unconfirmed | 122 (60%) | 35 (36%) | 11 (15%) | 168 (45%) |
| No conjecture | 28 (14%) | 15 (15%) | 7 (10%) | 50 (13%) |
| 202 (100%) | 98 (100%) | 72 (100%) | 372 (100%) |
S10 confirmed far fewer questions and conjectures than the other subjects, as was discussed in the confirmation section above, and formed far fewer true, confirmed conjectures. This subject had approximately the same percentage of false, confirmed conjectures as the other subjects. This would indicate that S10 only confirmed conjectures when the validity was uncertain.
It is theorized that if a complete design information were available with the data structured such
that retrieval was facilitated, more conjectures would be verified and fewer incorrect conjectures
would be incorporated into the finished design.
Combinations of Taxonomic Categories
The taxonomy presented above divides the questions and conjectures formed by the three
protocol subjects according to six defined taxonomic classes: Category, Topic, Age of Topic,
Nature, Confirmation, and Validity. These six classes can be combined into fifteen possible
pairs. Five of these combinations showing particularly interesting results are presented below
(others are discussed in Kuffner 90). Discussed are Question and Conjecture:
Nature versus Topic
Category versus Validity
Category versus Nature
Topic versus Confirmation
Nature versus Confirmation
The ten remaining pairs are not analyzed here.
Question and Conjecture Nature versus Topic
Comparing the nature (Construction, Location, Operation, and Purpose) of the question and conjecture passages versus the topic (Assembly, Component, Interface, and Feature) of these passages yields some interesting patterns. The number and percentage of questions and conjectures for each combination of nature and topic is shown in figure 1, below.
High percentages of questions and conjectures were formed concerning the construction of both features and components. Also of high interest were the location of components and the construction of both assemblies and interfaces. Uncommon were questions and conjectures concerning the purpose of assemblies or interfaces.
NATURE VS TOPIC FIGURE 1 - 4.25
This distribution should guide the design information structure of intelligent CAD systems. The
subjects of this study were interested in the construction of design objects, especially features and
components, so this information must be included in and readily obtained from an intelligent
CAD system. Less important is information on the purpose of assemblies and interfaces.
Though the data from three subjects is far from conclusive, the trend is clear.
Question and Conjecture Category versus Validity
41% of the questions and conjectures in the protocols contained conjectures that were externally
confirmed (Simple Conjectures, Conjectures with Verification, or Verification Questions with a
single implicit conjecture that was confirmed). The validity of these conjectures was determined
as either true or false. 25% of these measurable conjectures were false. This result is quite flat
across the three question categories that can be deemed true or false as is illustrated in figure 2,
below.
VALIDITY BY CATEGORY figure 2
The fact that this response is flat runs counter to the hypothesis that Simple Conjectures are
formed when the subject is fairly confident in the accuracy of the conjecture, Conjectures with
Verification when less sure, and Verification Questions when still less sure. This result would
indicate that the three conjecture types are all about equally likely to be valid. On the other hand,
because of their format Simple Conjectures are less likely to be confirmed, and the validity of
unconfirmed conjectures was not determined. 81% of the Simple Conjectures went unconfirmed
compared to only 6% of the Conjectures with Verification, 19% of the Verification Questions,
and 27% of the Open Questions. The Unconfirmed questions (of both the Verification and Open
type) were primarily rhetorical questions, questions that did not require an answer.
Question and Conjecture Category versus Nature
In examining the nature (Construction, Location, Operation, and Purpose) of the passages in each
question and conjecture category (Simple Conjectures, Conjectures with Verification,
Verification Questions, and Open Questions), as shown in figure 3 below, some interesting
trends become apparent. There are high instances of Conjectures with Verification about the
Location and Purpose of the various design objects and relatively few Simple Conjectures about
the Purpose of design objects. (The trend in Location is especially strong in S12 for whom 50%
of the Conjectures with Verification concerned location.) This behavior indicates that the
protocol subjects were less sure of their location and purpose conjectures therefore more likely to
seek verification.
NATURE BY CATEGORY Figure 3 -- 3.75"
Question and Conjecture Topic versus Confirmation
Next consider the topics of the questions asked and the conjectures formed (Assembly,
Component, Interface, and Feature) versus the confirmation of these questions and conjectures
(Examiner, Drawings, Specifications, and Unconfirmed) as shown in figure 4, below. The
source of confirmation of the questions and conjectures across all topics is fairly flat
proportionally, with 77% of those confirmed, confirmed by the examiner, 20% confirmed by
drawings, and the remaining 3% confirmed by the problem specifications. The proportion of
questions and conjectures confirmed by any source to unconfirmed questions and conjectures,
however is not as well behaved. Feature based questions and conjectures are confirmed 70% of
the time, compared to an average 55% confirmation rate. This higher confirmation rate would
imply that feature information, information at the finest level of detail, is more critical, therefore
more likely to be confirmed than other, coarser design information.
CONFIRMATION by TOPIC figure 4 -- 3.5"
Question and Conjecture Nature versus Confirmation
In studying the Nature of questions and conjectures (Construction, Location, Operation, and
Purpose) versus Confirmation (Examiner, Drawings, Specifications, and Unconfirmed) one
significant trend becomes apparent. As shown in figure 4 above, questions and conjectures
pertaining to the purpose of a design object tend to be confirmed, and confirmed by the
examiner, in higher proportion than questions and conjectures regarding the other Natures.
85% of the purpose oriented questions and conjectures were confirmed, 91% of these were
confirmed by the examiner. This compares to 55% of all questions and conjectures which were
confirmed, 77% of which were confirmed by the examiner. This result was consistent for all
subjects. The reliance of the subjects on the examiners knowledge about the design in
confirming purpose questions indicates two things: 1) the subjects were uncertain of any purpose
conjecture they were able to form, and 2) the other forms of design documentation available (i.e.
drawings and specifications) are unsatisfactory in answering purpose oriented questions and
confirming purpose oriented conjecture. This being the case, intelligent CAD must provide
purpose information to supplement the other documentation forms.
Limitations of the Research
This study is exploratory not definitive. The limited number of subjects and limited number of problems addressed by each subject give an indication of the design information sought by mechanical design engineers, but this does not constitute a rigorous, thorough study.
The protocols of the three subjects total over five hours. A total of 372 questions and conjectures
were identified from these protocols and studied. While the results of the study are not
conclusive, they are revealing. The researchers believe that mechanical design engineers
working on other design problems will form the same types of questions and conjectures and in
roughly the same proportions as the protocol subjects studied. This should be the case, not only
for re-design, but for other design analysis tasks as well.
Conclusions
This study showed that mechanical design engineers are interested in design information other than that which is contained in standard design documentation consisting solely of blueprints and specifications. This additional information should be made available to working design engineers. The need is evident from the following results. Of the 372 questions and conjectures studied in the three protocols, 115 (31%) were concerning the operation or purpose of a design object. 78% of the purpose questions and conjectures were confirmed by the examiner. Operation and purpose information is not typically contained in standard design documentation. Intelligent CAD should prove to be an ideal medium to provide this information.
42% of the questions and conjectures in the protocol experiments were confirmed by the examiner (with training, this percentage increased). Some of this information was available from the documentation provided. The subjects, however, relied on the examiner because of the ease with which the information was available. The examiner was the only available resource, however, for much of this information. Access to this design information had immeasurable impact on the design. 45% of the questions and conjectures went unconfirmed. Any tool developed to store and replay design information should have an interface which facilitates retrieval. If a tool were available which contained complete design information structured in such a way as to facilitate retrieval, more questions would be answered and more conjectures would be verified.
Ten percent of the confirmed conjectures formed by the subjects were false; 75% of these were refuted by the examiner. If this source of design information had not been available, these false conjectures would have likely been incorporated into the design.
The proper tools must be developed to structure, capture, and re-play design information.
Special interest should be paid to include information which is not contained in drawings and
specifications. Intelligent CAD tools provide an ideal platform for storing additional design
information.
Future study
This work explored the possibilities of having supplemental design information available to mechanical engineers making modifications to existing designs. The research suggests the utility of an intelligent CAD tool to provide this information. As this tool is being developed, further research needs to be performed.
Researchers need to develop efficient methods for capturing, storing, and re-playing design information. Research should also investigate the impact of additional design information on the design process. Other uses of supplementary design information should be investigated as well. While this study focused on engineers making modifications to existing designs, design understanding is a crucial factor in many other activities. One important task requiring thorough design understanding and worthy of further research is the diagnosis of design failure.
This study has shown that current design documentation is not complete, and it shows specific areas where supplemental design information was requested and used by mechanical design engineers. Further steps are needed to make full use of the information discovered in this research.
Bibliography
(Buckley 89) Patricia G. Buckley, "Product Liability: Industry's Solutions", Design News, 19
June 1989.
(Kato 86) Takashi Kato, "What `Question Asking Protocols' can Say About the User Interface",
International Journal of Man-Machine Studies, Academic Press Inc. London 1986.
(Letovsky 86) Stanley Letovsky, "Cognitive Processes in Program Comprehension" in
Proceedings of First Workshop on Empirical Studies of Programmers, June 1986, Washington
DC. Elliot Soloway and Sitharama Iyengar eds. Ablex Publishing, Norwood NJ. 1986.
(Kuffner 90) Tom Kuffner, "Mechanical Design History Content: the Information Requests of
Design Engineers" Master's thesis, Oregon State University, February 1990
(Newell 72) Alan Newell and Herbert A. Simon, Human Problem Solving Prentice Hall,
Englewood Cliffs, New Jersey, 1972
(Pare 63) E. Pare, L. Francis, and J. Kimrell, Introduction to Engineering Design, Holt, Rinehart
and Winston Inc., New York, 1963.
(Stauffer 87A) Larry Stauffer, David G. Ullman, and Tom G. Dietterich, "Protocol Analysis of
Mechanical Engineering Design" in Proceedings of the 1987 International Conference on
Engineering Design, August 1987, Boston Mass.
(Stauffer 87B) Larry Stauffer, "An Empirical Study on the Process of Mechanical Design" PhD
Dissertation, Oregon State University, September 1987
(Ullman 87) David G. Ullman, Larry Stauffer, and Tom G. Dietterich, "Toward Expert CAD",
ASME Computers in Engineering, Volume 6, Number 3, Nov-Dec 1987, pp21-29.
(Weber 84) Bernard R. Weber, "Product Liability -- Some Ounces of Prevention" SAE paper
841050 summarized in Automotive Engineering, Vol 92, September 1984.