Agenda Management: Understanding and Facilitating the Management of Flightdeck Activities

---

Synopsis: This website describes our efforts to understand how pilots manage -- and mismanage -- flightdeck (cockpit) activities, and how to facilitate that process through computational aids. This page provides an overview of the research with hyperlinks to more detailed descriptions.

Keywords: cockpit task management, agenda management, strategic workload management, human factors, ergonomics, aviation, flightdeck, flight deck, aviation safety, flight safety

Last update: 20 Jul 99

What's new.

Introduction

Commercial air transportation has an admirable safety record, yet each year hundreds of lives and hundreds of millions of dollars worth of property are lost in air crashes in the United States alone. About two-thirds of these aircraft accidents are caused, in part, by pilot error. Many of these errors are errors in performing flightdeck (or cockpit) functions, others are errors in managing flightdeck goals and the functions to achieve those goals. This website describes the development of a theory of flightdeck activity management and the development and evaluation of a prototype computational aids to facilitate it.

Background

Multitasking

The modern flightdeck (or cockpit) is a multitask environment. The flightcrew (whether one or more pilots) is constantly faced with multiple, concurrent, competing, often conflicting goals to accomplish and therefore must engage in multiple activities to accomplish them. As most pilots are aware, it is not only difficult to successfully accomplish such goals, it is often even more challenging to manage the activities directed towards them. We discovered this ourselves as we developed and evaluated the Task Support System (TSS), part of an experimental avionics system to aid military pilots.

Over the years, pilots have developed a priority scheme to facilitate this management of flightdeck activities:

1. aviate to keep the airplane in the air and pointed in the right direction;
2. navigate to determine where to go and how to get there;
3. communicate with the rest of the flightcrew and with air traffic control; and
4. manage systems, like engines, fuel systems, and hydraulic systems.

Although the process of managing flightdeck activities is intuitively well-understood by pilots, we formalized it in a preliminary, normative theory, which we called Cockpit Task Management (CTM). Briefly, a goal is a desired behavior of the aircraft and a task is an activity performed to achieve it. As there are generally multiple, concurrent tasks to attend to on the flightdeck, the flightcrew must create an initial list of tasks to perform then continually

• assess the current situation;
• activate new tasks in response to recent events;
• assess task status to determine if each task is being performed satisfactorily;
• terminate tasks with achieved or unachievable goals;
• assess task resource requirements (human and machine);
• prioritize active tasks;
• allocate resources to tasks in order of priority (initiating, interrupting, and resuming them, as necessary); and
• update the task list.

The Cockpit Task Management System

Although there are many potentially effective responses to this, we chose to investigate the use of computational aids to facilitate CTM. Our first such aid (not counting the TSS, which actually preceded the development of the concept of CTM) was the Cockpit Task Management System (CTMS). Our goals for the CTMS were that it should help the pilot initiate, monitor, prioritize, and terminate tasks. To achieve these goals, we determined that the CTMS should provide information about task state (upcoming, active, terminated), status (satisfactory or unsatisfactory performance), and priority.

We implemented the CTMS using Smalltalk, an object-oriented computer programming language. We used concepts of object-oriented design and distributed artificial intelligence in the CTMS implementation, where aircraft subsystems and flight tasks were represented by conceptual software units referred to as agents. In the CTMS, aircraft subsystems and pilot tasks were represented by system agents (SAs) and task agents (TAs), respectively.

Each TA used information from SAs and its own procedural knowledge to determine the state of its task: latent (not imminent), upcoming (imminent), in progress, suggested (requiring immediate attention), or finished. Task status (satisfactory or unsatisfactory) was determined in a similar way. The CTMS display provided state, status, and priority information about each task.

We performed a part-task simulator experiment to evaluate the effectiveness of the CTMS in facilitating CTM performance. Twelve subjects flew a part-task simulator under both aided (with CTMS) and unaided (without the CTMS) conditions. When subjects flew with the assistance of the CTMS, the mean task misprioritization rate was reduced by 41 per cent, the mean subject response time was reduced by 18 per cent, mean unsatisfactory aircraft control was reduced by 24 per cent, and the average number of incomplete tasks during simulator flights was reduced by 82 per cent.

Agenda Management

Our theory of CTM, as originally formulated, failed to address two important issues. First, human pilots are coming to depend more and more on automated aids, such as autopilots and centralized monitoring and alerting systems, to aid them in the monitoring and control of the aircraft and its subsystems. As machines perform certain goal-directed flightdeck activities, it is more appropriate to speak of those activities as functions since, technically speaking, a task is a function performed by a human. Second, with both humans and machines performing flightdeck functions, there is a potential for conflicting goals. Two recent aircraft accidents illustrate such goal conflicts. In 1994 in a China Airlines Airbus A300 on approach to Nagoya, Japan, the flightcrew inadvertently initiated an autoflight system go-around maneuver while trying to continue the landing. The goal conflict between the flightcrew and the autoflight system caused an out-of-trim condition that resulted in a stall and crash which killed 264 persons. In an American Airlines Boeing 757 on approach to Cali, Columbia in 1995, the flightcrew accepted an air traffic control clearance to fly direct to a designated navigational fix. They inadvertently configured the aircraft's flight management system to fly the airplane to a different fix. This goal conflict was not detected in time to prevent the aircraft from crashing into mountainous terrain, killing 159 persons.

To address these issues, which were clearly related to the original theory of CTM, we expanded the theory. Since an 'agenda' is a list of things to do, we called the new concept Agenda Management (AMgt). To formalize the concept, we developed a model of AMgt using IDEF0, a functional modeling language. IDEF0, whose name stands for ICAM (Integrated Computer Aided Manufacturing) DEFinition language 0, is a graphical modeling language. IDEF0 diagrams consist of boxes representing activities and arrows representing inputs and outputs to and from those activities, controls or constraints on the activities, and mechanisms that perform the activities. In an IDEF0 model of a process, each box represents an activity or function, which transforms its inputs to its outputs, subject to certain controls or constraints, by means of a set of mechanisms. The following summary theory of AMgt is based on the model.

An actor is an entity that does something in that it can control or change the state of the aircraft and/or its subsystems. Pilots are human actors; machine actors include autoflight and flight management systems. A goal is a representation (mental, electronic, or even mechanical) of an actor's intent to change the state of the aircraft or one of its subsystems in some significant way, or to maintain or keep the aircraft or one of its subsystems in some state. For example, a pilot might have a goal to descend to an altitude of 9,000 ft, a goal to maintain the current heading of 270E , and a goal to crossfeed fuel to correct a fuel system imbalance. If configured properly, the autoflight system in this example would also have a goal to descend to 9,000 ft and a goal to hold 270E . Goals come about as a result of planning and decision making in the case of human actors, and computation or human input, in the case of machine actors.

A function is an activity performed by an actor to achieve a goal. That activity may directly achieve the goal or it may produce sub-goals which, when achieved by performing sub-functions, satisfy the conditions of the original goal. Actors use resources to perform functions. Human actor resources include eyes, hands, memory, and attention; machine actor resources include input and output channels, memory, and processor cycles. Other machine resources include flight controls, electronic flight instrument system displays, and radios. In general, several goals might exist at any time, so several functions must be performed concurrently to achieve them. Actors must be assigned to perform those functions and resources must be allocated to enable them. An agenda then is a set of goals to be achieved and a set of functions to achieve those goals.

Agenda Management (AMgt) is a high-level flightdeck function performed cooperatively by flightdeck actors, which involves two sub-functions:

1. Goal management is the process of
1. recognizing or inferring the goals of all flightdeck actors;
2. canceling goals that have been achieved or are no longer relevant;
3. identifying and resolving conflicts between goals; and
4. prioritizing goals consistently with safe and effective aircraft operation.
2. Function management is the process of
1. initiating functions to achieve goals;
2. assigning actors to perform functions;
3. assessing the status of each function (whether or not it is being performed satisfactorily and on time);
4. prioritizing those functions based on goal priority and function status; and
5. allocating resources to be used to perform functions based on function priority.

1. there are no goal conflicts;
2. all goals and functions are properly prioritized; and
3. either
1. performance of all functions is satisfactory, or
2. if that is not possible, actors are actively engaged in bringing the highest priority unsatisfactory functions up to a satisfactory level of performance.

The AgendaManager

From the results of our CTM studies and our analysis of the Nagoya, Cali, and other aircraft accidents, we have concluded that AMgt -- and specifically the failure to perform AMgt satisfactorily -- is a significant factor in flight safety. The objectives of our most recent research task were to develop and to evaluate an experimental computational aid to facilitate AMgt. We call this aid the AgendaManager (AMgr).

Simulator

The part-task flight simulator that provides the context for the AMgr models a generic, twin engine transport aircraft. It is built from components developed at the NASA Langley and NASA Ames Research centers and in our own lab. It runs on one or two Silicon Graphics Indigo 2 computers and provides a simplified aerodynamic model (Langley), autoflight system (Langley), Flight Management System (Langley), primary flight displays (Ames), Mode Control Panel (Ames), and system models and system synoptic displays (OSU). The software is written in C, FORTRAN, and Smalltalk (VisualWorks 2.5).

Architecture and Function

From the IDEF0 model of AMgt we generated a data dictionary consisting of the entities that are the inputs, outputs, and controls of the activities in the model. We used this information to define the object-oriented architecture of the AMgr and the functions of its components. Major AMgr objects include System Agents, Actor Agents, Goal Agents, Function Agents, an Agenda Agent, and an Agenda Manager Interface. Each Agent is a simple knowledge-based object representing the corresponding elements of the cockpit environment. As a representative of such an element, the Agent's purpose is to maintain timely information about it and to perform processing that will facilitate AMgt. An Agent's declarative knowledge is represented using instance variables. Its procedural knowledge is represented using Smalltalk methods.

System Agents (SAs) represent systems modeled in the flight simulator, remembering their state and recognizing abnormal conditions such as malfunctions. System Agents provide situation information to the other AMgr Agents. Actor Agents (AAs) recognize actor (pilot or autoflight system) goals and instantiate Goal Agents. The Flightcrew Agent recognizes pilot goals by means of a Verbex VAT31 automatic speech recognition system as the pilot acknowledges air traffic control clearances. Goal Agents (GAs) represent actor goals. They detect conflicts and determine when goals are achieved. Function Agents (FAs) monitor the progress of activities directed towards the goals, noting whether that progress is satisfactory or unsatisfactory. The single Agenda Agent contains and coordinates the other Agents, introducing new Agents to its collections, checking GAs against each other to identify conflicts, and ordering Goal and Function Agents by priority. The AgendaManager Interface displays AMgt information to the pilot.

Operation

As the simulator runs it sends state data to the AMgr, whose SAs maintain a situation model of the simulated aircraft and its environment. AAs monitor real or simulated actors, detect or infer goals, and instantiate GAs. GAs look for conflicts with each other and monitor SAs to see if the goals are achieved. FAs monitor the progress -- if any -- made in achieving their associated goals. The Agenda Agent prioritizes GAs and FAs and keeps track of goal conflicts. The AgendaManager Interface presents this agenda information to the pilot.

Evaluation

We conducted an evaluation study to compare the effectiveness of the AMgr in facilitating AMgt with that of a model of an existing aiding system called the Engine Indication and Crew Alerting System (EICAS). Eight airline pilots flew the simulator in 30-minute scenarios under two conditions, one using the AMgr, the other using EICAS. We measured several types of performance, including how well subjects detected and resolved goal conflicts and how well they prioritized goals and functions. We also asked the subjects to rate the perceived effectiveness of the two systems in aiding their performance.

For all measures where AMgr and EICAS were functionally equivalent, there was no statistically significant difference in subject performance between the condition with the AMgr and that with EICAS. For all measures where AMgr and EICAS functionality differed significantly, AMgt performance was better with the AMgr than with EICAS, and the subjects rated AMgr effectiveness higher than EICAS effectiveness. All such differences were statistically significant at the alpha = 0.1 level. Four were statistically significant at the alpha = 0.05 level.

Discussion

AgendaManager Performance

The first set of findings (that there was no difference in measures related to functionally similar capabilities) is suggestive evidence that there was no experimenter-induced bias in favor of the AMgr. The second set of findings is strong evidence that the AMgr actually facilitated AMgt in the context of this experiment.

We must, however, be cautious concerning any inferences made from this finding. The fidelity of the simulator was fairly low and the fact that we observed a period effect (which could include learning) is an indication that perhaps the subjects did not receive adequate training. The simulator was a one-pilot version whereas all of our subjects fly on a two-pilot flightdeck. Finally, the success of the AMgr depends to a very large extent on its ability to correctly recognize the pilot's goals. In five to 10 percent of our subjects' goals the automatic speech recognition system (an old model) did not recognize the goal from the subject's utterance and the Goal Agent had to be instantiated by the experimenter.

Nevertheless, our findings are suggestive that AMgt performance, which is significant to flight safety, can be enhanced by means of a computational aid. Especially in light of recent advances in automatic speech recognition technology and the Federal Aviation Administration's plans to introduce datalink technology to deliver clearances to aircraft, we believe that further development of the AMgr is warranted.

Related Systems

The relationship of the AMgr to several existing aiding systems should be noted. First, the AMgr can be considered a logical extension of the Engine Indication and Crew Alerting System (EICAS) used in present-generation Boeing aircraft, and similar centralized monitoring and alerting systems in other aircraft. EICAS and related systems have been very successful and well received by the operational community. However, they are limited in the extent to which they can tailor the information to the phase of flight and they are not capable of merging the information in case of multiple failures. Of much greater significance is that little or no effort is made to consider the flightcrew's intent at any given moment. The AMgr builds on the success of EICAS by adopting EICAS display philosophy and coding and overcomes the latter limitation by basing its operation on the pilot's declared goals.

The AMgr also has some affinity to Pilot's Associate, Rotorcraft Pilot's Associate, and CASSY (Cockpit Assistant System), all of which are aiding systems designed to offer integrative and active assistance to the pilot. The AMgr is distinguished from these and similar systems in that it does not attempt to be a general, active aid. Rather, the AMgr focuses on passively assisting the flightcrew in performing AMgt by supplementing human memory and attention, not action.

Further Research

Work remains to be done on the AMgr and the concept of AMgt. For example, the AMgr should be evaluated in a more realistic scenarios in a full-mission simulator. This is necessary to be sure that the effects that we saw in this evaluation were not merely artifacts of the simplified part-task environment.

During AMgr development, we experimented with a goal communication method that integrated overt communication (via clearance acknowledgement) and covert communication (via script-based intent inferencing). Although we chose to include only overt goal communication in the current version of the AMgr, covert methods offer the potential of low pilot workload and should be further investigated.

An enhancement we have begun exploring is fuzzy function assessment. Function Agents in the current version of the AMgr use conventional (crisp) logic to assess how well functions are being performed. In some cases (for example, aviate functions) fuzzy logic may be more appropriate, so we conducted a study of fuzzy function assessment to provide more human-like function assessments. Through interviews with pilots we extracted fuzzy if-then rules to model human function assessment. Then we fine-tuned the rules with the application of a genetic algorithm which minimized the discrepancy between human and machine assessments of sample scenarios. Although a preliminary evaluation of fuzzy function assessment revealed performance comparable to that of human pilots, the method needs further refinement.

Although the AMgr has potential as an operational aid, its near-term benefits may be realized in other ways. For example, with suitable modifications, the AMgr could be embedded in a part-task trainer to facilitate AMgt training. Another possible role is as a research tool. With relatively minor changes the AMgr could be used to capture AMgt data on-line in full-mission simulator experiments. In fact, the greatest value of the AMgr may be in this capacity, helping us understand the phenomenon of Agenda Management better.

Acknowledgements

Much of this research was performed under NASA Ames Research Center grant NAG 2-875. Kevin Corker and Barbara Kanki, of the Ames Aviation Operations Branch, were our technical monitors, and we greatly appreciate their support and encouragement. Greg Pisanich, of Sterling Software, Dr. Corker's assistant, was a constant and essential source of technical and moral support. And we especially appreciate the participation of our subjects. They not only helped us evaluate the AMgr, but also gave us many valuable insights into line operations, which will greatly benefit our future efforts.

What's New

20 Jul 99

• link to new CTM website

• fuzzy function assessment paper