When developing large and complex systems like satellites, the development process becomes large and complex as well. As a result, it is almost impossible for a single person to cover the entire process in every details. In such a situation, communication between designers becomes critical in achieving a successful result. There are many cases where poor communication results in a failure of the system. For example, in the case of NASA's Mars Climate Orbiter, although the root cause of the failure was a mistake in the navigation software, the failure could have been avoided if a good communication was present between navigation team and development team; the navigation team did not conveyed their concerns to the development team even though they were aware of errors in the trajectory estimation of the spacecraft. In this paper, we address the problem of e-mail communication between developers, and show how we can apply the framework which the authors are proposing for capturing design rationale at low cost.
Nowadays, e-mail is widely used for communication. Especially mailing lists can be used by a group for announcements, discussion, etc. When mailing list is used for discussion which involves some kind of decision making, it is hard to track the argument and the reasons behind the decisions made. It is because the standard e-mail system only provides loose mechanism for organizing e-mails, i.e. In-Reply-To references. It is often the case that several arguments proceed in parallel, and they are mixed in e-mails. Thus, finer grain structure is needed if we are to track discussions in e-mails more precisely.
To approach this problem, we focus on capturing argumentation structures in the communication as design rationale during the development process. Design rationale is the reasons, deliberations, and arguments on the design. By recording design rationale, designs become more understandable, and can be reflected on or reconsidered afterwards. The capture and use of design rationale has become one of the crucial aspects in design research since the 1970's. However, it has been reported that there is a tremendous cost associated with capturing design rationales formally. Similar problem is recognized as the 'capture bottleneck' in the field of knowledge management. The capture bottleneck problem in knowledge management refers to the lack of enough incentives to the users to invest time for sharing their knowledge or resources.
Motivated by the capture bottleneck problem, we are building a prototype system called IDIMS (Integrated Design Information Management System). The purpose of the system is to provide developers a unified repository of design information and to capture design rationale at low cost. As the use of computer is no more limited to analysis of artefacts and CAD, and widely used for communication and documentation, we believe that by actively utilizing all the electronically available information, we may reach to a solution to the capture bottleneck problem. Using the analogy to the recycling of wastes, we call the idea `knowledge recycling'. We intend to implement and test the idea with IDIMS.
To capture design rationale at low cost, we propose a scheme in which users are asked to annotate the design documents or e-mails to indicate `design issues' or `design decisions'. IDIMS then processes documents and e-mails to collect annotations and stores them to Issue/Decision Repository. We regard the annotation of issues and decisions as a light-weight version of design rationale representation, compared to existing representations such as IBIS or QOC. Instead of requiring users to build a separate design rationale structure, the method aims at capturing design rationale with minimal additional cost.
To illustrate how the method improves the communication and development process, we analyzed the log of a satellite project developers mailing list. One of the authors manually annotated about 700 e-mails from the log, and we got a set of argument structures. We counted the number of 'unresolved issues', which are those issues that do not have decisions associated with them, and categorized them. We compared two thread structures of the e-mails: the structure built from e-mails' In-Reply-To references and the structure built from Issue/Decision annotation.
During the analysis, we have found several cases in the log which illustrate the communication problem. One of the problem found was that the same argument on an issue reappears, and it is not based on the decisions that have already been made. To avoid these kind of reappearing arguments, we propose thread-based searching mechanism, which searches for similar threads in the Issue/Decision Repository based on a modification of TF/IDF ranking. In an experiment, although thread-based search did not show significant improvement over keyword-based search compared in precision, thread-based search was more robust in finding similar threads, while keyword-based search was sensitive to the keyword selection.
The analysis shows that the proposed annotation scheme is promising in capturing design rationale and improving communication problems. However, further user studies are required to prove the cost-effectiveness of the method, which is the authors final goal.