A Methodology for Usability Evaluation of Corporate Memory Design Reuse Systems

2023-08-13 来源：我们爱旅游

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

Peter Demian and Renate Fruchter

ASCE Member, Lecturer in Construction Management , Department of Civil and Building

Engineering, Loughborough University, Leicestershire LE11 3TU, UK, Direct Line: +44 (0) 1509

22854Fax: +44 (0) 1509 223945, E-Mail: P.Demian@lboro.ac.uk

ASCE Member, Director of Project Based Learning Lab, Civil and Environmental Engineering Department, Stanford University, Stanford CA94305, USA, fruchter@ce.stanford.edu

Abstract

Improving and supporting the process of design knowledge reuse in engineering design can increase productivity, improve the quality of designs, and lead to greater corporate competitive advantage. Whereas internal knowledge reuse from one’s personal experiences is very effective, external knowledge reuse from an external digital or paper archive often fails. Based on a formalization of the internal reuse process from ethnographic studies, a prototype system, CoMem (Corporate Memory) is presented, which supports the reuse process, particularly the steps of finding and understanding. This paper presents a usability testing framework and methodology for the evaluation of reuse systems such as CoMem. The two pertinent variables are (1) the type of finding task, and (2) the size of the repository. Preliminary results from the evaluation of CoMem are resented as an example of the application of this framework for studying and assessing corporate memory design reuse systems.

Keywords

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

Design reuse, human-computer interaction, user testing

Introduction

How does one find the proverbial needle in a haystack? Design reuse from large archives of content from previous projects is an important task, but one that is rarely supported in today’s engineering design practice. The average designer, whether consciously or subconsciously, draws from a vast well of previous design experience. “All design is redesign” (Leifer 1997). This can be experience acquired by the individual or by his/her mentors or professional community. This activity is referred to as design knowledge reuse. Specifically, design knowledge reuse is defined as the reuse of previously designed artifacts or components, as well as the knowledge and expertise ingrained in these previous designs. This research distinguishes between two types of reuse:

• Internal knowledge reuse: a designer reusing knowledge from his/her own personal experiences (internal memory).

• External knowledge reuse: a designer reusing knowledge from an external knowledge repository (external memory).

Internal knowledge reuse is an effective process, which some researchers place at the very center of human intelligence: “Reminding is the mind’s method of coordinating past events with current events to enable generalization and prediction. Intelligence depends upon the ability to translate descriptions of new events into labels that help in the retrieval of prior events. One can’t be said to know something if one can’t find it in memory when it is needed. Finding a relevant past experience that will help make sense of a new experience is at the core of intelligent behavior.” (Schank 1990).

On the other hand, external knowledge reuse often fails. This failure occurs for numerous reasons, including that knowledge is not captured, it is captured out of context,

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

or there are no mechanisms or software tools to enable reuse of captured knowledge.

Empirical observations of designers at work (Demian and Fruchter, under-review) show that internal knowledge reuse is effective because the designer can quickly find (mentally) reusable items and can remember the context of each item, and can therefore understand it and reuse more effectively. These observations of internal knowledge reuse are used as the basis to improve external knowledge reuse. The three key activities in the knowledge reuse process are: (1) Finding a reusable item; (2) exploring this item’s project context which leads to understanding; and (3) exploring this item’s evolution history which leads to understanding. Our hypothesis is that if the designer’s interaction with the repository enables him/her to rapidly find relevant items of design knowledge and view each item in context in order to understand it, specifically, explore its project context and explore its evolution history then the process of reuse will be improved. This improved reuse will lead to higher quality design solutions, and save time and money.

Related Research

This research is the latest in a line of research projects on design knowledge management conducted at the Project-Based Learning Lab at Stanford University. These projects are based on Schön’s reflective practitioner paradigm of design (Schön 1983). Schön argues that every design task is unique, and that the basic problem for designers is to determine how to approach such a single unique task. Schön places this tackling of unique tasks at the center of design practice, a notion he terms knowing-in-action (Schön 1983).

To Schön, design is an action-oriented activity. However, when knowing-in-action breaks down, the designer may consciously transition to acts of reflection. Schön calls this reflection-in-action. In a cycle which Schön refers to as a reflective conversation with the situation, designers reflect by naming the relevant factors, framing the problem in a certain way, making moves toward a solution and evaluating those moves. Schön argues that, whereas action-oriented knowledge is often tacit and difficult to capture or convey, what can be captured is reflection-in-action.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

This reflection-in-action cycle forms the conceptual basis of knowledge capture in the Semantic Modeling Engine (SME) (Fruchter 1996). SME is a framework that enables designers to map objects from a shared CAD product model to multiple semantic representations and to other shared project knowledge. SME supports Schön’s reflection-in-action by enabling the designer to declare his/her particular perspective on the design (i.e. framing the problem) by creating a discipline object. Next he/she proceeds to name the individual components of the problem as he/she sees it by creating component objects. SME discipline objects are exported to external analysis tools to derive building behavior and evaluate it by comparing it to functional requirements. The designer uses these as the basis for making design decisions, i.e., making moves towards the solution and evaluating those moves.

The ProMem (Project Memory) system (Fruchter et al. 1998, Reiner and Fruchter 2000) takes SME as its point of departure and adds to it the time dimension. ProMem captures the evolution of the project at the three levels of granularity identified by SME as emulating the structure of project knowledge: project, discipline, and component. ProMem automatically versions each SME object every time a change is made in the design or additional knowledge is created.

This paper presents CoMem (Corporate Memory), a prototype system that extends ProMem firstly by grouping the accumulated set of project memories into a corporate memory, and secondly by supporting the designer in reusing design knowledge from this corporate memory in new design projects.

Although much research is dedicated to design theory, considerably less focuses specifically on reuse. Research studies on design knowledge reuse focus either on the cognitive aspects or on the computational aspects.

Research into the cognitive aspects of reuse has helped to identify the information needed by designers. Kuffner and Ullman (1990) found that the majority of information

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

requested by mechanical engineers was concerning the operation or purpose of a design object, information that is not typically captured in standard design documents (drawings and specifications). Finger (1998) observed that designers rarely use CAD tools to help them organize and retrieve design information. Ye and Fischer (2002) go further, noting that an important cognitive barrier to external reuse is the user’s unfamiliarity with the contents of the repository. Users are not aware of what is in the repository and so do not know to look for it.

On the computational side, research into design knowledge reuse focuses on knowledge representation and reasoning. Knowledge representation ranges from informal classification systems for standard components (see for example Culley 1998) to more structured design rationale approaches (Regli et al. 2000 gives an overview). Highly structured representations of design knowledge can be used for reasoning. Two common approaches are case-based reasoning and model-based reasoning.

The premise behind case-based reasoning is that new designs can be generated by modifying old designs. Model-based reasoning tools use both general domain knowledge as well as knowledge from specific cases. Both case-based and model-based reasoning are often divided into two phases: case retrieval and case adaptation, and attempt to automate certain aspects of the process. However, these approaches usually require manual pre or post processing, structuring. and indexing of design knowledge.

This research brings together the cognitive and computational approaches. Reuse is considered to be a combined effort involving both the human and the computer and addresses the issue of design knowledge reuse as a human-computer interaction problem. We aim to provide a knowledge reuse experience that leverages natural idioms and metaphors in order to support the designer in doing his/her work, and we consider automatic reasoning approaches to constrain the user’s knowledge reuse activities. In this approach, capture and indexing take place in real time, with the least possible intrusion on the design process, by supporting the typical communication and coordination activities that occur during collaborative design. By using CoMem to

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

request information from design teammates, discuss changes with teammates, make changes in the shared CAD model, etc., CoMem captures the evolution in the design as well as the rationale driving this evolution.

Preliminary Study of Design Knowledge Reuse

In a preparatory study, about 25 professionals from the architecture, engineering, and construction industry were asked to rate the frequency with which they reuse designs and domain expertise from various knowledge sources – internal memory, external memory, or mentors (Figure 1), and the frequency with which various obstacles to reuse occur in current practice (Figure 2) – “don’t know what to find,” “can’t find item,” “don’t know about design,” “don’t know about project,” “don’t know rationale.”

From Figure 1 it can be seen that engineering design professionals reuse previously designed components just as frequently as they reuse abstract domain expertise unrelated to specific designs. Therefore any reuse tool should. support both types of reuse. For both abstract domain knowledge as well as designs from a specific project, the frequency with which external repositories (such as paper or digital archives) are used is

comparable to that with which internal memory is used (one’s personal memory, or one’s mentors).

Figure 2 illustrates typical obstacles to reuse. The results indicate that these obstacles occur at comparable frequencies. Even though the inability to find was rated to occur between “occasionally” and “frequently”, that obstacle to reuse was given the highest frequency rating. This supports the claim that finding is an important step in the reuse process, as the inability to find was rated to occur particularly frequently.

CoMem – A Corporate Memory Design Reuse System

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

Following the preliminary study, a corporate memory design reuse system was developed over several years. The system is called CoMem, short for Corporate Memory. During the iterative development of this system (Fruchter and Demian 2002), a framework was developed to study and evaluate such tools.

CoMem is based on the principle of “overview first, zoom and filter, and then details-on-demand” (Shneiderman 1999). Based on the three reuse activities identified– find, explore project context, explore evolution history – CoMem has three corresponding modules (Fruchter and Demian 2002): an Overview that supports the task of finding reusable items, a Project Context Explorer that supports the task of exploring the context of an item, and an Evolution History Explorer that supports the exploration of an item’s history (Figure 3).

The Overview presents the projects, disciplines, and components as nested rectangles using the squarified treemap visualization (Bruls et al. 1999). The size of each rectangle denotes the amount of content contained in that item (number of versions, annotations, linked documents, etc.). The color of each rectangle denotes that item’s relevance to the current design task based on text analysis (Demian and Fruchter 2005). The Overview supports the designer in finding reusable items. The objective is to enable the designer to view the entire corporate memory at a glance. The Overview gives the designer an indication of which “regions” of the corporate memory contain potentially reusable items (Figure 4 (a)).

Once the user has selected an item from the Overview, the Project Context Explorer supports the designer in exploring this item’s project context. This shows the project and subsystem to which this item belongs, as well as related components and disciplines that would help the designer understand the found item. The item selected from the Overview becomes the focal point of the Project Context Explorer (Figure 4 (c)). The focal point and its related items are arranged in a two-dimensional space where the vertical axis represents level of granularity (from entire corporation down to a single component) and

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

the horizontal axis represents the degree of interest (how closely related is an item to the focal point).

In the third module, the Evolution History Explorer, the designer can explore the evolution history of any item selected from the Overview (Figure 4 (b)). This view tells the story of how this item evolved from an abstract idea to a fully designed or physical artifact or component.

Evaluation Methodology

CoMem was designed specifically to support the steps of finding and understanding reusable items from a corporate memory. The purpose of evaluating tools such as CoMem is to assess the extent to which they enable the designer to find and understand reusable items from the corporate memory, and the extent to which this ability to find and understand improves the effectiveness of the reuse process. Since it is difficult to evaluate statements such as “designer can find and understand” or “external reuse is effective” in absolute terms, the strategy of the evaluation should be to identify metrics for the validity of such statements and then to compare these metrics for CoMem (or the reuse tool being evaluated) versus “traditional tools”, as shown in Figure 5. Traditional tools are tools that reflect the current state of practice of design reuse in industry. A set of variables can be introduced into the comparisons to identify specific circumstances under which the tool being evaluated leads to more effective external reuse. This methodology can be demonstrated by describing the comparison between CoMem and two traditional tools.

CoMem offers the following tools to reuse items from the corporate memory: the Overview (Figure 6 (a)), the Project Context Explorer (Figure 6 (b)), and the Evolution History Explorer (Figure 6 (c)). The following tools were developed for the purpose of evaluating CoMem, and were used by the test participants as being representative of traditional tools used in current practice:

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

• Outline Tree. This is a prototype interface that uses indented lists of files and folders in the same way as Windows Explorer (Figure 7 (a)). The designer can use the Outline Tree to explore the corporate memory as if it were a set of files and folders on a computer, which reflects the nature of digital archives today, and the way current operating systems facilitate retrieval and exploration. It has an additional function to Windows Explorer: the generic icons for folders and files can be replaced by colored rectangles denoting the CoMem measure of relevance (the same relevance that is indicated on the CoMem Overview module, as shown in Figure 7 (b)). When the user selects an item from the Outline Tree, the versions of this item are displayed in a table similar to a spreadsheet or database program (Figure 7 (c)). The table displays the version number as well as the parent version and other textual information.

• Hit List. This is a prototype web interface (Figure 8) that returns a list of hits in the same format as a web search engine, such as Google (Brin and Page 1998). Given a problem the designer is working on, he/she can bring up the Hit List at any time, and it will display a list of items from the corporate memory ranked by their relevance to the designer’s current task. The user can also search the corporate memory by keyword. The user may select any item from the Hit List to displays all the versions of that item in a web-based table similar to a spreadsheet or database program (Figure 8 (b)).

Variables and Metrics

The aim of evaluating innovative reuse tools is not merely to determine whether they offer improved support for reuse, but also to identify the specific circumstances under which traditional tools break down and these innovative tools offer genuine added value (and vice versa). From extensive ethnographic studies (Demian and Fruchter, under-review), the following variables were identified as being pertinent.

• Type of finding task. There are two main kinds of finding tasks: retrieval and exploration. Retrieval occurs when the designer is looking for a specific item: “I

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

am looking for the cooling tower frame (component) from the structure (discipline subsystem) of the Bay Saint Louis Hotel (project) that we worked on five years ago”. Exploration occurs when the designer has no idea what to look for, only that it should be a relevant item: “I am stuck trying to design a hotel cooling tower, is there anything in the system that can help me get started?” • Size of the repository. Innovative reuse tools should be designed with large repositories in mind, as this is where traditional tools often fail. To what extent do these tools designed specifically for large repositories also support smaller repositories, and what is the repository size for which traditional tools break down?

The following metrics for effective finding can be measured:

• Time taken to complete retrieval tasks. For retrieval tasks, the time taken to find that item is the most important metric.

• Recall score for exploration tasks. For exploration tasks where the user is exploring the database for anything useful given his/her current design task, the number of relevant items found can be recorded. For each exploration task, an exhaustive list of useful items in the repository can be prepared in advance by a human expert. This list can be used to calculate a recall score for each test subject: the number of useful items found and listed by the user divided by the total number of useful items as judged by the human expert.

• Time taken to explore the entire repository for exploration tasks. The test subject can be instructed to continue exploring the repository and list all useful items until he/she feels that all useful items have been found. The time taken to feel confident that the user has found everything to be found is a good metric for the support for exploration finding provided by the reuse tool.

The following metric for effective understanding can be measured:

• The ability to answer contextual questions. After exploring the project context and evolution history of any item considered potentially useful for the current task, the user can be scored on his/her ability to answer contextual question about

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

the data in the repository. A typical question might be: “Why did the design team choose that building material?” A context score can be calculated for each user by dividing the number of correctly answered questions by the total number of questions asked. This is intended to measure the extent to which the tool enables the user to understand why that item was designed the way it was, and so overly technical or abstract questions should be avoided.

The metrics for effective external reuse are more challenging to measure. If the reuse tool were used in a real project as part of the evaluation of the tool, possible metrics would have been the percentage of the designed artifact based on reused components, or the quality of the final design. For cases where the reuse tool is used in the context of synthetic experiments where test participants complete fictional tasks, the best measurable metric that assesses the effectiveness of the external reuse process is the extent to which the user agrees with the following statements is the best measurable metric that assesses the effectiveness of the reuse process:

• If I had this system in my work, I would reuse content from previous projects more frequently than I do currently.

• If I had this system in my work, I would reuse content from previous projects more appropriately than I do currently.

User Tests Procedure

The above framework was put into practice by conducting preliminary user tests using CoMem, Hit List, and Outline Tree. Twenty participants were recruited from amongst students and researchers in the Department of Civil and Environmental Engineering at Stanford University, as well as professionals from local design offices in California.

It was not possible to test a larger sample of software users because of the unavailability of local candidates who were able to give up the large amount of time

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

(over 10 hours) required to complete the test. An experiment-design decision was taken to test fewer participants using the entire lengthy test, rather than reduce the length of the test and recruit more participants.

The participants were chosen to be as close as possible in age, computer experience, and design experience to eliminate any variability in the data due to these factors. In addition, care was taken to ensure that the sample of users was representative of the population of users for whom CoMem was designed: architectural and civil engineering designers in general, with a bias towards novice designers. The procedure for each test subject was as follows:

• Brief. A standard passage describing each of the prototypes, the nature of tasks, and

the objective of the user tests was read to the participant.

• Warm up. The participant was invited to familiarize him/herself with the prototypes

by exploring data unrelated to the tasks for about five minutes. After this warm-up, the formal experiment started.

• Retrieval tasks. The participant was asked to complete three different randomly

chosen retrieval tasks with CoMem, the Outline Tree, and the Hit List. Retrieval tasks are simple: “find the component called… which is in the discipline called… in the project called…” All retrieval tasks were of comparable difficulty. The time to complete the retrieval task was measured.

• Exploration tasks. A standard passage describing a randomly-chosen synthetic

scenario and a related exploration task was read to the participant. The participant was asked to explore the repository using CoMem and list all reusable items, until he/she feels confident that he/she has found all the reusable items in the repository. This was repeated for the Outline Tree and the Hit List with different scenarios and tasks. Exploration tasks are of the type: “you are working on this problem, find anything you think would be helpful in the corporate memory to help you complete your design task.” There were a total of 6 previously-prepared exploration tasks, all of which were designed to be comparable in difficulty (e.g., having the same number of reusable items and contextual questions). For each participant, the task chosen to

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

be completed using each prototype was randomly chosen. The participant was asked to explore the corporate memory and make a list of all potentially reusable items found. After the task was completed, the participant was asked to answer some questions about each of the items listed, e.g.: “why did the design team choose a building material?” For each task, we measured:

o Recall score: the proportion of potentially reusable items as judged by a

human expert that were actually found by the participant.

o Context score: the proportion of questions about helpful items that could be

correctly answered by the participant.

o Time taken: the time taken to feel confident that all helpful items had been

found.

• Repository size. The exploration tasks and the retrieval tasks were run first with a

large repository, and then repeated with a small repository in the cases of CoMem and the Outline Tree. To limit the length of time required to complete the test, small repository tasks were not conducted with the Hit List.

• Questionnaire. The participant was asked to complete three questionnaires, one for

each of the prototypes, asking them about their subjective reactions to the prototype. • Debrief. Short, informal interview.

For each test participant, the order of testing the three prototypes was randomly chosen, in an attempt to eliminate the effects of learning and increased familiarity with the data.

Pilot Corporate Memory and Preliminary User Tests

A pilot corporate memory of real construction project content was put together for the purpose of these preliminary user tests. The large repository tests described in this paper were conducted on a pilot corporate memory consisting of 10 project objects (i.e. ten construction projects in the corporate memory), 35 discipline objects (i.e. about 3 or 4 disciplines or building subsystems such as “structure” or “electrical” per project), and 1036 components (i.e. around thirty components per building subsystem). Of the 1036

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

component objects, approximately 30% were annotated with note objects. The small repository tests were conducted with the smallest possible subset of projects in the large repository that would include all the data required for the exploration tasks.

Following the completion of the tasks, questionnaires were given to test subjects to solicit subjective feedback on each of the three prototypes. The questionnaire was loosely based on Brooke (1996). The CoMem questionnaire is shown in

Figure 9. Similar questionnaires were handed out for the Hit List and Outline Tree, but with questions omitted that are specific to CoMem.

Attention was paid to ensure that the repositories were densely populated in several areas related to each exploration task. For example, if the exploration task involved roof design, care was taken to ensure that at least 5 or 6 projects had rich content related to roof design: annotations, hyperlinked documents, team interactions, images, design alternatives, and so on.

There was a pool of six standard exploration tasks from among which a task was randomly chosen for each prototype and repository size. Those were: • Roof design • Post-tensioned slab • Shear walls • Atrium • Elevator • HVAC System

Figure 10 shows the first exploration task where the user is working on a roof design.

Preliminary Retrieval Results

To illustrate the use of this evaluation framework, results from the preliminary user tests are presented below.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

The time to complete a simple retrieval task is shown in Figure 11. The best performance in the case of retrieval was achieved by the Outline Tree which allowed retrieval tasks to be completed in the shortest time. The Outline Tree is effective for retrieval in the same way that binary search is effective for sorted arrays. By first selecting the project and discipline from much smaller lists than the list of all component objects in the corporate memory, the list of components that need to be visually scanned is greatly reduced. Further research should investigate the effectiveness of the Outline Tree for hierarchies with varying branching factors.

Pirolli et al. (2000) conducted closely related evaluations of visualizations of large tree structures. They did not include treemaps in their analysis, but compared Windows Explorer (equivalent to the Outline Tree) to hyperbolic trees. They conclude that the performance of the hyperbolic tree, because it attempts to crowd more data into a compressed space, is sensitive to “information scent” (the labels or colors used to guide the user to the appropriate piece of information).

After the Outline Tree, CoMem allowed retrieval tasks to be completed in the next shortest time. In spite of the fact that it was not developed with retrieval tasks in mind, CoMem still provides support for such tasks. Future research will investigate the role CoMem can play in retrieval tasks.

Because we are comparing more than two independent samples (in our case three, for each of the three prototypes) the significance of these results was tested using the one-way analysis-of-variance method (ANOVA, Chapter 10 in Dowdy et al. 2004) to determine whether the differences in retrieval time observed were due to sampling errors or rather due to differences in the performance of the prototypes. Indeed, the results do appear to be significant at the 0.01 and 0.05 levels. This only confirms that the three samples were taken from three populations among which there is at least one inequality. Nothing can be said about the performance of CoMem specifically. In fact, it can be seen from Figure 11 (and the displayed confidence intervals) that the high performance (short

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

retrieval time) of the Outline Tree prototype is largely responsible for the positive analysis-of-variance result.

Preliminary Exploration Results

The average time to complete an exploration task was comparable for the three prototypes CoMem, Outline Tree, and Hit List (Figure 12), even though, as discussed below, the user’s performance in terms of recall score and context score varied to a greater extent from tool to tool. Again, analysis-of-variance was performed. The differences between he three prototypes in exploration time was not found to be significant.

Figure 13 shows the fraction of relevant items successfully recalled by the test participants during exploration tasks (Baeza-Yates and Ribeiro-Neto 1999 gives a discussion of the measurement of recall). CoMem performed well in exploration recall. The Outline Tree had the poorest performance in exploration recall. This can be explained by the fact that in most cases reusable items were buried deep inside the hierarchy (i.e. at the component level) and left very little information scent at the higher levels that appear initially in the Outline Tree. Information scent is the user’s perception of the value, cost, or access path of information sources. In the Outline Tree, projects and disciplines are displayed first and must be expanded by the user to display their component children. This requires that, for a relevant component, that component’s parent discipline and grandparent project objects must also be relevant in order to encourage the user to expand those sub-trees and find the reusable component. This is rarely the case in the CoMem relevance measure.

Analysis-of-variance did not show that the differences in exploration recall was significant.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

Figure 14 shows the fraction of contextual questions that could be answered correctly by test participants about the items they retrieved. CoMem performed better than the Outline Tree and Hit List and had a slightly larger confidence interval. Most of the contextual questions were based on interactions between the designers, and the resulting version history of the item in question. The CoMem Evolution History Explorer was rated very highly by test participants. It was used during exploration tasks much more extensively than the Project Context Explorer, and was repeatedly praised by the participants during the debriefing interview.

Analysis-of-variance shows that this result is significant at the 0.05 level (but not 0.01). This means that there is a significant difference in context score between the three prototypes, and Figure 14 shows that CoMem was the highest performer in this respect.

Questionnaire Results

Figure 15 shows the subjective opinions of the test participants about CoMem, the Outline Tree, and Hit List. For the questions regarding general usability characteristics (learnable, complicated, cumbersome), which are not displayed in Figure 15, CoMem attained comparable scores to the Hit List and Outline Tree. Even though CoMem uses radically different interaction techniques, whereas the other two prototypes are tools with which any average computer user would be very familiar and experienced.

CoMem received higher scores particularly for questions 8-12 (

Figure 9). Questions 11 and 12 are the main metrics for the extent to which external reuse is effective: does the user feel that if he/she had that prototype in his/her work, he/she would reuse designs more frequently and more appropriately (last two questions in Figure 15).

Questions 8, 9, and 10 (first three questions in Figure 15) measure the user’s perceived ability to find and understand:

• “I would feel very confident reusing some content that I found using this system.” • “I had a good understanding of the items I was exploring.”

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

• “I felt that I was able to find all potentially reusable items in the corporate memory in

the given time.”

The high score awarded to CoMem in these questions supports the higher recall and understanding performance measures achieved by the test subjects when using CoMem for exploration tasks.

The users were asked to rate the three CoMem modules: the Overview, the Project Context Explorer (PCE), and the Evolution History Explorer (EHE) (Figure 16). The highest-rated module is the Overview, which validates the claim that providing a succinct overview of the entire corporate memory is extremely valuable, and that a treemap is a good visualization for this purpose. The Evolution History Explorer was rated very highly. By observing the users during the tests, it is clear that this module enables the users to reconstruct the evolution of the designs and understand the rationale behind this evolution much more effectively than a list of versions or displays of single versions one at a time. The lowest-rated module, although by very slightly, is the Project Context Explorer. Many users found it unclear because it shows the same items as those in the Overview, but positioned and colored differently. Further development is needed to couple the Project Context Explorer more tightly with the Overview, so that a change in one display triggers a corresponding change in the other. It is suspected that advanced users of CoMem would make more use of the Project Context Explorer.

Discussion of Results

At a macro level, the results test the hypothesis of this research. Traditional tools do not support the ability to find and understand and traditional tools do not lead to effective reuse. CoMem supports the ability to find and understand and CoMem leads to effective reuse. This supports the claim that the steps of find and understand lead to effective reuse, as shown in Figure 17.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

At a micro level, a comparison between the metrics from CoMem and those from traditional tools helps to identify the specific circumstances under which CoMem performs better than traditional tools. The first variable in this evaluation is the type of task: exploration versus retrieval. CoMem performs better in exploration scenarios rather than retrieval. The other variable that was introduced into the evaluation is repository size.

Figure 18 shows the hypothesized effect of repository size on the performance of CoMem and traditional tools. Figure 19 and Figure 20 show the actual effects observed on exploration time and retrieval time. In the case of exploration (Figure 19), the size of the repository seems to have little effect. A more subtle aspect such as the amount of text that needs to be read to complete the task is more likely to have an effect on exploration time than the relatively simple count of the number of items in the repository. In the case of retrieval (Figure 20) the results are more similar to the hypothesized effect. As the repository size is increased, the performance of CoMem is assumed to stay approximately constant (logic dictates that the slight decrease in retrieval time for larger repositories must be due to statistical sampling errors), while the performance of the Outline Tree begins to deteriorate (takes more time for the larger repository). By simple extrapolation, it can be imagined that a point would be reached beyond which CoMem outperforms the Outline Tree.

Conclusions

The usability evaluation results presented in this paper support the hypothesis of this research, that the ability to find and understand does indeed lead to more effective reuse. CoMem offers greater support for finding and understanding than traditional tools, and reuse using CoMem is consistently rated to be more effective by test participants. It is noteworthy that test participants were chosen who had generally little engineering design experience, but were experiences users of information technology and software. The results presented in this paper therefore apply to young engineers at the beginning of their careers, which is the population for whom CoMem was designed.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

More importantly for the current discussion, his research makes a methodological contribution through the development of a useful framework for studying and evaluating corporate memory reuse tools such as CoMem. The same data can be explored using different interfaces. Hit List, Outline Tree, and CoMem cover the spectrum of information interfaces, from traditional to innovative. Search engines and expandable/collapsible folder trees can be used to represent traditional information interfaces. The important dimensions of the evaluation space are the size of the repository and the type of task. One third dimension that we were not able to test in this first round of experiments (but defer for future research) is the effect of the user’s familiarity with the contents of the repository.

Future work is needed to investigate the effect of familiarity with the contents of the repository. Innovative reuse tools such as CoMem must support novice users who are unfamiliar with the contents of the corporate memory as well as advanced users who are able to formulate explicit queries. In practice it will be impossible to be completely familiar with the corporate memory because it is constantly growing and evolving.

It is hypothesized that traditional tools rely on the user’s familiarity with the data to formulate explicit queries. CoMem should be less sensitive to familiarity and therefore provide greater support for novice users who tend to be unacquainted with corporate repositories (Figure 21). The effect of familiarity can be studied by conducting two rounds of testing. Test participants would be chosen who are unfamiliar with the data. They would be asked to complete one set of tasks, then given a “familiarity-building exercise”, and then asked to complete a second round of tasks. Their performance in the first and second sets of tasks would be compared to investigate whether the effectiveness of the tool is dependent on familiarity with the data.

The three dimensions of size, task, and familiarity together define a three dimensional space Figure 22). It is suspected that CoMem is particularly supportive in one corner of this space, while traditional tools support the opposite corner.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

Acknowledgements

This research was funded by the Stanford Center for Integrated Facility Engineering (CIFE) and the Project-Based Learning Laboratory (PBL). The authors would like to thank Kushagra Saxena for his assistance to conduct the user tests, and are grateful to the research community at CIFE and all test participants.

References

Baeza-Yates R. and Ribeiro-Neto B., 1999. Modern Information Retrieval, Addison-Wesley, Harlow, UK.

Brin S. and Page L., 1998. “The anatomy of a large-scale hypertextual web search engine”,

Computer Networks and ISDN Systems, Volume 30, Issue 1-7, pages 107-117. Brooke J., 1996. “SUS: A quick and dirty usability scale”, Usability Evaluation in

Industry, (Jordan P. W., Thomas B., Weerdmeester B. A., and McClelland I. L., Eds.), Taylor and Francis, London, UK, pages 189-194.

Bruls D. M., Huizing K., and van Wijk J. J., 1999. “Squarified treemaps”, Data

Visualization 2000, Proceedings of the Second Joint Visualization Symposium organized by the Eurographics and the IEEE Computer Society Technical Committee on Visualization and Graphics (TCVG), (de Leeuw W. and van Liere R., Eds.), Springer-Verlag, Vienna, Austria, pages 33-42.

Culley S. J., 1998. “Design reuse of standard parts”, Proceedings of the Engineering

Design Conference on Design Reuse, Design Reuse - Engineering Design

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

Conference '98, Sivaloganathan S. and Shahin T. M. M., Eds., UK: J. Wiley, ISBN: 1860581323, pages 77-88.

Demian, P. and Fruchter, R., 2005, “Measuring Relevance in Support of Design Reuse

from Archives of Building Product Models.” ASCE Journal of Computing in Civil Engineering, volume 29, issue 2, pp. 119-136

Demian, P. and Fruchter, R., under-review, “An Ethnographic Study of Design Knowledge

Reuse in the Architecture, Engineering and Construction Industry” Submitted to Research in Engineering Design

Dowdy S., Wearden S., and Chilko D., 2004. Statistics for Research. Third Edition. John

Wiley & Sons, Hoboken, New Jersey.

Finger S., 1998. “Design reuse and design research – Keynote paper”, Proceedings of the

Engineering Design Conference on Design Reuse, Design Reuse - Engineering Design Conference '98, Sivaloganathan S. and Shahin T. M. M., Eds., UK: J. Wiley, ISBN: 1860581323, pages 3-10.

Fruchter R. and Demian P., 2002. “CoMem: Designing an interaction experience for reuse

of rich contextual knowledge from a corporate memory”, Artificial Intelligence for Engineering Design, Analysis and Manufacturing (AI EDAM), Volume 16, Issue 3, pages 127–147.

Fruchter R., 1996. “Conceptual, collaborative building design through shared graphics”,

IEEE Expert: Intelligent Systems, AI in Civil and Structural Engineering, Volume 11, Number 3, pages 33-41.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

Fruchter R., Reiner K., Leifer L., and Toye G., 1998. “VisionManager: A computer

environment for design evolution capture”, Journal of Concurrent Engineering: Research and Applications (CERA), Volume 6, Number 1, pages 71-84.

Kuffner T. A. and Ullman D. G., 1990. “The information requests of mechanical design

engineers”, Proceedings of the Second International Conference on Design Theory and Methodology (DTM), Sponsored by ASME Design Engineering Technical Conferences (DETC), pages 167-174.

Leifer L., 1997. Design Project Laboratory (ME310), Stanford University, Course Notes. Pirolli P., Card S. K., and Van Der Wege M. M., 2000. “The effect of information scent on

searching information visualizations of large tree structures”, Proceedings of the ACM Working Conference on Advanced Visual Interfaces (AVI), Sponsored by ACM-SIGCHI, ACM-SIGMM, SIGCHI Italy , pages 161-172.

Regli W. C., Hu X., Atwood M., and Sun W., 2000. “A survey of design rationale

systems: Approaches, representation, capture and retrieval”, Engineering with Computers, Volume 16, Numbers 3-4, Springer-Verlag, Vienna, Austria, pages 209-235.

Reiner K. and Fruchter R., 2000. “Project memory capture in globally distributed facility

design”, ASCE Proceedings of the Eighth International Conference on Computing in Civil and Building Engineering (ICCCBE-VIII), Volume 2, pages 820-827. Schank R. C., 1990. Tell Me a Story: A New Look at Real and Artificial Memory, Scribner,

New York, NY.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

Schön D. A., 1983. The Reflective Practitioner: How Professionals Think in Action, Basic

Books, New York, NY.

Shneiderman B., 1999. “Supporting creativity with advanced information-abundant user

interfaces”, University of Maryland, College Park, MD, The Institute for Systems Research (ISR), Technical Report 1999-73. Also available from Frontiers in Human-Centred Computing, Online Communities and Virtual Environment, 2001, (Earnshaw R., Guedj R., Van Dam A., and Vince J., Eds.), Springer-Verlag, London, UK, pages 469-480.

Ye Y. and Fischer G., 2002. “Supporting reuse by delivering task-relevant and

personalized information”, Proceedings of the Twenty-Fourth International Conference on Software Engineering (ICSE), Orlando, Florida, Sponsored by the Association for Computing Machinery, ACM Special Interest Group on Software Engineering, IEEE Computer Society/Technical Council on Software Engineering, pages 513-523.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

LIST OF FIGURES FIGURE 1: THE FREQUENCY WITH WHICH PROFESSIONALS REUSE DESIGNS AND DOMAIN EXPERTISE FROM

VARIOUS KNOWLEDGE SOURCES.................................................................................................................27

FIGURE 2: THE FREQUENCY WITH WHICH VARIOUS OBSTACLES TO DESIGN REUSE OCCUR IN CURRENT PRACTICE.

.....................................................................................................................................................................28 FIGURE 3: COMEM USER INTERFACE. TRANSFORMATION FROM OBSERVED REUSE STEPS TO USER

INTERACTIONS.............................................................................................................................................29

FIGURE 4: COMEM MODULES: (A) THE OVERVIEW PROVIDES A SUCCINCT SNAPSHOT OF THE ENTIRE

CORPORATE MEMORY WITH EACH ITEM COLORED ACCORDING TO ITS RELEVANCE TO THE DESIGNER’S CURRENT DESIGN TASK, (B) THE EVOLUTION HISTORY EXPLORER DISPLAYS THE VERSIONS OF ANY ITEM SELECTED FROM THE OVERVIEW AND ALLOWS THE USER TO EXPLORE THE EVOLUTION OF THAT ITEM BEFORE REUSING IT; (C) THE PROJECT CONTEXT EXPLORER ALLOWS THE USER TO EXPLORE RELATED ITEMS IN THE CORPORATE MEMORY TO BETTER UNDERSTAND THE ITEM BEING REUSED, (D) A SPECIFIC VERSION SELECTED FROM THE EVOLUTION HISTORY EXPLORER IS DISPLAYED IN A SPECIAL CONTENT VIEWER........................................................................................................................................................30

FIGURE 5: TOOL SUCH AS COMEM CAN BE ASSESSED BY COMPARING THEM TO TRADITIONAL TOOLS IN

CURRENT PRACTICE.....................................................................................................................................31

FIGURE 6: THE COMEM INTERFACE. (A) OVERVIEW (B) PROJECT CONTEXT EXPLORER (C) EVOLUTION

HISTORY EXPLORER....................................................................................................................................32 FIGURE 7: OUTLINE TREE PROTOTYPE. (A) OUTLINE TREE WITH GENERIC ICONS. (B) OUTLINE TREE WITH

COLORED ICONS TO INDICATE RELEVANCE, USED FOR EXPLORATION TASKS. (C) VERSION TABLE WHICH LISTS ALL VERSIONS OF AN ITEM IN A TABLE..............................................................................................33

FIGURE 8: HIT LIST PROTOTYPE. (A) MAIN PAGE OF HIT LIST FOR SEARCHING THE CORPORATE MEMORY. (B)

WEB BASED VERSION TABLE. (C) WEB VIEW OF AN ITEM SELECT FROM HIT LIST.....................................34 FIGURE 9: THE COMEM QUESTIONNAIRE, LOOSELY BASED ON BROOKE (1996). TEST SUBJECTS WERE GIVEN

SIMILAR QUESTIONNAIRES FOR HIT LIST AND OUTLINE TREE, BUT WITH QUESTIONS 13, 14, AND 15 OMITTED......................................................................................................................................................35

FIGURE 10: A SAMPLE EXPLORATION TASK, WHERE THE USER IS SEARCHING FOR REUSABLE ITEMS FOR ROOF

DESIGN.........................................................................................................................................................36

FIGURE 11: TIME TO COMPLETE A SIMPLE RETRIEVAL TASK WITH 90% CONFIDENCE INTERVALS......................37 FIGURE 12: TIME TO COMPLETE AN EXPLORATION TASK WITH 90% CONFIDENCE INTERVALS............................38 FIGURE 13: RECALL SCORE DURING EXPLORATION TASKS WITH 90% CONFIDENCE INTERVALS.........................39 FIGURE 14: CONTEXT SCORE DURING EXPLORATION TASKS WITH 90% CONFIDENCE INTERVALS......................40 FIGURE 15: A SELECTION OF THE QUESTIONNAIRE RESULTS................................................................................41 FIGURE 16: USER RATING OF USEFULNESS OF THE THREE COMEM MODULES.....................................................42 FIGURE 17: MACRO EVALUATION TO TEST THE HYPOTHESIS OF THIS RESEARCH.................................................43

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

FIGURE 18: A DIAGRAMMATIC REPRESENTATION OF THE HYPOTHESIZED EFFECT OF REPOSITORY SIZE ON THE

PERFORMANCE OF INFORMATION TOOLS.....................................................................................................44

FIGURE 19: THE EFFECT OF REPOSITORY SIZE ON EXPLORATION TIME WITH 90% CONFIDENCE INTERVALS

DISPLAYED...................................................................................................................................................45

FIGURE 20: THE EFFECT OF REPOSITORY SIZE ON RETRIEVAL TIME WITH 90% CONFIDENCE INTERVALS

DISPLAYED...................................................................................................................................................46

FIGURE 21: A DIAGRAMMATIC REPRESENTATION OF THE HYPOTHESIZED EFFECT OF FAMILIARITY WITH THE

DATA ON THE PERFORMANCE OF INFORMATION TOOLS..............................................................................47

FIGURE 22: THREE-DIMENSIONAL SPACE DEFINED BY SIZE, TASK, AND FAMILIARITY. COMEM AND

TRADITIONAL TOOLS SUPPORT OPPOSITE CORNERS OF THIS SPACE.............................................................48

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

Frequently21.5Reuse DesignsReuse DomainExpertiseInternalmemoryExternalReositoryMemtorsFrequencyOccasionally10.5Rarely0Source for Reuse

Figure 1: The frequency with which professionals reuse designs and domain expertise from various

knowledge sources.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

Frequently21.5FrequencyOccasionally10.5CAN'TFINDDON'TKNOWWHATTOLOOKFORDON'TKNOWABOUTDESIGNERDON'TKNOWABOUTPROJECTObstacle toReusDON'TKNOWRATIONALERarely0

Figure 2: The frequency with which various obstacles to design reuse occur in current practice.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

Reuse step Find reusable item

“overview first, zoom and demand”

User interaction Overview Evolution history explorer

Project context explorer

Explore item’s evolution history filter, and then details-on-Explore item’s project context

Figure 3: CoMem user interface. Transformation from observed reuse steps to user interactions.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

(a) (c)

(b) (d)

Figure 4: CoMem Modules: (a) The Overview provides a succinct snapshot of the entire corporate memory with each item colored according to its relevance to the designer’s current design task, (b) The Evolution History Explorer displays the versions of any item selected from the Overview and allows the user to explore the evolution of that item before reusing it; (c) The Project Context Explorer allows the user to explore related items in the corporate memory to better understand the item being reused, (d) a specific version selected from the Evolution History Explorer is displayed in

a special content viewer.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

Designer uses CoMem Leads toDesigner can find and understand (externally) metricscomparemetricsLeads toExternal reuse is effective metricscomparemetricsDesigner uses Traditional ToolsLeads toDesigner can find and understand (externally) Leads toExternal reuse is effective Figure 5: Tool such as CoMem can be assessed by comparing them to traditional tools in current

practice.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

a) b)

Figure 6: The CoMem interface. (a) Overview (b) Project Context Explorer (c) Evolution History

Explorer.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

(a) (b)

(c)

Figure 7: Outline Tree prototype. (a) Outline Tree with generic icons. (b) Outline Tree with colored icons to indicate relevance, used for exploration tasks. (c) Version table which lists all versions of an

item in a table.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

(a) (b)

(c)

Figure 8: Hit List prototype. (a) Main page of Hit List for searching the corporate memory. (b) Web

based version table. (c) Web view of an item select from Hit List.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

CoMem Questionnaire 1. I think that I would like to use this system frequently 2. I found the system unnecessarily complex 3. I thought the system was easy to use 4. I would imagine that most people would learn to use this system very quickly 5. I found the system very cumbersome to use 6. I felt very confident using the system 7. I needed to learn a lot of things before I could get going with this system 8. I would feel very confident reusing some content that I found using this system. 9. In the exploration tasks I completed using this system, I felt I had a good understanding of the items I was exploring. 10. In the exploration tasks I completed using this system, I felt that I was able to find all potentially reusable items in the corporate memory in the given time. 11. If I had this system in my work, I would reuse content from previous projects more frequently than I do currently. 12. If I had this system in my work, I would reuse content from previous projects more appropriately than I do currently. 13. I think the Overivew / Map would be very useful in my work. 14. I think the Storyteller / Evolution history would be very useful in my work. 15. I think the Fisheye Lens / Project context would be very useful in my work. COMMENTS: Strongly Disagree Strongly Agree 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 Figure 9: The CoMem questionnaire, loosely based on Brooke (1996). Test subjects were given similar questionnaires for Hit List and Outline Tree, but with questions 13, 14, and 15 omitted.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

TASK I = ROOF DESIGNProblemIndex=136ReusablePacific 1999Pacific 2001Pacific 2001>Structure>449-RoofPacific 2001>Structure>444-PT SlabsPacific 2001>Construction>481-Roof SystemWhat material was used for this roof? (Metal panels)Why? (Lighter than concrete, simpler connections than steel, ease of construction)What did the roof look like over the auditorium? (Pyramid)Why will the roof be expensive? (Because of the curvature)Which other building component had to be coordinated with the roof? (PT slabs)Can you name some architectural concepts that were considered? (Gable, mansard/French gable)What was the CM's feedback on the architect's ideas? (Complicatated, hieght restriction, snow/rain accumulating)What materials were considered? (Timber, steel)What equipment will go on the roof? (Air handling unit)What impact will this have on the structure? (Larger columns)Coral 2002Coral 2002>Structure>890-roof1(columns)Can you describe the roof system? (prefab roof truss, elevated on columns and beams, prestressed roof slab)Why was the roof truss elevated? (Natural ventilation, aesthetics)Reusable ItemsXContext QuestionsWave 2001Wave2001>Arch>366-RoofWave2001>Eng>363-Roof structureWave 2001>Construction>404-Air Handling UnitCoral 2002>Structure>888-roof1(beams)Coral 2002>Structure>892-slab1(roof)Coral 2002>Structure>894-rooftrusses1(rooftrusses)Ridge 2002XIsland 2002XBay Saint LouisBSL>Arch>25-Ballroom (roof)BSL>Arch>17-Hotel roof (roof)BSL>Arch>24-Hotel Penthouse (roof)Grand Californian HotelGCH>Structure>59-Disney Store Roof (Steel dome)GCH>Structure>46-Area1 roof (roof truss)GCH>Structure>49-Area2 roof (roof truss)GCH>Structure>55-Area3 roof (roof truss)San Rafael Bridge RetrofitXImaginary HotelX

Figure 10: A sample exploration task, where the user is searching for reusable items for roof design.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

90807060Time (seconds)50403020100Hit ListOutline TreePrototype

Figure 11: Time to complete a simple retrieval task with 90% confidence intervals.

CoMem37

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

252015Time (minutes)1050Hit ListOutlineTreePrototypeCoMem

Figure 12: Time to complete an exploration task with 90% confidence intervals.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

0.800.700.600.50Recallscore0.400.300.200.100.00Hit ListOutline TreePrototypeCoMem

Figure 13: Recall score during exploration tasks with 90% confidence intervals.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

0.80.70.60.5Contextscore0.40.30.20.10Hit ListOutline TreePrototypeCoMem

Figure 14: Context score during exploration tasks with 90% confidence intervals.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

765Rating (1=Disagree,7=Agree)432Confidentreusingcontent?Foundall?1OutlineTreeHitListCoMemReusemorefrequently?Understoodcontent?Reusemoreappropriately?

Figure 15: A selection of the questionnaire results.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

765Rating (1=Disagree,7=Agree)4321OverviewisusefulEHEisusefulPCEisuseful

Figure 16: User rating of usefulness of the three CoMem modules.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

Designer uses CoMem Leads toDesigner can find and understand (externally) metricscomparemetricsLeads toExternal reuse is effective metricscomparemetricsDesigner uses Traditional ToolsLeads toDesigner can find and understand (externally) Leads toExternal reuse is effective MEASUREMENTS OF METRICS SUPPORT THIS HYPOTHESISDesigner uses CoMemLeads toDesigner can find and understand (externally) Leads toExternal reuse is effective Figure 17: Macro evaluation to test the hypothesis of this research.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

Time to find CoMemTraditional toolsRepositorysize

Figure 18: A diagrammatic representation of the hypothesized effect of repository size on the

performance of information tools.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

2520ExplorationTime(minutes)151050SmallLargeCoMemOutlineTreeRepositorySize

Figure 19: The effect of repository size on exploration time with 90% confidence intervals displayed.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

90CoMem807060Retrieval Time (seconds)50403020100SmallRepositorySizeLargeOutline Tree

Figure 20: The effect of repository size on retrieval time with 90% confidence intervals displayed.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

Ease of finding CoMem Traditional tools Familiarity with

repository

Figure 21: A diagrammatic representation of the hypothesized effect of familiarity with the data on

the performance of information tools.

Demian, P. and Fruchter, R., 2006 “A Methodology for Usability Evaluation of Corporate Memory Design Reuse

Systems.” ASCE Journal of Computing in Civil Engineering, Volume 20, Issue 6, pp. 377-389.

Size of therepository Large (~ 10,000 items)This corner is wheretraditional tools focusand provide support1Time UseExplorationType ofFinding Task100Time UseThis corner is where traditional tools breakdown and CoMem canadd valueSmall (~ 100 items)RetrievalFamiliaritywith therepositorythst

Figure 22: Three-dimensional space defined by size, task, and familiarity. CoMem and traditional

tools support opposite corners of this space.

因篇幅问题不能全部显示，请点此查看更多更全内容

查看全文

全部栏目

A Methodology for Usability Evaluation of Corporate Memory Design Reuse Systems