 |
ExperienceFactory.com
|
|---|
Experience Factory Management explained. Jeff Thomson
Introduction
Software development has become more demanding, time-to-market has shortened, application complexity has increased and quality expectations are higher. In order to meet these demands the practice of reusing sections of software code multiple times on the same project, or on multiple projects was developed. This practice of reuse can be to reuse of all kinds of knowledge in software development. This is the basis for a framework for building software competencies and maintaining project information
known as Experience Factory Management (EF).
According to Victor Basili, a noted author of Experience Factory research, “As software engineering is a fairly young discipline, its technologies are not as mature as those of other engineering disciplines. On the other hand, requirements to software systems are steadily growing concerning their complexity, performance, etc. Therefore, the capturing and reuse of explicit software development know-how is essential for continuous improvement. What represents relevant software know-how differs among companies regarding their environmental context and their specific goals. Therefore, organization-specific software know-how, which comprises the core of a mature software organization, has continuously to evolve based on experiences gathered in software projects. In that endeavor, software organizations require support in collecting experiences from their projects, packaging those experiences (e.g., build models from empirical data, formalize informal knowledge), and in validating and reusing experiences in future projects. The support of these tasks requires comprehensive learning information systems which are based on explicit knowledge representations.” [Basili 1996]
QIP - the Quality Improvement Paradigm
One aspect of Experience Factory Management is the approach, generally known as the Quality Improvement Paradigm QIP approach, which uses the EF framework. [Basili 1988] Its goal is to develop and evaluate a set of techniques, methods, and tools that support reliable and low-effort customization of complex domain-specific software systems.
Briefly defined, QIP is “An iterative, goal-driven framework for continuous improvement of software development. This QIP is a closed-loop process which includes steps for planning, executing, and evaluating improvement to software development environments, as well as for incorporating experience gained from improvement efforts into future development.” [Hood 1997]
QIP consists of the following six steps:
1. Characterize the current project and its environment with respect to existing models and metrics
2. Set quantifiable goals for successful project performance and improvement
3. Choose the appropriate measurement models and supporting methods and tools for this project
4. Execute the processes, construct the products, collect and validate the prescribed data, and analyze it to provide real-time feedback for corrective action
5. Analyze the data to evaluate the current practices, determine problems, record findings, and make recommendations for future projects
6. Package the experience as updated and refined models and other forms of structured knowledge gained from these and subsequent projects and save it in an experience base to be reused in future projects.” [Pihlava]
Knowledge gained while using the Experience Factory is continuously analyzed and re-stored into the Experience Base (EB) portion of the EF. “Once the Knowledge is stored it is called experience and covers models (such as process models, product models, quality models), instances (such as process traces, products, measurement data, techniques, tools), and qualitative experience (such as lessons learned). Any kind of artifact representing experience is known as an experience element.” [Feldmann 1997]
An important aspect of the Experience Factory is the organizational memory. “Organization-specific software know-how is essential for continuous improvement . . . Organization-specific software know-how differs among companies regarding their environmental context and their specific goals . . . For the continuous build-up of software knowledge in an organization, the experience factory approach has been proven to be a successful solution. The experience factory organization complements the project organization by enabling the continuous learning on software development from examples of individual software projects and communication of software knowledge across the organization.” [Tautz 1998]
Experience Factory can improve Software Development in a variety of ways. I will overview some of the ways that it can be used: Training, Applying a new Technology, and Improving on Technology.
Training
Since software engineering is a relatively new science and many of the methods are changing quickly, software engineers are seldom trained on all or even most of the popular methods. In addition to popular software engineering methods, a software project manager frequently comes upon the need for an obscure practice that is required by a particular project. For example, a project scope document may be required of a software development team by a client. This particular project team may have never needed to do a project scope document before, or only rarely. Information on completing this documentation may be contained in the Experience Base. Training on such methods and practices can be circumvented or at least significantly shortened.
“When a new technology is introduced, people who are to apply it, have to be trained. Otherwise, the technology may be applied incorrectly, and the results may not be as good as expected. . . Fortunately, not everybody has to know everything about a technology. In case of software inspections, everybody needs to know what the goals of inspections are and how inspections work in principle. But not everybody needs to know how to read a document systematically (only those who are supposed to read the inspection document need to).” [Feldmann 1989]
Experience Factory can suit this type of need very well. The people who need to be extensively trained can take the necessary training, everyone else involved in the project, who need only rudimentary training, can use the Experience Factory information to get the amount of training they require without the loss of time extensive training would take. Using this method does require that an administrator of the Experience Factory, or some other expert insures that the right information is made available and can easily be found in the Experience Base.
Applying a Technology
It isn't’t enough just to train members of a team on new technologies, individuals applying technologies have to continually keep their knowledge up to date. “If a technology has not been applied for quite some time by an individual, the individual might need help on particular topics. This requires the availability of a reference guide.” [Feldmann 1989]
When a new technology is needed, the development team (or individuals on the team) learns the new technology and tracks that knowledge in the Experience Factory. Then future team members or members of other teams can gain the information they need with regard to the new technology from the Experience Factory.
Improving Technology
“Experiential knowledge . . . can be used to evolve a technology continuously.” Lessons learned is an example of this. Anything learned from past software projects could be saved. “Lessons refer to certain elements. Once a list of improvement suggestions exist for an element description, the element description should be updated. Otherwise a novice would have to read the element description and all improvement suggestions in order to apply the state of the art.” [Feldmann 1989]
Lessons Learned is one of the most useful parts of the Experience Factory. Many times development teams make the same mistakes or similar mistakes to development teams that have gone before. Sometimes they repeat mistakes within the same development team. When the Experience Factory records Lessons Learned it may be able to prevent many of these repeat mistakes – saving a lot of time for the development team in working out problems that should not have happened in the first place. In order for the Lessons Learned section to be of full use, though, the development team must be active in reading and applying the case histories recorded in the Experience Base.
Requirements
There are several types of Experience Factories in use or under construction. Tautz claims that: “For an Experience Factory to meet the needs of an organization it must:
· Define all artifact types to be stored in the experience base as well as the kinds of relationships between the artifacts.
· Guide software engineers to specify the knowledge to be retrieved.
· Facilitate the maintenance of an experience base and
· Allow similarity-based retrieval.” [Tautz 1998]
Although I feel these are necessary, the list is too limited to be the only guidelines an organization should consider when choosing an Experience Factory. Organizations must also make the same considerations they would in evaluating any software development tool. For example, ease of use, support, documentation, etc.
Strategies
The experience factory can not only select what experience should be stored in the experience base, it can also define strategies for gaining the experience. [Althoff 1998 -3]
“In order to operationalize reuse and learning and provide tool support for it, a process model is required.” The organizational structure contains a relationship between organizational units – the information flow between the organization units – and the number of processes and organizational units in a typical software engineering business unit. [Althoff 1998 -2]
As part of the building process, studies are performed to best comprehend organization and process related issues. These studies can be done by the Experience Factory Manager or the Development Team leader. These issues may be things such as Training needs, whether or not all the team members understand the overall goals, whether or not the individual team members are communicating properly, if they are learning from the experienced team members, and many other issues that may be helped by an Experience Factory. Agent Dependency Model techniques are used as part of this step. The idea behind the studies is to understand and identify issues facing the software development organization.
After the studies are done, a model is developed representing the Software Organization. This model is called an Agent Dependency Model and is used to show all aspects of the Software Organization and all its components. It captures the identities and properties of the organizational context of the Software Organization Team’s process and can help with understanding the cause and effect mechanisms leading to the problems the organization faces. This qualitative data has to be complemented with quantitative data. [Melo 1995]
After this first step is completed, the outputs performed are then used to justify and define a relevant measurement program. This measurement program includes what data to collect, when to collect the data, and how to conduct the data collection. This will facilitate interpretation of the data coming from the program because the collection process increases the level of understanding the process in the first place.
GQM (Goal/Question/Metric Paradigm)
Part of the process of the Experience Factory is Quality Assurance. In order to insure quality a good measurement must be in place. According to Victor Basili, a Goal-Oriented approach is both broad and flexible enough to support the measurement activities required for software development involved in a Quality Improvement Paradigm for developing core competencies and an organizational infrastructure inherent in an Experience Factory.
The QIP mentioned earlier uses GQM paradigm and the concept of the Experience Factory to improve software development. CQM is based on the idea that measurement should be goal-oriented, in other words, that all data collection in a measurement program should be based on a documented rationale. CQM involves a hierarchy at three levels: The conceptual level – in which a goal is defined for an object, The operational level – in which a set of questions is used to define the goal in a quantitative way, and quantitative level – in which a set of data is associated with every question so that it can be answered in a quantitative way. [Pihlava 1997]
This Goal-Oriented approach he calls GQM or Goal/Question/Metrics. “The GQM approach supports the operational definition of project goals during the planning phase, supports the analysis of measurement data and its feedback into the ongoing project, and allows the explicit capturing of measurement plans for reuse in future projects.” [Hood 1997]
This GQM method was developed by Basili and his research group at the University of Maryland in cooperation with NASA Software Engineering Laboratories. The method is based on a process in which the software and quality engineers and managers define the goals that the software is to achieve and then refine those goals into a set of questions to be answered. Then they identify the metrics that should be provided in order to answer the questions. Thus the GQM provides a top-down approach to the definition of metrics and the interpretation of the measured data is done in a bottom-up manner.
There are three basic parts of the GQM measurement plan.
1. A set of goals which are defined according to a template using the following five dimensions: the target of measurement, purpose of measurement, the measured property of the target (called the quality focus), the subject of measurement (the viewpoint), the context and environment of measurement.
2. A set of questions refining the goals and characterizing the target of the measurement, including: the quality model(s) for the measured property and the factors influencing these quality models.
3. A set of metrics associated with every question in order to answer it in a quantitative way. A single metric may contribute to different questions and one question may be answered by taking into account several metrics. The data may result from a subjective or objective measurement and the measurements may have different scales: nominal, ordinal or interval.
The net result of applying this measurement should be specifications and implementation plans for a particular set of goals and a set of rules which describe how to interpret the measurement data in the context of the goals.
The GQM-based software measurement includes a process that consists of the same steps used in QIP:
1. Analyzing Current Status
This includes characterizing the organization, the problems to be solved and the goals to be achieved. Different types of analysis techniques can be used during this analysis, for example, CMM, Bootstrap and Spice which are existing software process assessment techniques. Also, TQM (Total Quality Management) professionals have developed team-based and individual analysis techniques.
2. Identifying and Defining Measurement Goals.
This includes refinement and prioritization of the informal improvement goals into GQM goals. This can take the place of defining the GQM goals with templates, QFD tables, spreadsheets or similar templates. Some good ways of defining goals include techniques applied in defining customer requirements, product quality criteria and so forth.
3. Produce the GQM Plan.
This includes the metrics, questions, etc.
4. Produce the GQM Measurement Plan
Part of this is to carefully choose the management goals, measuring everything is time consuming and leaves so much information that it cannot be managed. It is also crucial to know how to interpret the measured data.
5. Collect, Validate and Analyze the Data.
This involves the cooperation of the software developers, managers and quality engineers. Analysis tools are useful but application of these should be considered after team-based data interpretations are made.
6. Produce a Continuous Improvement Plan
This plan must be composed of smaller pieces. It can be applied first in a pilot project or a limited number of pilot projects. Then the measurement process, goals and questions should be evaluated based on the experiences gained from the pilot projects.[Hood 1997]
One of the benefits of this GQM approach over other approaches includes that of general applicability. Different software organizations are at differing levels – that includes the size of the organization, the maturity of the organization and their differing structures. Organizations at a low level of maturity must be interested in creating initial models of their products, their processes and the quality aspects of interest in their organization. In this case, the measurement goals will be to further understanding. When their models have been developed, they can be used to improve engineering and management practices, methods and tools. GQM supports both these types of goals.
Another benefit is that the GQM approach supports identification and tailoring of metrics, support for interpretation of measurement data, and support for involving all interested parties into the measurement process. “The guidelines of the GQM approach provide an ideal tool for actively involving all interested parties from the very beginning. Goals should be chosen which reflect the needs of the project personnel, then the viewpoint listed in each goal definition points to the type of personnel who need to be involved. Third, the GQM guidelines represent mean for communicating the measurement plans to all interested parties and involving them in the review process.” [Hood 1997]
One of the problems with GQM can be the difficulty setting goals for measurement. Software managers may be poorly informed and may identify irrelevant or unattainable goals. Also, the top-down approach makes it difficult to introduce questions which have been successfully investigated in other organizations; the bottom-up information can conflict with hierarchical decomposition of goals into metrics. Construction of these hierarchies can lead to poor definition of attributes and metrics. Another obstacle can be that the initial goals set may be short-term concerns that won’t work over the long term and won’t remain a priority throughout the project.
Reusing Existing Experience
The object of reusing experience is to save effort and deliver better quality in less time. Instead of developing new artifacts from scratch, the Software Development team finds appropriate reuse artifacts. The appropriate reuse candidate is retrieved and modified. Knowledge within the context of the reuse artifact is elicited from the experts that worked on the project where the artifact is being retrieved from. This retrieving can be broken down into identifying the candidate experience and evaluating it. The person or team that will be adding the artifact to the Experience Factory selects the best-suited experience or experiences from the project. This experience will need to be modified to suit the needs of the experience base. “The objective of identifying candidate experience is to find a set of objects with the potential to satisfy project-specific reuse requirements. This is done by characterizing the required object according to the characterization schema and then matching this characterization with the characterizations of the existing artifacts in the EB.” [Althoff 1998 – 3]
Relevant information like detecting problems and risks and making experience-based decisions are retrieve and decomposed into identifying candidate experience in
the EB. The material is evaluated, selecting the best-suited experience, modifying the selected experience to suit reuse. The object of evaluating the experience is to find degree of discrepancy between the reuse requirements and the reuse artifact. Also, to predict the cost of bridging the gap between that candidate and the requirements. Based on this evaluation, which is done by the person, or group of people, who are proposing the experience for reuse, the best-suited candidate or candidates are selected. [Althoff 1998 - 3]
“The objective of reusing experience is to save effort while delivering better quality in less time.” [Althoff 1998 - 3] According to Althoff, this can be done in different ways. Instead of creating artifacts for reuse by hand, he suggests that old artifacts can be modified to suit use by new projects. Although it is certainly better to start with experience artifacts that in which you have reuse in mind, if you have relevant information within a certain context it may be possible to create artifacts that can be retrieved for supporting management in finding problems and risks and making experienced decisions.
The object of identification is to find a set of objects with potential of satisfying project-specific reuse requirements. This is done by characterizing the object according to the previously laid out scheme, and then matching the characterization with existing artifacts.
The object of evaluation is to characterize the degree of discrepancies between a given set of reuse requirements, and predicting the cost of bridging the gap between reuse candidates and reuse requirements. Then the best-suited reuse candidates are selected.
The object of modifying experience is to bridge the gap between the selected reuse candidates and reuse requirements. Either the reuse candidate can be modified into a new artifact or several candidates can be merged into one artifact. [Althoff 1998 – 3]
These tasks of reuse that have been broken down could lend themselves to an automated system, possibly using an Expert System. None of the papers I found suggested doing this but the tasks seems to be readily adaptable to an Expert System.
Retrieval of Experience
Retrieval is one of the most important parts of the process. Recording all the information possible cannot be useful if it cannot be easily retrieved by the members of the Software Organization. Retrieval is supported in different manners by the different Experience Factory Bases. Most of these systems are web based. One retrieval system is based on a library science by the concept of similarity. In other words, concepts are grouped together the same way that subjects are grouped in a library. Also, like a library, they are cataloged.
“A Case-Based reasoning system extend faceted classification by allowing the facets to have arbitrary types . . . for small sets of artifacts, browsing will suffice, other techniques are specialized toward specific kinds of experience, for instance, component specifications.” [Althoff 1998 - 3]
Other Experience Bases use enumerated classification, faceted classification and information retrieval. One of the problems dealt with in creating a retrieval system is that an artifact exactly matching the need will rarely exist, so retrieval must also support a search for a loose similarity Prieto-Diaz created his system using this system of similarity calling it a Faceted Classification. [Prieto 1991] Althoff builds on this system extending faceted classification to allow the user to browse the artifacts using hypertext. [Althoff 1998 – 3]
According to Althoff, if several types of experience are to be cataloged, different types of retrieval systems may be used, or one generic retrieval system for all the types. In either case, all kinds of experience should be characterized using the same formalism. [Althoff 1998 - 3] Whether or not to use a generic retrieval system should depend on the size and complexity of the Experience Base. A separate retrieval system for Lessons Learned, for example, may be in order for a large Organization that is building an extensive Experience Base, as this area may lend itself to different retrieval needs.
Retrieval Process
To begin the project, artifacts may be collected in a storage binder and retrieved by flipping through on demand. Once enough artifacts have been collected they may be transferred into a report describing the project and the lessons learned. The improvement of this is that the report can be made available on line. Then the reports can be organized into the web-based tool. [Althoff 1998 - 1]
Techniques for retrieval include enumerated classification, faceted classification, and information retrieval. Case-based reasoning systems further classification methods by allowing the facets to have arbitrary types. Domain analysis for software engineering artifacts is being researched, but most domain analysis is focused on reuse of software components only.
Options of Retrieval:
There are different ways to try retrieving the data stored. The first of these is based on incomplete information.
“Not all the information needed for comprehensive characterization of an artifact is necessarily contained within the artifact. . . Nevertheless, an effective search must be possible.” [Birk 1998]
For this to work, some basic information must be available. This would include:
§ Information about the artifact itself (what language it is written in for instance),
§ Information about its interface (what was the purpose of the experience, what global variables are needed, etc.),
§ Information about its context (what application it was developed for).
The other method is to retrieve based on similar artifacts. Since software projects are all different, it is unlikely to find exactly the knowledge needed. Therefore, the retrieval method must be flexible enough to find all similar artifacts. These, in turn must be able to be tailored to specific needs of future projects. [Birk 1998]
In Tautz’s proposal of an Experience Base, context specific knowledge can be retrieved using a query. He takes queries in the form of input from the user. The Input involves a query specification in the form of one main and/or several related instances. The Output to the user’s query includes either an error message telling which attribute value specified is out of range or a list of instances similar to the main instance together with the similarity value. The first argument to the similarity function is the instance in the experience base, while the second argument is the specified instance. “The list of instances returned is limited to the extension of the concept of the main instance. . . The extension includes the instances of the sub-concepts of the main instance’s concept.” [Tautz 1998]
In order for retrieval to be effective, knowledge must be categorized/stored according to the defined reference structure. Each element must be related to all other elements that are closely associated. This reference structure enables queries incorporating the specification of several related elements. Other items captured include policies and guidelines concerning tool usage, and individuals with expertise in specific areas. [Feldmann 1989]
System Requirements
The architecture of the various Experience Factory types are all intended to be generic and scalable. They should allow for the reuse of all kinds of software engineering experience, using the semantic relationships among experience items. The environment should not be restricted to storage and retrieval, but should support the complete reuse and teaming processes.
The Storage and Retrieval process for one particular system is based on Representation Formalism for Software Engineering Ontology’s. An Ontology being an explicit specification of a conceptualization. For example, a tool used for elicitation may allow for capture of different types of requirements, such as functional requirements and predefined design decisions. This may be done without specifying how these are related and what properties they have. Only if this conceptualization is made explicit is it an ontology. Software Engineering artifacts are developed and maintained by the various development projects. The project teams supply the information on the artifacts in the form of conceptualizations. [Basili 1996] Conceptualization is a set of concepts, instances, and other entities assumed to exist in an area of interest and their relationships.
Other terminology used in this are include: Epistemistic Primitives, a class of alike characteristics. The level of generality does not limit it to a single area of discourse or even a class of discourse areas. In other words, epistemistic primitives are domain independent. Standard Vocabulary, is used to represent characteristics which are specific enough to occur in every discourse but general enough to describe several characteristics.
Modeling Levels
Different Modeling levels are used in the building of an Experience Base. The first is the Knowledge Level, knowledge contents are visible on this level, but not the internal structures. Knowledge levels are implementation independent. Next, the Symbol Level, Internal structures are visible on the symbol level. The symbol levels is defined by the chosen implementation. The knowledge level represents “what,” while the symbol level represents “how.”
The Epistemological Level, defines the standard vocabulary and is domain independent. It defines concepts, attributes, relationships, etc. The Conceptual Level defines the standard vocabulary, is domain specific. The ontology can be defined using epistemistic primitives. The Linguistic Level defines concrete instance of the constructs on a conceptual level. This is domain and context specific. It includes a concrete measurement plan for measuring the effort of a specific project at a given company. [Tautz 1998]
System Structures
ARCHITECTURE
Feldmann explains his system’s architecture by dividing the system into specific areas: The Technologies area, the Process Modeling Area, The Background Knowledge Area, the Measurement Area, and the Lessons Learned Area.
The Technologies Area
The Technologies area contains the following sub-categories:
Techniques
“A technique is a prescription of how to represent a software development product and/or a basic algorithm or set to be followed in constructing or assessing a software development product. . . Over time, experience about a technique is gained by applying it in projects. Experience may be spontaneous and unanticipated (captured in the form of lessons learned) or planned (captured through a measurement program). These experiences will reveal under which conditions a technique is effective and under which it is not. If more than one technique is available for the same task, the most adequate technique can be chosen. We call such as set of techniques, that is, a set of predictably applicable techniques for a given task, a method. Thus, the application of a method achieves some task using a particular technique which is selected based upon the project goals and context.” [Feldmann 1998]
Method
A method is a set of techniques working together. The application of a method achieves some task using a technique that is selected based on the project goals.
Tools
“Techniques can be supported by tools, that is, tools are a computer-based implementation of a technique.” [Feldmann 1998]
The Process Modeling Area
Process Modeling is made up of product descriptions and activities. An activity can be either a process step or can be further decomposed. “An activity consumes and produces products. These are described by product descriptions.” A product description can play two roles, namely, the description of an input product, or the description of an output product. This area requires the resources of tools and people. Generally, the people are grouped into roles, such as System Engineer, Project Manager, etc. One person may be playing multiple roles, for example, a software developer may be a reader during an inspection. [Feldmann 1989]
The Background Knowledge Area
Knowledge, in the case of a knowledge base is the set of all statements about the represented world that are both believed to be true by the knowledge sources and are in fact, true. Knowledge would not include: Statements that are not believed to be true by the knowledge source or, Statements that are believed to be true by the knowledge source but are not true. This latter one, of course, is difficult to determine. In the case of the Background Knowledge area, Background knowledge consists of things like handbooks, scenario descriptions, demonstrations, and points of contact. [Feldmann 1989]
The Measurement Knowledge Area
A measurement program refers to the measurement object. A method, technique or tool can be a measurement object within the measurement program. [Feldmann 1989]
The Lessons Learned Area
Any experience gained is recorded as a lesson learned. This can be in the form of a problem-solution statement, a guideline, an improvement suggestion, or a success report. [Feldmann 1989]
Relationships between Elements of the Architecture
The architecture of the stored data is based on relationships between the various elements:
In this system the authors define three kinds of semantic relationships between the basic elements, derived-from/derives, has-part/part-of, and is-a/generalization-of. “Derived-from/Derives relationship identifies the sources from which a particular element has been derived . . .Has-part relationship identifies details of a basic element” Is-a/generalization-of relationship can be defined by the statement, if a basic element type X is-a element type Y, it inherits all attributes and relationships of Y. [Feldmann 1989]
CBR (Case-Based Reasoning) ARCHITECTURE
One Experience Management Factory System known as CBR-PEB, has similar architecture to that specified by Feldmann above, but with a little variation:
“CBR-PEB does not contain any experience packages, i.e., a CBR system. It only consists of the case base as an index and a contact address in each case that models the reference. The measurement data base is implemented as a simple file using the system-library file-access functions. CBR-Works from TecInno GmbH (Kaiserslautern,
The organization structure of CBR-PEB is based on the organizational structure in a previous generic model of an Experience Factory by Althoff. One of the main differences is that this model is available with a WWW interface and can be shared by multiple companies. The link between the CBR-PEB Experience Factory (EF) and the organization is the Internet. There is no parent organization for all project organizations. Each software organization may have its own experience factory. Although some experiences will be useful to other companies, many companies, particularly those large enough to maintain their own Software Development staff, would not allow their information to be shared with other corporations. This would be a useful tool for institutions that routinely encourage sharing of information, such as educational institutions, though.
Another possibility with this system is that the existing Experience information could be ported into an new installation such that the system could be useful right away, before the Software Organization has had time to develop and extensive collection of experience.
“The EF server must be installed at a WWW server or included one . . . the data storage components are installed either at the same computer or can be distributed over the network. The EF server manages the distribution of the EB data across the case base and experience package DB (database).” [Althoff 1989 - 2]
The user interface works with queries sending requests to the EF Server. This in turn handles requests using SQL (Structure Query Language) to the data storage components. For example, in Spotfire, the remote machines send requests to the EF server using an EF-proprietary protocol to an EF Server running JDBC that sends DBMS-proprietary protocols to a Database server running Oracle 8 under Windows NT. [Webby 1999]
Organization of the Software Business Company
Several of authors of the papers on Experience Factory suggested that the software development team must be organized into groups with tasks assigned by roles. The entire software development group consists of a development organization, the EF organization and the management group. The development organization is made up of individual project organizations. Each project organization works on its own deliverable parts of a shared project or individual projects. The Experience Factory organization supports the reuse of life-cycle products and related experience like guidance for the developers. It creates and maintains the experience base and performs tasks involved in the reuse (these are specified later). It can also provide technology for the performance of these tasks. The management of the business company influences the development organization and the experience factory. Management can introduce the reuse program and offer employees incentives to contribute data. [Althoff 1989 - 2]
By comparison, another model known as the Capability Maturity Model (CMM) establishes a set of criteria that describes the characteristics of mature software organizations. Then these criteria are used by less mature software organizations to improve their current processes for developing and maintaining software. This is done using small, evolutionary steps. It provides a framework for organizing these evolutionary steps into five maturity levels that are on an ordinal scale for measuring the maturity of an organization’s software process. The maturity levels are:
1. Initial – characterized by ad hoc, chaotic methods with few defined processes;
2. Repeatable – in which basic processes are established in order to track cost, schedule etc.
3. Defined – activities are documented, standardized and integrated into standard software processes.
4. Managed – there are detailed measures of the software process and product quality is collected.
5. Optimizing – in which continuous process improvement is enabled via quantitative feedback from the process. [Ceori 1999]
SEEE ARCHITECTURE
The Experience Factory must be set up by choosing or developing a tool, then developing characterization schemes, and finally implementing the experience base. Recording within the Experience Factory includes checking whether the object and/or experience is important enough to store, and if it is, storing the new experience in the experience base. In addition to experience gained from within the organization, data can be obtained from experience outside the organization. [Althoff 1989 - 1]
In his paper on the architecture of a Software Engineering Experience Environment (SEEE), Althoff proposes an architecture that is generic, and scalable for the learning and reuse of different kinds of software engineering experience. Storage and retrieval for this system are based on REFSENO – Representation, Formalism for Software Engineering Ontology’s. Representation, meaning a set of constructs together with the interpretation how the constructs map onto the characteristics of the represented world. Representation captures some characteristics of the represented world. Representation does not capture all characteristics of the represented world. Representation Formalism is a notation for specifying representations and the meaning of the representation. Representation formalism is made up of symbols with a defined meaning from which representations can be assembled. Another example of a Knowledge Representation Model is that by Prieto-Diaz. [Prieto 1991]
The software engineering artifacts are developed and maintained by projects. The software teams supply information about artifacts; the EF determines what kinds of artifacts should be collected, how they are related and how they should be cataloged.
REFSENO includes a method for storage. This method includes such classifications as Project Characterization, Process Model, Technique Description, Code Module, Lesson Learned, all rolled up into an Artifact. Althoff explains that with his Industrial install base he found a knowledge base system based on formalized general knowledge was not necessary for his objectives since his customer’s queries are restricted to characterizations of particular concepts selected by the user.
The Architecture of the EB Section of REFSENO
Retrieval is context-specific using semantic relationships. This allows people to find useful experience they were not aware of. “Often, the EB installation starts small, but it must be able to grow in size. Frequently, organizations already use specialized tools that must be integrated into the EB. Artifacts might change in the course of an improvement program from informal to a more formal representation. . . Also, additional tool functionality might be supplied as the scope of the reuse program extends.” [Althoff 1998 – 3]
The EB contains:
§ CBR Tool (case-base reasoning tool) - stores the artifact characterizations.
§ General Purpose Browser – provides searching, viewing and manipulating of characterizations.
§ EB Server
Figure 1 –Open tool architecture with examples of tools. - from [Althoff 1998 - 3]
Learning
The objective of learning is to improve software development by allowing efficient and effective reuse. In order to do this, a repository is created of well-specified and organized experience. The actual “learning” is done by the recording and packaging piece. In other words, the task “learn” can be broken down in to the sub-tasks record and package. The objective of recording experience is to fill the Experience Base with well-specified and organized experience according to the needs of the Experience Base users.
 |
Figure 2 Processes in reusing Lessons Learned– [Althoff 1998 - 3]
The Record task can be broken down into collect, store, qualify and publish. The object of collecting is to capture new experience. The Software Team Manager can do this by searching for artifacts and information that will be worth recording for reuse. Project teams can make suggestions for new artifacts or updates of already stored artifacts. To store an artifact it is copied into the Experience Base. Some artifacts need to be split into reusable parts before they can be further processed. Each artifact is characterized in so far as the characterization is known. Each experience is then known as an Experience Package, which consists of an artifact and its characterization.
Artifacts can be collected at various phases of a project. In the case of a small project, it makes sense to collect experience only after the end of the project. For long-term projects, event-driven collection at various milestones makes the new experience available earlier. For example, as soon as the project has a firm set of specifications, the development team may have a task that says to stop at this point and look for any experience that should be stored. However, this can lead to collecting artifacts when they might not have passed all tests in the project. [Althoff 1998 - 3]
Managing Lessons Learned
Lessons Learned are too frequently left in the minds of the project team members. Initially postmortems (meetings held to discuss a project after the project is concluded) have been held in order to gather this sort of information but even then the information is seldom reused. The Lessons Learned Management aims at building a bridge between the people who gained the experience and the people who can benefit from the experience. The project-specific experience statements must be packaged. These lessons can be packaged in two formats, abstracts from the project information – in which the individuals are simply represented by their functions; and complete lessons that include each individual and their role in the lesson.
The packaging process begins with an informal experience statement recorded during the postmortem meeting. Then the process description follows. This includes:
§ Validating and correcting the experience statement
§ Identifying the types of information contained in the experience statement
§ Splitting the experience statements such that each artifact contains only one type of content information.
§ Assigning resulting statements to categories of experience that correspond to the content information types
§ Rephrasing the statements such that each forms a complete sentence.
§ Standardizing the terms used.
§ Supplying all the information to each statement needed to make it self-contained.
§ Supplying to each statement an explicit description of the context situation.
§ Distributing the information in each statement over its category’s facets and providing any lacking information for those facets that are not filled.
§ Rephrasing any statements that don’t conform to whatever uniform style the EF uses.
Logical Structure of Lessons Learned
Each artifact contained in the lessons learned section should contain the following categories: Object of the lesson learned, Problem involved, Solution to the problem, Context to which it pertains. The object can be a process, a product, a technique or a method. Problem and solution are the core parts of the lesson learned. The context describes the situation in which a problem and solution pair is relevant.[Althoff 1998 - 1]
Figure 3 Process Structure for Lessons Learned– [Birk 1998]
Strategies for gaining Lessons Learned
“There are three basic strategies to gaining experience statements:
Use available technical knowledge sources. Use goal-oriented knowledge acquisition (centralized knowledge elicitation), Accumulate knowledge during everyday work (decentralized knowledge elicitation).” [Althoff 1998 - 1]
The latter strategy is the ultimate goal of the system – to fully integrate the Lessons Learned section of an Experience Management System into everyday project management. This approach can be worked into gradually. [Althoff 1998 - 1]
Building a Lessons Learned Area:
Subtasks of “Learn:”
A. Record – the Manager must record the important parts of the lesson to be learned.
1. Collect – He must collect any necessary data related to the lesson
2. Store – The data must be entered into the system and stored
a. Copy – It must then be copied into the Experience Base
b. Split – It has to be split into key elements
c. Initially Characterize – It has to be characterized to match the categories currently in the Experience Base
3. Qualify – Lessons learned must be qualified to make sure they are useful to new groups
a. Verify/Validate – the lesson must be validated by outside groups or unbiased members of the team
b. Correct – corrections should be made to make sure everything is accurate
c. Analyze/Artifact – Each artifact must be analyzed to make sure it stands on its own
d. Evolve – After several artifacts are collected they evolve into a more complete system
e. Complete Characterization – Once the system is large enough to support it, new categories can be added and re-characterization can be done
4. Publish – Then the material is made available to new users o f the system.
B. Package – Once the system is up and on line, it can be packaged for other Software Development groups.
Recording the experience fills the EB with specific organized experience. Collecting captures new or improved experience. Storing experience is making a new artifact and its characterization available in the EB. It is copied into the EB to avoid accidental modification by the project team. Some artifacts can be split into reusable parts. The package is correct as to verification/validation. Each artifact is analyzed regarding its reuse potential. Then characterization is completed. Publishing is to make an experience stored and qualified and available for reuse. In some cases, preliminary publication of draft material can be useful.
Four ways of developing and collecting experience:
§ Spontaneous collection – only the scope is defined
§ Event-Driven collection – at particular milestones
§ Periodical collection – periodically checking of projects for new modules
§ Exactly-planned collection – specified experience is developed and/or collected.
Running the Experience Base
Deployment of an actual EB requires that the EB be embedded into an organizational infrastructure to provide funding and strategies to consolidate reuse and learning.
Organization
Figure 4 Project Organization showing roles – [Birk 1998]
The Project Organization consists of project teams:
· The EF provides the goals for the EB
· The manager defines strategic goals, initiates improvement programs, acquires and manages resources
· The supporter is responsible for recording new experience and supporting the project teams.
· The engineer packages and analyzes existing experience
· The librarian is responsible for technical tasks like creation and maintenance of the EB
In a small project, one person can perform all these tasks. [Althoff 1998 - 3] Although, to work effectively, at least three people are recommended.
Implementation of Experience Factory
“Knowledge management is a rapidly growing discipline in information systems concerned with the use of information technology to collect, organize, store and share knowledge in organization.” [Webby 1999]
Spotfire
Spotfire [Webby 1999] is a tool with implements dynamic querying approaches to do rapid data analysis and information retrieval. Spotfire technology has been applied to visual data mining problems in the pharmaceutical, chemical and biotechnical industries. The technology is also useful as a retrieval interface for an experience factory.
The interface is unusual in that it displays the retrieved information as squares grouped into the categories of data selected for retrieval. “Each data point [viewed as a square] represents an experience package, which is analogous to a document in this case.” [Webby 1999] The user views three charts to provide three different views on the same set of documents. Query devices are check boxes, radio buttons, and range bars that can be used interactively to select the desired packages based on their attribute values.
Spotfire had some initial limitations including the fact that it ran only under Windows, and it was difficult to integrate different types of packages under one application. Also, it did not provide adequate functionality for search and retrieval of distributed packages.
The package was re-implemented in Java to be cross-platform compatible and allow for support of different types of packages and ease of use across an Intranet or the Internet. [Webby 1999]
Scenarios of Possible Uses of Experience Factory
Scenario One
The first scenario project is developing the software for a new fuel injection system. The design team has never done a design review so the manager looks up information from projects that required a design review. In addition to finding information on preparing a design review, the manager finds information on preparing design inspections, and process models from two previous projects that contained information that could be reused on this project.
Scenario Two
In the previous example project, the development team records this project for future reuse. During the project a lesson-learned is gleaned from a problem with a database management system and its solution. A post-mortem of the project gleans further information for re-use including the fact that the project team had been increased by a member and the project duration was prolonged by a month. A review confirms and validates any artifacts entered. Analysis is done to determine re-use potential. The publishing can be done remotely using the www-based technology and an intranet.
During the project planning phase, a preliminary characterization of the project is recorded. The advantages of this is that the project can be notified when errors are found in artifacts reused by that project. The Experience Base can be used as a knowledge map to knowledge in currently running projects. Also, any preliminary characterization can be specified as context for all experiences from that project. This is particularly helpful since lessons learned cannot be added without a project characterization. Note: This scenario may not be applicable to many organizations. For example, this may require that the individual project spend effort in packaging the project artifacts, which may prove to be unstable.
During the project a lesson learned about a database management system problem and its solution are spontaneously recorded. The lesson includes its characterization provided by the project team as a suggestion for the Experience Base collect and store processes. Afterwards, the lesson learned is analyzed and published.
After the project is completed, a post-mortem meeting shows the new process model, developed during the project is worth being stored in the Experience Base. The project characterization is updated and completed based on the results of the post-mortem. Finally the updated and validated project characterizations are also published. Recording project characterizations would be mandatory for all projects. [Althoff 1998 - 3]
Examples of Actual Experience Factories
The above scenarios were examples proposed by Althoff as possible uses of Experience Factory systems. The following are actual systems in use and how they are being used.
Software Inspections
During regular software inspections, one German company used an on-line inspection handbook as training material and a reference guide. This handbook is actually an Experience Factory. The activities outlined in the inspection handbook include:
§ Objective: This describes what should have been achieved once the performance of the activity has been completed.
§ Description: This is a 2-3 sentence explanation of what the process step is about.
§ Inputs: These list the documents which are available, or should be made available, for performing the activity.
§ Outputs: This lists the documents which are produced during the performance of the activity.
§ Task: This describes what is to be done during the activity.
§ Prerequisites: This describes what must be fulfilled to start the activity
§ What to Do: This lists the detailed instructions on what to do.
§ What is Not to be Done: This lists what should be avoided
§ Completion: This lists questions. If the questions can be answered positively, the task is considered to be completed.
This handbook is available on line in order facilitate adding lessons learned at each objective point to the Experience Base. [Feldmann 1989] This system is also in use at the University of Kaiserslautern below.
University of Kaiserslautern
Twice a year a software engineering lab is offered for undergraduate students at the University of Kaiserslautern. Within these labs software projects have to be developed or maintained. Within these projects they use a Revision Control System that for versioning the developed products and controlling the access to those documents. The students can follow the relationships in the system to retrieve background knowledge about the system. [Feldmann 1989]
NASA
At NASA’s Goddard Space Flight Center they have developed specific guidelines for the lessons learned program. “Among the criteria to decide whether to store a lesson learned in a central repository are its relevance, comprehensibility, utility, validity, and reliability [Althoff 1998 - 1]
The goals of this system are to:
“1 – Understand the software process in a production environment,
2 – Determine the impact of available technologies, and
3 – Infuse identified/refined methods back into the development process.” [Basili 1996]
The strategy is to make improvements based on reuse of prior experiences and flexible automation. This consists of three elements, an improvement paradigm, a dedicated experience organization, and a contingency approach.
The improvement paradigm which involves plan, execution, analysis, and synthesis, is applied to integrate efforts on an organizational level as well as a project level. The project uses a contingency approach because it was felt the organization should be able to change its configuration.
The Software Engineering Laboratory, part of the Flight Dynamics Division of the Goddard Space Flight Center manages and controls NASA’s Earth-Orbiting scientific satellites. It maintains over 100 different software systems including mainframe systems and UNIX workstations or PC based systems. The SEL researchers and database team use Experience Factory for software developers in the FDD. The Experience Factory serves the project organization by analyzing and synthesizing knowledge into models that support the improvement of software development by concentrating on activities of the QIP (Quality Improvement Process).
“The experience factory collects and analyzes the data from the project organization. It stores these data and analyzes in an experience database. It also packages the best of these experiences into products, guidelines, and models, which it feeds back to the project organization to help improve its processes. . . The experience factory for maintenance operates the same as the experience factory for development, with three differences: First, the experience factory for maintenance must address releases. Second, analysis for release feedback requires quicker response; development life cycles re on the order of 18-24 months, whereas maintenance release cycles are on the order of 6 months. Third, software maintenance emphasizes product evolution more than software development does, so experience includes past experience on the same project.” [Valet 1998]
Organization
The project’s organization is responsible for planning and development. Their focus is on problem solving. For this group, the Experience Factory is primarily responsible for learning and technology transfer activities.
Experience Factory for NASA is divided into several sub-organizations. Each of these organizations is dedicated to a particular kind of experience. They are further divided into specific application domains, like production control systems and financial systems. The component factory section has as its purpose developing and packaging reusable software components. Among software components that can be re-used are code components, designs, collections of code components and documents in general. Each software component is packaged with its code, its taxonomy and a re-user’s manual.
“Project-organized units are responsible for planning and development. These units receive support from specialist units. Focus is on learning, training, and technology transfer.” [Aaena 1997] The focus is on skills and experience rather than procedures in coordinating work. [Aaena 1997]
Future Work
Q-Labs
Q-Labs, a software improvement consultant organization with offices in Sweden, Norway, Germany and the US are planning a program of experimental trials and a pilot study. The goals of this study are:
“To evaluate the current set of attributes in terms of completeness, usefulness, and clarity Are there any attributes that would be useful to include?
Are there any attributes that are not clear in their meaning?
What attributes are most used in searches?
To evaluate the Spotfire-based retrieval interface in terms of usefulness, usability, and appropriateness.
Are the subjects able to find what they are looking for using the interface?
Do subjects get frustrated by the interface?
Are there any suggested improvements?
To evaluate the data entry interface (a simple forms-based interface) in terms of feasibility, usability, and impact on working procedures.
Are all information requests clear and understandable to the user?
Does the interface ask for information that is difficult to provide?
Does using the interface represent a significant change in the way things are done?” [Webby 1999]
Using the results of this questionnaire, the group should come up with some interesting information on ways to improve the Experience Factory process.
SEL
Software Engineering Laboratory (SEL) has also been working on furthering the work on Experience Factory. “From this researchers have made a good start at baselining this process. Preliminary quantitative data analysis is based on only nine complete maintenance releases. More releases need to be studied. Also, baseline models need to be extended to include an understanding of maintenance cost and cost estimation, plus a better understanding of error rates. Beyond this, future maintenance study activities need to provide a more complete understanding of the testing process and the inspection and certification process.” [Valett 1999]
The FDD is currently working on porting most of its software from IBM mainframes to Unix workstations. This undertaking will give good insight into how well the Experience Factory helps in this process. Among the questions they hope to answer are, “How might we know when a product has outlived it’s usefulness? Can we predict the most error-prone modifications? and How can we more accurately estimate the cost of software changes?” [Valett 1999]
Summary and Conclusions
As the software development environment becomes more demanding and the time frame from the beginning of the project to the end becomes severely fore-shortened, the need for training, code reuse and learning from past software development experience becomes more and more acute.
Several software development organizations are pioneering Experience Factory management and coming up with many techniques and systems for reusing not just code but every aspect of the software development process. Some parts of the process that can be reused include: requirements, design and documentation as well as design method tools, software best practices, technologies, lessons learned, various models (resource, product, process, quality and cost models), measurement plans, and data.
Most of the systems in development or already developed include a Knowledge Base, a method for storage, processes for information gaining, processing and storage. And then on the other end, a method for retrieval using key pieces of requested information, and a process for documenting and reusing Lessons Learned.
In addition to reuse, training time can be shortened for someone new to a process or procedure on the project. Various members of the software development team are identified and their roles in the capture and storage of information are outlined.
Several software development teams are currently using this process and are examining the process for future enhancement to the reuse experience.
References:
[Aaena 1997] Aaena, Ivan, Peter Bottcherb and Lars Mathiassenc “The Software Factory: Contributions and Illusions” IRIS20 Proceedings, University of Oslo, Department of Informatics, August 1997.
[Althoff 1998 - 1] Althoff, Klaus-Dieter, Frank Bomarius and Carsten Tautz, “Using Case-Based Reasoning Technology to Build Learning Software Organizations,” IESE-Report No. 027.98/E, Version 1.0, July 9, 1998