It is a characteristic of processes at this level that the focus is on continually improving process performance through both incremental and innovative technological changes/improvements.

At maturity level 5, processes are concerned with addressing statistical common causes of process variation and changing the process (for example, to shift the mean of the process performance) to improve process performance. This would be done at the same time as maintaining the likelihood of achieving the established quantitative process-improvement objectives.

This online upgrade provides students who have taken CMMI Version 1.2 training the opportunity to understand the improvements made in CMMI Version 1.3. It will help you to successfully make the transition from CMMI Version 1.2 to CMMI Version 1.3 for all three of the CMMI models: CMMI for Acquisition (CMMI-ACQ), CMMI for Development (CMMI-DEV), and CMMI for Services (CMMI-SVC).

For a more detailed, see http://cmmiinstitute.com/training/cmmi-version-13-model-upgrade 

A Software Engineering Process Group (SEPG) is an organization’s focal point for software process improvement activities. These individuals perform assessments of organizational capability, develop plans to implement needed improvements, coordinate the implementation of those plans, and measure the effectiveness of these efforts. Successful SEPGs require specialized skills and knowledge of many areas outside traditional software engineering.

Following are ongoing activities of the process group:

• Obtains and maintains the support of all levels of management.
• Facilitates software process assessments.
• Works with line managers whose projects are affected by changes in software engineering practice, providing a broad perspective of the improvement effort and helping them set expectations.
• Maintains collaborative working relationships with software engineers, especially to obtain, plan for, and install new practices and technologies.
• Arranges for any training or continuing education related to process improvements.
• Tracks, monitors, and reports on the status of particular improvement efforts.
• Facilitates the creation and maintenance of process definitions, in collaboration with managers and engineering staff.
• Maintains a process database.
• Provides process consultation to development projects and management.

CMMI is an application of the principles introduced almost a century ago to achieve an enduring cycle of process improvement. The value of this process improvement approach has been confirmed over time. Organizations have experienced increased productivity and quality, improved cycle time, and more accurate and predictable schedules and budgets.

The CMM^® was based on a vision by Watts Humphrey and developed over a period of 3 years, from 1984 to 1987, by the Software Engineering Institute funded by Carnegie Mellon University. It was in response to a growing perception that the weakest link in software quality was the development process. The overall ability of most software companies to deliver product with good quality on time and under budget was becoming more difficult as the technology and speed issues changed so frequently. During the next 5 years, there were several revisions and by 1992, the CMM^® as we know it today was put into motion. Three important things to remember about CMM^® are that the maturity framework is based on actual practices, the framework reflects the best state of these practices, and the needs of the individuals performing the tasks in the processes are the most important inputs to the CMM^®.

Depending on the CMMI areas of interest (acquisition, services, and development) used, the process areas it contains will vary. Process areas are the areas that will be covered by the organization’s processes. The table below lists the process areas that are present in all CMMI areas of interest in CMMI Version 1.3. This collection of sixteen process areas is called the CMMI core process areas.

Abstract

Most of organizations have improved their processes not only to upgrade their products but also to accredit themselves in world market as an international certificate which is an essential demand. The critical success factors (CSFs) of the official Standard CMMI Appraisal Method for Process Improvement (SCAMPI) appraisal in software development comes from a cooperation of stakeholders in organizational unit (OU) especially Lead Appraiser (LA), Appraisal Team Member (ATM), and Software Engineering Process Group (SEPG). LA plays role to manage ATM following SEI’s CMMI appraisal process which is supported by established guidelines such as CMMI model, SCAMPI, ARC, and MDD. ATM needs to understand its details practically. Moreover, Software Engineering knowledge is required because ATM is expected to recommend appropriate techniques to encourage the organizations. Better appraisal result can come from better ATM works. Thus appraisal is the technical challenge of ATM.   This paper aims to present the capability of ATM via performance and quality measurement. The CSFs to encourage ATM for better appraisal and achieve established goals have been determined and analyzed. We believe that the analysis result will be useful information for selecting the appraisal environment appropriately.

Keywords: Capability Maturity Model Integration (CMMI), Software Engineering Process Group (SEPG), Appraisal Team Member (ATM), Standard CMMI Appraisal Method for Process Improvement (SCAMPI), Appraisal Requirements for Capability Maturity Model Integration (ARC), Method Definition Document (MDD), Lead Appraiser (LA), Organizational Unit (OU)

1. Introduction 

The successful adoption and implementation of software development process standards seem to be providing inspiration in other business domains [1]. It is also important to discover how to indicate the work environment of successful implementation via assessment as CMMI appraisal. This research presents the appropriate factors to establish good quality and performance of CMMI appraisal team based on assessment model called “Standard CMMI Appraisal Method for Process Improvement (SCAMPI)” The implementation of models like CMMI can be difficult due to resource constraints and the expertise required in an appraisal of CMMI practices [2].  The appraisal lessons learned have been accumulated to evaluate related measures.

2. Related Works and Background

2.1 Related works

Understanding CMMI based Appraisal will help SPI practitioners to identify what they need to address and when they can develop CMMI appraisal strategies to manage environment and team likes Appraisal Team Member (ATM). The Knowledge of appropriated strategies may also help with the development of new or improved an assessment of SPI implementation approaches, which better match organizations’ objectives as internationally level.

Based experience and lessons learned from 27 SCAMPI Appraisals for Small Companies as a Process Performance Models, they can significantly ease the pain of CMMI appraisals for all sizes of companies; including small companies (reduce effort, increase quality) [3]. All of the companies reported major benefit from the assessments [4]. The development of a low-overhead method of software process appraisal specifically targeted at Irish software small-to-medium-sized enterprises (SMEs). The SMEs preferred to have an appraisal performed using a balanced mixture of well-performed process areas and less efficient process areas [5].

The Complete assessments of the process maturity or capability of software developing organizations are not feasible if their sole objective is to improve the predictive power of software reliability growth models [6]. The integrated SE and software models should save time and money and reduce redundancy in assessments for both software and SE process improvement. The successful implementation of SPI can lead organizations to business benefits and help them improve their software development capabilities [7]. Nevertheless, those previous researches had not been mentioned specific CSFs which reflect to appropriate appraisal environment.

2.2 Background

In the next following paragraph, we describe terminology related to CMMI Appraisal as CMMI model for Development, SCAMPI, MDD, ARC and ATM.

2.2.1 CMMI model for Development

CMMI model for Development are tools that help organizations improve their ability to develop and maintain quality products and services. CMMI model for Development are an integration of best practices from proven discipline-specific process improvement models, including the CMM® for Software, EIA 731, and the Integrated Product Development CMM. The CMMI has five levels that measure the path from immaturity to maturity is presented in Fig. 1.

Figure 1. CMMI Maturity Level.

2.2.2 Standard CMMI Appraisal Method for Process Improvement (SCAMPI)

The CMMI appraisal is based on SCAMPI which is designed to provide benchmark quality rating relative to Capability Maturity Models and the people CMM [2]. The appraisal Method is presented in Fig. 2 and appraisal types and its requirement is shown in Fig. 3.

 Figure 2. Appraisal Method

Figure 3. Appraisal Types and its Requirements

2.2.3 The SCAMPI Method Definition Document (SMDD)

SMDD describes the requirements, activities, and practices associated with the processes that compose the SCAMPI method. The SMDD also contains precise descriptions of the method’s context, concepts, and architecture [9]. SCAMPI objectives provide an accurate picture of the organization’s processes relative to CMMI which is to understand the current implemented process. Identify process weaknesses (and strengths) in the organizational unit, to determine degree of satisfaction of CMMI Process Area goals investigated and to assign ratings, if requested by appraisal sponsor. The SCAMPI phase structure and activities are demonstrated in the Fig. 4 and 5 respectively.

Figure 4.  SCAMPI Phase Structure

Figure 5. Appraisal Activities

2.2.4 The Appraisal Requirements for CMMI (ARC)

ARC consists of a set of high-level design criteria for developing, defining, and using appraisal methods based on CMMI models [8]. These requirements constitute an evolutionary progression from the CMM Appraisal Framework Appraisal Requirements for the ARC has been created to accommodate these new discipline models and their staged and continuous representations. The ARC has also been influenced by the EIA/IS 731.2 Appraisal Method [15] and ISO/IEC 15504. Finally, the requirement to encompass both assessment (for internal process improvement) and capability evaluations (for source selection and/or process monitoring – U.S. Department of Defense-DOD), has influenced the ARC.

2.2.5 Class A-Appraisal team member (ATM)

ATM is a team of experienced individuals that conducts a process appraisal of an organizational unit and may contain members internal to the organization as well as those external [9]. Typically consists of 4-9 members, one of whom is the appraisal team leader. Beside, Teams are Important to appraisals: appraisal results reflect the knowledge, experience, and skill of the appraisal team. Objectivity of the appraisal results is dependent on the objectivity of the appraisal team. Credibility of the results is dependent on the credibility of the appraisal team and its decision making process.

3. Measurement and Validation

Simply stated “If you cannot measure where you are, you cannot demonstrate that you are improving” [10] and “If you cannot measure it, you cannot control it.” [11] make us to realize importance of measurement which can visualize abstract data. Based on 19 appraisals experience and lessons learned of 100 ATMs, data is collected, summarized, and validated for identifying performance and quality of appraisal participants and for analyzing their influenced factors.

3.1  Performance

In general, Performance evaluationis a systematic and periodic process that assesses an individual employee’s job performance and productivity in relation to certain pre-established criteria and organizational objectives. Performance results are interested in cost, schedule, productivity, quality, customer satisfaction and ROI [12]. A central reason for the utilization of performance evaluation is performance improvement. Performance evaluation can aid LA and sponsor in the formulation of job criteria and selection who is the best suited to perform the required tasks as ATM for an appraisal. Other aspects of individual ATM are considered as well, such asaccomplishments, potential for future improvement, strengths and weaknesses.To collect process evaluation data, there are three main methods consisting ofobjective production, personnel, and judgment evaluations. Judgment evaluation is the most commonly used with a large variety of evaluation methods.

Appraisal process evaluation is typically conducted at the end of appraisal process milestones including Phase 1-Planning, Preparation and Readiness reviews, Phase 2-Onsite (PIID reviews, Preliminary Findings), Phase 3-Reporting as Final Findings, and Phase 4-Postmortem. Performance management systems are employed “to manage and align” all of an appraisal’s resources in order to achieve highest possible performance which here will focus on the variation of planned schedule versus actual schedule by using the collected data from 100 ATMs within 19 appraisals. Other fundamental performance sides such as productivity or cost effectiveness are not reasons for considering due to SEI procedure has indicated the best practices of mature able performance expectation include a basis for the appropriate risk management and parameter limitation to conduct an appraisal by LA’s planning.

We realize that better individual ATM performance reflect to better appraisal performance. ATM performance evaluations are meant to be an objective assessment of an ATM’s work performance based on established criteria. They act as an official record of performance and can establish the basis for a quality increase. The most important issue for the evaluation is uncomfortable to rate companion by LA at the end of phases (1-5 scores as 1 is “not applicable”, 2 is “extremely difficult”, 3 is “difficult”, 4 is “challenging but manageable”, and 5 is “not difficult”).  Thus he or she must know the LA well enough to evaluate his or her performance.

Fig. 6 presents schedule performance which had been managed in appraisals leading to success or failure. Performance evaluation can help facilitate ATM communication, solve problems and keep lesson learnt for future appraisals. The analysis result is able to be transferable cross-culturally, various types of appraisal such as either first appraisal or renewal when appraisal certificate has expired, and differentiation of experienced ATM and non-experienced ATM.

Figure 6. Schedule Performance Evaluation

The above graph is presenting the schedule performance by phases and the comparison between planned time and actual time that used for working in each phase of appraisal activities (4 phases).    From established criteria, norm value is 1.00, blue is planned time, red is actual time, x is phased task, and y is percent work completed, the graph slope has represented the values of schedule performance which can interpret as following;

1)       The actual time that ATM worked in phase 1- preparation and planning is slower than planned time, the causes of delay is from planning task which LA, ATM, Site sponsor spent to develop the appraisal plan. Some points are confused with how to select assessed projects and participants, scheduling for ATM related activities.

2)       The actual time that LA spent for phase 4- postmortem including repository of all artifacts, appraisal records to baseline, etc. Mostly actual time is not big difference from planned time, but some appraisals which ATM forgot user name and password for feedback at SEI Appraisal System (SAS) can make a delay for completion of appraisal submission. Furthermore, some appraisals have required for revising the details about appraisal information at SAS by SEI QA team and Lead appraisal needs more time to revise and waiting the result from SEI QA team. In some cases, the assessed companies needed to waiting for approval of budget office who gave supporting to the appraisal, even if they found any issues and the site-sponsor and/or LA must be remedial the request.

3.2  Quality

Because the appraisal results depend on appraisal team, we assume that better team quality leads to better appraisal results. Thus, quality of appraisal results comes from quality of ATM work. On the other hand, ATM will be measured and evaluated work quality by the number of defects. The defects can be collected by verification and validation method. The team charter is a mechanism to set up team and support its work to reach its goal. However, ATM can come from internal and external organizations which have different norm and culture. Quality is used to be a criterion for evaluating ATM quality. ATM works consists of 13 activities which indicated in SMDD. SEI establishes procedures to verify and validate ATM work products formally in each referred phase. Moreover, ATM evaluation process is clearly defined. Quality rating scales for ATM evaluation consist of 5 levels (1 is “Poor”, 2 is “Fair”, 3 is “Adequate”, 4 is “Good”, and 5 is “Excellent”). In this paper, ATM types are categorized in 8 groups for determining quality of phased works. They are as following.

1)       Experience ATM , External ATM , First Appraisal

2)       Experience ATM , External ATM , Renewal Appraisal

3)       Experience ATM , Internal ATM , First Appraisal

4)       Experience ATM , Internal ATM , Renewal Appraisal

5)       Non-Experience ATM , External ATM , First Appraisal

6)       Non-Experience ATM , External ATM , Renewal Appraisal

7)       Non-Experience ATM , Internal ATM , First Appraisal

8)       Non-Experience ATM , Internal ATM , Renewal Appraisal

Fig. 7 presents defect removable rate by phases from 100 ATMs of 19 appraisals.  We focus on comparing between defect removable rate by phases of the second ATM type and the sixth ATM type because these types are expected to work independently in experienced organizations. The graph shows that external experience ATM in renewal appraisal that is evaluated to perform the best can remove most defects in the third phase because this phase needs technical analysis and understanding to report the final finding properly. Some appraisals found 0% defects removable rate in phase 1 because of no defect occurred. It is advantage from SEI defined procedures of planning and reviewing readiness.  Conversely, ATM does not need to remove any defects found in the phase because they will be managed by risk assessment for preventing injected problems in the next phases. Another view of removing defects is shown in Fig. 8 which indicates #defects in each type including removed defects (green bar), injected defects (red bar), and escaped defects (blue bar). It emphasizes to technical challenge of ATM in eliminating defects which occurred in different phases.

Figure 7. Removable Rate by Phases

Several conditions allow LA to replan for optimizing performance of ATM team. Whenever conflict occurs, consensus will be tool to solve it in both mini-team and whole team. All influenced problems and defects in each phase needs to be solved before starting the next step.

Figure 8. Defects found in each phase

The result presents the score for individual ATM. However LA has to control team systematically. Team quality impacts to success through several channels such as customer (satisfaction, concern for wishes & needs of customer, focus on every decision, quality goods and services, make customer feel good),  reliability (product & service, zero defects in scope, knowledge, responsible), resource (method to gain, active worker, knowledge, attendance, participation,),  management (problems, decision-making, evaluate change, commitment, communication, follow plan),  benefit (long term success, recommendation) [13]. The points of quality improvement have 3 parts including quality definition, quality measurement and quality improvement and 4 basic approaches to conduct quality improvement are (a) individual problem solving, (b) rapid team problem solving, (c) systematic team problem solving, and (d) process improvement [14].

4. Conclusion and Future works

From above analysis, we can summarize that performance and quality of ATM influencing to quality of the appraisals. These attributes vary in each ATM type and the best ATM type for the appraisal is type ofExperience and External ATM in Renewal Appraisal. Following all historical data of 19 appraisals from 100 ATMs, we found that performance and quality of all ATM types are not different as Fig. 9. Why is the comparison compatibly shown? An important reason heads toward to SEI organized appraisal process. And it can assure why the process improvement does affect to product improvement. Another crucial point is LA’s skill to manage the ATM team. LA has to control ATM to follow the phased appraisal process (referring Fig. 4) objectively.

Figure 9. Comparison between Performance and Quality of each ATM type

The future work aims to research other attributes affecting better appraisal performance such as organization culture, risk mitigation, conflict of interests, stakeholder performance.

5. References

[1]     CMMI Product Team, “CMMI for Development, Version 1.3”, CMU/SEI-2010-TR-033, Software Engineering Institute, November 2010.

[2]     SCAMPI Upgrade Team, “Standard CMMI Appraisal Method for Process Improvement (SCAMPI) A, Version 1.2: Method Definition Document”, CMU/SEI-2006-HB-002, Software Engineering Institute, August 2006.

[3]     Bosheng Zhou and Pu Bai, “Process Performance Models – Lessons Learned from 27 SCAMPI Appraisals for Small Companies”, Proceedings of the First International Research Workshop for Process Improvement in Small Settings, January 2006.

[4]     Fergal MC Caffery, Gerry Coleman, “Analyzing the Cost of Lightweight SPI Assessments”, Dundalk Institute of Technology, Ireland.

[5]     Fergal MC Caffery, Gerry Coleman, “The Development of a Low-Overhead Assessment Method for Irish Software SMEs”, Journal of Information Technology Management Volume XVIII, Number 2, 2007.

[6]     Michael Grottke, “Software Process Maturity Model Study”, IST-1999-55017, 2001.

[7]     Randall R. Wright, Software Technology Support Center, “Process Standards and Capability Models for Engineering Software-Intensive Systems”, CROSSTALK- The Journal of Defense Software Engineering, October 1998.

[8]     SCAMPI Upgrade Team, “Appraisal Requirements for CMMI Version 1.3 (ARC, V1.3)”, CMU/SEI-2011-TR-006, Software Engineering Institute, April 2011.

[9]     SCAMPI Upgrade Team, “Standard CMMI Appraisal Method for Process Improvement (SCAMPI) A, Version 1.3: Method Definition Document, CMU/SEI-2011-HB-001, Software Engineering Institute, March 2011.

[10]Mindlance, “Testing Metrics”, pp. 2.

[11]John Grebe, Ray Boundy and J. Laurence Amos (eds.), “A History of the Dow Chemical Physics Lab”, The Freedom to Be Creative (1990), pp. 53.

[12]Dennis R. Goldenson and Diane L. Gibson, “Measuring Performance: Evidence about the Results of CMMI”, 5th Annual CMMI Technology Conference & User Group, Denver, Nov 2005.

[13]Gerard M Blair, “How to Build Quality into your Team”.

[14]Thada Bornstein, “Quality Improvement and Performance Improvement: Different Means to the Same End?” QA Brief, pp. 6-12.

Electronic Industries Alliance/Interim Standard (EIA/IS) 731.2 Appraisal Method, “Systems Engineering Capability Model Appraisal Method