June 2001
Volume 3 • Number 3

Contents

PROCESS IMPROVEMENT
Case Study of an Improvement Program Featuring Reviews and Inspections

by: Robert MacFarland, Ericsson Ltd.

The effective use of reviews and inspections (R&I) was one of a number of initiatives chosen within a program to increase the improvement pace globally in order to reduce costs. This was achieved by reducing the time to market and increasing the quality of software produced.

This case study reports on the effort made to spread the use of R&I throughout Ericsson, a large multinational telecommunications company, over a number of years. The struggles, successes, and lessons learned from this experience are shared.

Among the innovative ideas employed were: the use of local design centers to peer review and assess each other’s progress; multinational process teams to make use of global competence; packaging improvements to accommodate cultural differences; and the importance of supporting infrastructure and local champions.

Key words: change management, culture, global business, process improvement, promotion

INTRODUCTION

It is well known today that it is difficult to implement successful improvement programs that deliver lasting results. Most improvement programs, unfortunately, end in failure. The difficulty is in building the necessary infrastructure and developing a culture that supports getting new working methods adopted and practiced successfully. This could be due to the fact that often within large multinational organizations there are "flavor of the month" type improvement programs that receive corporate sponsorship. These, unfortunately, have the habit of coming and going before delivering their full promise.

This article tells the story of Ericsson, which has unfolded over a period of five years. It describes an initiative that has continually striven to get the reviews and inspections (R&I) processes and procedures it has developed adopted within Ericsson as the normal way of doing business. The possible reasons for the success experienced as a result of this initiative include the way the improvement was organized and introduced to users, together with the suitability of the improvements’ outputs to address users’ real needs.

The author hopes that this case study will not only benefit readers who are involved in the promotion and implementation of R&I in their own organizations, but also those who are charged more generally with introducing changes in whatever methods their company may employ.

THE COMPANY’S MOTIVATION TO CHANGE

During 1995, within Ericsson’s Public Network Switching organization, it was generally recognized that for operations to remain profitable, a breakthrough was needed. It was agreed that the reason for this state of affairs was the long project schedule overruns and the resulting poor software product quality.

It was also agreed that the causes of these problems had to be eliminated, and in response, representatives from 14 local design centers (LDCs) who were involved in creating the software, were asked to come and contribute ideas. A three-day workshop was held to help identify the necessary improvements. The solutions identified at that meeting resulted in the establishment of the following improvement teams: incremental development, teamwork, processes, methods and tools, and R&I.

A LARGE-SCALE IMPROVEMENT INITIATIVE PROGRAM

The previously mentioned initiative teams were part of an overall initiative program called "Time to Market 15 Months." This initiative was sponsored by Goran Ohlsson, the responsible product provisioning manager. At that time, the program was managed centrally from the Swedish headquarters as a set of focused improvement teams.

The various LDCs were visited and asked to offer participants who were interested and competent in the specified improvement areas. In hindsight this was probably one of the factors for the successful introduction of the improvement initiatives outputs. The LDCs, as the customers for implementing the improvements, had a say in and were involved in the actual development of the improvements. This helped break down the "not invented here" barrier and meant that on the local level they saw peers from other design centers involved in developing the improvements rather than just the involvement of the company’s Swedish headquarters.


Establishment of R&I Team

The names of the volunteers from the LDCs were put forward, short listed, and formed into teams, with one design center performing the leader role and administrative duties. The R&I team consisted of six or seven members from five different countries, mainly Northern Europe. During January 1996 the R&I team discussed its ideas for implementation by first conducting a detailed analysis of the features and usage of the different types of R&I at the various design centers.

In February 1996 a road show was set up in conjunction with the other improvement teams to visit the design centers participating in the next major project. The teams presented their suggestions for improvements, and the views of the design centers were collected. The R&I competence team visited 15 design centers in 10 different countries. Apart from the presentations at the LDCs, the use of R&I at the LDCs was examined and their areas of interest for support and further development were evaluated and discussed with a nominated representative at the LDC. It should be noted that these local representatives who were responsible for R&I later formed the backbone of the "champions" network.

The results of the interviews were analyzed, and areas of improvement were prioritized. Suitable documents already available were identified. These were modified to produce a comprehensive package that consisted of R&I planning guidelines, work instructions for review procedures, inspection process improvements, data capture, and data analysis guidance. As a result of the analysis of the interviews, it was also concluded that there was tremendous inconsistency in the types of R&I used at Ericsson, as well as a wide range in the maturity level of the implementation of the R&I process being employed. It was concluded that R&I education was to be a vital area to address the issues currently being raised.

R&I Definitions as Used by the Initiative

Reviews and inspections are two terms that are normally and wrongfully interchangeable. The benefits that reviews and inspections provide are often diminished because these two different types of evaluations are not used to their full extent.

A review is an evaluation of the software element(s) or project status to ascertain discrepancies from planned results and to recommend improvements. Different review methods have different purposes and they complement inspections.

Many different review methods have been used in Ericsson without common terminology. The R&I initiative selected and recommended some review methods to be used during the development of design in conjunction with the inspection process. These are termed internal reviews:

• Frequent reviews. These reviews focus on finding defects during the development of a document or design. Each review examines only new, changed, or influenced design. The review includes verifying compliance to applicable rules, procedures, and methods.

• Walkthroughs. These are similar to a lecture and are used to obtain early feedback and consider alternative solutions. The author of the design leads the audience through a document and the audience actively participates in asking questions, raising concerns, and commenting on the presented solution, possible defects, and so on.

• One-third presentations. These are particularly useful in new design. They are held when the author of the document or design has a good understanding of the problem to solve and has developed a preliminary solution for it (usually about one-third of the way through the project). The document itself does not necessarily need to be presented because there are other possible means to present the proposed solution.

These internal reviews focus on early defect detection and feedback, and are performed while the document is under development. An inspection, on the other hand, is normally held when the document is completed.

An inspection is a formal evaluation technique in which software requirements, design, or code are examined in detail by more than one person to detect defects, violations of development standards, and other problems. The review performed to technically approve a document by the system group is called an external review. It is carried out after the document has exited from an inspection.

Training Program

In parallel to the development of the process, a training course was developed that consisted of slides and documents. These were available on the Ericsson intranet to be either downloaded and given as part of a traditional course or to be used for self-study in association with the guide that was produced for that purpose. Thirty champions from various design centers participated in the training during August and September 1996. It was their job to assist with planning and training at their LDC and to provide on-site support for the R&I process (see Figure 1). The R&I competence team provided detailed specialist knowledge backup support to these local champions (see Figure 2).

LDC Implementation Plans

To be offered any support from the R&I initiative, each participating design center was required to produce an implementation plan detailing how it would implement the various elements of the R&I improvement package. The competence team member allocated to support the design center reviewed the plan and checked that the implementation fulfilled the minimum requirements necessary, and the support provided was sufficient to ensure successful results.

The minimum implementation requirements are:

• A baseline of compliance with the proposed R&I processes
• Planning performed with planning constants
• All moderators and checkers have received training in the selected review and inspection methods
• Data collection performed to aid process control
• Project manager participates in the implementation

Establishing the Champions Network

After planning R&I activities within the LDC, subprojects were completed. A two-day forum was held in Brighton, England, in October 1996 between those actively involved and was overseen by consultant Dot Graham. At the forum the various LDC’s implementation plans were presented and workshops were held to identify what effective support could be given toward the implementation.

The conclusion of the forum was that there was still a need for a common understanding between the champions. A weeklong forum was then arranged to be held in Rome in March 1997 with consultant Tom Gilb. Training was provided by Tom and Kai Gilb to the champions, and a master class for R&I competence team members was conducted by Gilb, with further evening presentations from the champions of their local innovations. See Figure 3 to learn how the various roles within the network interworked.

Reported Outcome of First Year

Apart from the various champion forums and training sessions held throughout the year, there were also eight R&I competence team meetings and five common workshops between the various initiatives. There were 12 implementation plans produced, and 15 design centers ended up implementing the initiative package to varying degrees within the Helios project (the Helios project was the major development project used as a pilot for implementing identified improvements, including R&I).

According to the survey by a team using goal question metrics (GQM) to measure the impact of the various initiatives, the level of implementation of the R&I initiative measured from data received by 10 design centers was 57 percent (MacFarland 1997). The improvement program also arranged an experience exchange with the Helios main project team, and the feedback given was generally favorable toward the R&I initiative, resulting in recommendation of the result of the R&I initiative to other projects.

THE SPREADING OF IMPROVEMENT TO OTHER PROJECTS

It was decided to improve and encourage the implementation of the R&I initiative along with some of the other initiatives undertaken by the switching organization. This was carried out via a new program, which was also sponsored by Goran Ohlsson, titled the "Model, Master, Apprentice" program.

The principal strategy for R&I to achieve this goal was in getting the package of improvements that were developed during the initiative incorporated within the Ericsson standard process suite used in the design of the software for Ericsson’s latest switch. An expanded R&I competence team was established based upon the original, and now included members from Southern Europe, the United States, and Australia. They were given the task of developing and supporting this globally.

Local Design Champions

The LDC managers were asked to nominate champions who would motivate the implementation, not just within one project but within all local projects. They were asked to ensure that the correct support structure was in place to carry this out successfully and to document this in an updated implementation plan.

Promoting the Initiative

In May 1997, the R&I competence team, with the help of consultant John Mattock, produced a promotional package. The promotional package was carefully constructed so that selected parts of the argument put forward were aimed at different target groups. This was to persuade as many of the audiences as possible to adopt the outputs from the R&I initiative—to achieve a "critical mass" of adopters. Use was made not only of logical arguments of why the changes were needed, but pictorial "hooks" were also embedded so that they bypassed the normal filters of the left side of the brain and were then retained in the memory.

Shortly later, Mattock provided training in change management to the R&I competence team and local champions. He also addressed how to present and argue the case for R&I through role-play of the different intended target groups. Because of the popularity of Mattock’s facilitation skills in making the forums "fun," he has consequently helped plan, organize, and facilitate most of the R&I champion forums.

Level of Design Center Adoption

Of the 24 LDCs associated with the switching organization, 21 applied to become apprentices as part of the R&I initiative. Fifteen LDCs actually provided implementation plans detailing how they would implement and support R&I.

The adoption of "Model, Master, Apprentice" R&I as a good practice by the Ericsson corporate best-practice improvement initiative helped capture data gained from the LDCs and produced success stories and return on investment (ROI) figures, which were also incorporated into the promotion package.

Observed Results for Second Year

For the results from implementing inspections from the LDC of the author during 1997, see Figure 4. These are as reported by the local champion Jonathan Clark (no data available for benefits of reviews) in (Clark 1997). The figures give a return-on-investment (ROI) figure of 4-to-1. This design center calculated its savings based on the assumption that each major defect discovered saved 10 design hours.

The inspection process employed at the design center also asked the designer who discovered the defect to estimate how much savings he or she thought discovering the defect would provide. Not surprisingly, this came to a figure similar to 10 hours, as all the designers had undergone the same training. This training defined a major defect as one that is considered to give a savings of approximately 10 hours or more.

More significant, however, is the fact that the other success stories calculated their savings from inspections by a different method based on the estimation of the savings due to reduction in the rework cost of implementing the associated trouble reports and corrections. The range of ROI figures obtained by the other success stories ( 4-to-1, 5-to-1, and 6-to-1) are similar to the 4-to-1 figure mentioned previously, thus supporting and validating both methods of calculating the benefit.

Assessment of LDC

The champions received three days of auditing/assessment training from consultant Nigel Bauer in October 1997 to further encourage implementation. The intention was for the LDCs to assess each other’s practice of R&I on a peer-review basis. Toward the end of 1997 the first LDC assessment was conducted, which was perceived as beneficial for motivating further improvement. Other design centers were encouraged to carry out further assessment of each other’s practice of R&I.

Institutionalization

Because of the changing face of sponsorship, in 1998 the master LDC for R&I was appointed by Marie Olofsson (who was then responsible for development process management) to act as process owner for R&I within Public Networks Switching. The main function of the role was to help further institutionalize the practice there.

The assignment also required that the R&I champions network be kept running, with newsletters and twice-yearly self-financing forums for experience sharing between champions. External keynote speakers recognized as being leading experts in the field have always been invited. Creating and maintaining a Web page dedicated to support the R&I network was also included in the assignment.

It should be noted that now the adoption of the methods and participation of the LDCs within the network was voluntary as a result in the change of focus of the master role. It was no longer a requirement from the master process owners for LDCs to provide implementation plans detailing their level of R&I implementation. The level of local practice was now seen as their own responsibility. This move toward local ownership of their R&I practice has continued. Institutionalizing the process meant ensuring that they were:

• Defined and documented
• Supported
• Trained
• Practiced and enforced
• Measured and improved
• Tailored

CHANGE IN STANDARD PROCESS OWNER

Within Ericsson in 1999 there was a move away from the use of its own standard proprietary processes to making more use of externally available commercial tools and processes to help with the development of open systems for the future.

An agreement was made between Rational and Ericsson to make available supply support, and provide training of Rational Unified Processes to the users within Ericsson. This resulted in a major change in the "ownership" of the proprietary processes. Ericsson Business Consulting took over the responsibility for part of these. This was to ensure that the legacy systems that generated much of Ericsson’s revenue at the time were still supported.

The author’s LDC was given an assignment from Ericsson Business Consulting to supply support to the users and act as mentor to the new owner of the R&I process. This was because this process was based upon the Public Networks Switching R&I process of which the LDC had previously been acting as developer and owner.

Part of the assignment received was also keeping the R&I network alive as previously described, except in a larger capacity, as the scope covered both the wireline (fixed network) and wireless (mobile) areas of Ericsson’s organization. Supporting this expanded area, however, was made easier because of the merging of the wireless and wireline design center’s operations resources, leading to a distribution of the R&I champions to cover both previous areas.

Expanded Responsibility

At the forums and also from the communications received during 1999 from the R&I champions representing both wireline and wireless, it was good to see that both organizations held similar views and had the same needs. This made it easier to support them.

At the R&I Champions forum in Umea, Sweden, it was agreed that the benefits that could be gained from implementing R&I were available to a much wider scope of applications than just design. It was also agreed that to keep up with the changing development environment, it was necessary to update the R&I process so it supported R&I of artifacts produced from a modeling environment.

In the end, the champions network agreed that a generic Ericsson R&I process needed to be developed that supported R&I of all types of outputs, including those from modeling, and that it should be adopted for use by all those currently working within Ericsson.

ESTABLISHMENT OF NEW IMPROVEMENT PROJECT

In early 2000 Ericsson Business Consulting secured the funding to begin a project to produce a generic process framework that could provide assets for implementation of the R&I in any conceivable situation. The subsequent project was called "Gforce" and its goal was to produce an R&I framework for supporting application by organizations of differing levels of R&I process maturity. This was to ensure that as many design centers and implementation projects would be able to make use of the output produced.

Along with Hans Ohrvall, the R&I process product manager, the Gforce competence team was established consisting of interested champions from the United States, Netherlands, Spain, Ireland, England, Croatia, and Sweden. To keep the costs of developing this new process framework down, many of the former R&I process elements would be reused (see Figure 5). The solution enabled the team to capitalize on recent improvements developed locally, as most LDCs used the old R&I process as their base and published their processes on the Web, thus making their identification and transfer easier.

A similar approach and structure was adopted by the Gforce project as the initial R&I initiative, namely by involving the LDCs with the review and development of the new process artifacts. Part of this approach involved the trial of the preliminary versions of the outputs at the last R&I champion forum, which was held at Tilberg in the Netherlands in September 2000.

CURRENT STATUS

Ericsson Business Consulting discontinued operations at the end of 2000. The part of that organization that formally owned the R&I process, as well as other selected parts of the former processes, was transferred to a new operation called Ericsson Process & Applications Consulting (EPAC).

Piloting of the process, tools, and training is currently being undertaken, and feedback from the pilots will be presented at the next R&I champions forum to be held in Dubrovnik, Croatia, in May 2001. It is hoped that Ericsson having its own propriety world-class R&I process framework will provide the company with a business advantage resulting from its implementation.

Future Plans

EPAC is keen on establishing the Ericsson Review and Inspection Academy, offering support to the local champions and providing them implementation advice, as well as making contacts and partnerships with leading institutions and consultants in the field. One of the main goals of the Gforce project was to align the process, tools, and training. The next step will be to unify these into a simplified, fully automated paper-less tool solution that addresses all needs of the users. A prestudy will be made to specify and analyze what is involved in its production.

CONCLUSION

It is gratifying to see just how many LDC R&I processes today are based upon the former R&I process. This is in spite of the fact that there is no longer a central improvement program driving membership; now all participants do so voluntarily.

Despite this, all forums are self-financing, and the R&I champions network is lively and well subscribed. One reason for this could be that the network acts as a reference group on behalf of the users toward the corporate standard R&I process owner and helps to suggest, participate, and direct the improvements.

Many of the most motivated LDCs participating within the network are now assessed internally within Ericsson as at level 3 of the Software Capability Maturity Model (CMM®). The measurements aspects, process improvement, and root-cause analysis capability of the R&I package are becoming more important as the LDCs strive toward CMM levels 4 and 5 and may require still further development.

It is to this end that the major update currently being undertaken by the Gforce project is to keep the process relevant in today’s paperless model-based development environments starting to incorporate already identified improvements necessary for CMM level 4 and 5 organizations.

Lessons Learned

Following are some of the lessons learned from the R&I initiative:

• Do not just produce process improvements without ensuring that adequate support is provided long term for their introduction, as the competence and processes have to be developed and it takes time for results to bear fruit.
• Ensure dialogue is established with the intended users so that what is produced serves the purposes of the users.
• For a global solution, the finished product and implementation must be flexible enough to allow for cultural differences in their ways
of application.
• The benefits of utilizing the improvement must be promoted and communicated to all target groups.
• Every possible means of communication—from newsletters, presentations, memos, phone calls, publications, and direct contact—must be used with details of success stories using real data.
• The use of local champions supporting and promoting the improvement is vital, as well as having a network set-up between them and the competence team for communication, exchange of ideas, feedback, and recruitment into the competence team from enthusiastic champions.
• The selected use of consultants using their areas of expertise to supplement and enhance the skills and company knowledge of the competence team.
• Highly motivated members of the competence team have a great impact on a successful outcome.

Bottom Line

The fairly low 4-to-1 ROI figure reported in (Clark 1997) is due to the fact that the high initial implementation costs were included within the calculation. The ROI figure for subsequent periods can be expected to be more favorable, especially with the introduction of process improvement and root-cause analysis within their inspection process by certain LDCs.

In spite of this, assume the figure for the master LDC is representative of savings obtained per LDC for calculation purposes. Then a "guess" can be made of the potential total savings gained from implementing the R&I initiative globally (assuming 20 LDCs both official and unofficial participate) over the last five years using the following formula:

Number of design centers x expected savings per year x number of years = total savings, so 20 x $1894.2K x 5 = $189.4M This rough-and-ready calculation, while extrapolated from only one local champion’s figures for the year 1997, is certainly valid to within + – $150M (supported by other LDCs reporting figures obtained for their ROI ranging from 4-to-1 to 6-to-1 based on real rework cost reductions calculations).

In summary, taking a ballpark figure for the central corporate sponsorship funding to be around $1.4M figure + – $0.7 M to seed corn the local R&I initiatives globally over the last five years certainly proves the case that whichever way one looks at these figures, from the corporate global perspective, a very good return on investment has been obtained. It should be noted, however, that R&I is the only one of the original initiatives that is still operating in a similar manner as it was originally.

References

Clark, J. 1997. The economics of inspections for ETL/XD in 1997. ETL/XD/BZ-97:137, Rev A. Internal Ericsson Ltd. document.

MacFarland, R. 1997. Final report, TTM 15 reviews and inspections group. ETL/XD/DZ-97:131, Rev A. Internal Ericsson Ltd. document.

BIOGRAPHY

Robert MacFarland joined Ericsson in June 1981 as a software system engineer. He later became a programmer and project manager. In 1994 he began working as a member of the software engineering process group as a principal process engineer. He spent much of his time helping to drive the introduction of CMM into Ericsson’s local design centers. During this period he acted as a leader of the R&I team. Since January 2001 MacFarland has been employed as a senior consultant within Ericsson Process and Applications Consulting, mainly working with reviews and inspections in different development environments. He can be reached at Ericsson Ltd., Ericsson Way, Burgess Hill, West Sussex, RH15 9UB England, or by e-mail at robert.macfarland@etl.ericsson.se.

If you liked this article, subscribe now.