 |
 |
|
|
 |
|
|
|
|
|
| GAO Report on Audits GAO Report on Audits This file contains an ASCII representation of the text of a GAO report. Delineations within the text indicating chapter titles, headings, and bullets are preserved. Major divisions and subdivisions of the text, such as Chapters, Sections, and Appendixes, are identified by double and single lines. The numbers on the right end of these lines indicate the position of each of the subsections in the document outline. These numbers do NOT correspond with the page numbers of the printed product. No attempt has been made to display graphic images, although figure captions are reproduced. Tables are included, but may not resemble those in the printed version. A printed copy of this report may be obtained from the GAO Document Distribution Facility by calling (202) 512-6000, by faxing your request to (301) 258-4066, or by writing to P.O. Box 6015, Gaithersburg, MD 20884-6015. We are unable to accept electronic orders for printed documents at this time. Cover COVER Office of Policy August 1993 AN AUDIT QUALITY CONTROL SYSTEM: ESSENTIAL ELEMENTS GAO/OP-4.1.6 Abbreviations ABBREV AICPA - American Institute of Certified Public Accountants CM - Communications Manual GPPM - General Policies/Procedures Manual IIA - Institute of Internal Auditors NALGA - National Association of Local Government Auditors NSAA - National State Auditors Association OIGs - Offices of the Inspector General PCIE - President's Council on Integrity and Efficiency PREFACE Chapter 0 Government audits, evaluations, and investigations assess the efficiency, effectiveness, and accountability of government agencies and their programs. These assignments provide information, unbiased analysis, and recommendations that the organization's customers and stakeholders use to make informed decisions. This guide is intended to reinforce the Government Auditing Standards on quality control; to provide helpful hints for use by federal, state, and local audit organizations in designing or improving their systems; and to ensure consistent quality products that can be relied on by the organizations' customers and stakeholders. This guide describes the approaches presently being used by GAO. While the General Policies/Procedures Manual and the Communications Manual provide guidance on the various facets of doing our work, this guide pulls together in one place the essential elements of GAO's quality control system. Today's total quality management environment offers excellent opportunities to reassess and continue to improve the quality control system that helps to provide customers and stakeholders the service to which they are entitled. Key questions that should be considered in assessing an audit organization's quality control systems effectiveness include the following. Are we: Doing the right jobs? Doing the jobs right? Getting results? Achieving consistent quality? These questions are pertinent regardless of the audit organization\'s role, mission, size, or constituency. A good system should also provide the audit organization with performance indicators and feedback from its customers, attesting to the consistency of quality work. Werner Grosshans Assistant Comptroller General for Policy OVERVIEW Chapter 1 THE NEED FOR AN APPROPRIATE QUALITY SYSTEM -----------------------------------------Chapter 1:1 Government Auditing Standards require each organization to have an appropriate quality system in place. The quality assurance system should provide reasonable assurance that the organization (1) is following applicable Government Auditing Standards and (2) has established and is following appropriate policies and procedures. The Government Auditing Standards quality control standard, the fourth general standard, states: "Audit organizations conducting government audits should have an appropriate quality control system in place and participate in an external quality control review program." THE IMPORTANCE OF AUDIT QUALITY ----------------------------------------- Chapter 1:2 A high-quality job greatly increases the probability that audit results will be relied on and recommended improvements will be seriously considered and implemented. The organization's reputation for consistent high-quality work helps ensure that decisionmakers will more readily and more assuredly accept findings and implement recommendations. Reputations are built over time by producing consistent, high-quality work. A hard-earned reputation is on the line with each product. To maintain and continue to build excellence requires total commitment on the part of every member of the team and the organization. Challenges to findings and recommendations can be expected. As an organization increasingly deals with tougher and more sensitive issues, challenges to its work increase. It is not unusual for various constituencies to believe that they would be better off if results could be disproved or called into serious question. A successful challenge demonstrating minor errors or inconsistencies may call into question the quality of work supporting the principal finding or recommendation. Regardless of the reason for the challenge, it can be successfully refuted by demonstrating that findings, conclusions, and recommendations are warranted and supported. An effective quality control system is the basis for ensuring that the results will meet customers' needs time after time and withstand challenges directed at them. -------------------- 1 This guide uses the word "audit" to include audits, evaluations, inspections, and investigations. It uses the words "auditor" or "staff" to include the range of skills and disciplines employed in such work. INVOLVEMENT OF TOP MANAGEMENT ----------------------------------------- Chapter 1:3 The quality control system should be rooted in top management's expectation of and insistence on quality and the principles, policies, and procedures by which it can be achieved and will be evaluated. For example, the following establishes basic goals and expectations that are a sound basis for GAO's quality planning and performance: "We seek to achieve honest, efficient management and full accountability in government programs and operations. We serve the public interest by providing policymakers with accurate information, unbiased analysis, and objective recommendations on how best to use public resources in support of the security and well-being of the American people." "Commitment to quality is the single most important principle governing our work." The Comptroller General and other top GAO managers participate in the early direction of work to be done and in major decisions at key points in assignment planning and performance. COMMUNICATING SYSTEM GUIDANCE ----------------------------------------- Chapter 1:4 The quality control system should define principles, policies, and procedures that will achieve the consistent quality of work that the organization expects. System guidance should establish what is expected at each phase of an assignment, leaving room for initiative and creativity on how it is done. It should be readily available to staff at all levels. For example, GAO maintains the General Policies/Procedures Manual (GPPM) and the Communications Manual (CM) to give guidance on achieving audit quality. Each chapter has a succinct policy summary, followed by procedures to be used in complying with the stated policies. In addition, GAO publishes more detailed guidance on technical subjects. Technical guidance publications are normally referred to as "Gray Books." A list of these appears in appendix I. GAO's guidance material is accessible either in hard copy or in electronic mode. PURPOSE OF THIS GUIDE ----------------------------------------- Chapter 1:5 An assignment can go wrong at any stage. It can be ill-conceived, improperly directed, poorly planned, badly implemented, and its results can be ineffectively communicated. For a variety of reasons, it can fail to meet its customers' needs. An appropriate quality control system identifies or flags those factors that could jeopardize the quality of an audit and establishes processes or procedures that promptly identify and correct problems before they occur. For example, faulty assignment design could be detected during referencing or in a report review stage, but that is far too late to deal effectively with the problem. At that point, little more can be done than to write around the problem, salvaging what is at best a bad situation. To be most effective and to reduce assignment cost, design flaws must be detected in the assignment planning phase or early in the data collection and analysis phase to allow for appropriate intervention and redirection. The purpose of this guide is to provide helpful hints for use by federal, state, and local audit organizations in designing their systems to ensure consistent quality products that can be relied on by customers and stakeholders. It raises key questions that managers and staff should be able to answer at key stages of the assignment. KEY QUESTIONS ------------------------------------------Chapter 1:6 Figure 1.1 illustrates key questions that an appropriate quality control system should address and the remainder of this guide's chapters attempt to address these key questions. Figure 1.1 (See figure in printed edition.) Selecting those jobs that will make a contribution--doing the right job. Each job requires resources that could have been used on another job. Most audit organizations have "must do" jobs. They also have considerable latitude in using the rest of their resources to seek a balanced portfolio--based on needs, capability, and resources. In exercising that latitude, staff should be able to answer questions such as: Is the job selection a wise one? Does it respond appropriately to a request or to user needs? Does the job help build staff capability? Are the benefits of the job greater than could have been obtained if other work were done? How do you know? (See ch. 2.) Ensuring the quality of each assignment--doing the job right. Doing a job right requires efficient use of resources and high effectiveness. Key questions include the following: Are assignment objectives clear and responsive to customer needs? Is the assignment scoped to meet objectives? Is the methodology appropriate? Is job planning adequate? Are staff motivated and well-supervised? Are assignment results effectively communicated? (See ch. 3.) Accomplishing intended results. Audit work is performed for a wide variety of reasons--to accomplish a range of objectives. Most jobs seek results that improve the auditee's operation. The right job done the right way provides the best opportunity to get desired results--the bottom line for the auditor and the audit organization. Were the results of our work used? Did we have a beneficial impact? Did we make the difference our work sought? If staff can answer those questions positively, they are providing the quality service that stakeholders can expect every time. (See ch. 4.) Demonstrating consistent quality. Care is taken to build quality into job selection, planning, performance, reporting, and followup. Individual jobs are given a final quality check before they go out the door. But how well have all those policies, procedures, and processes actually worked? Are you satisfied that they were followed, fit together, and accomplished intended results? Can we satisfy peers that the organization's work is of high quality, meeting applicable professional standards? (See ch. 5.) DOING THE RIGHT JOB Chapter 2 PURPOSE ------------------------------------------Chapter 2:1 To do the right job requires planning--long range and day to day. This chapter gives guidance for developing a planning system that should be in place to help an organization determine what jobs should be done immediately and what jobs should be done in the future. It should show how the mosaic fits together to achieve longer-range objectives. WHAT ARE THE RIGHT JOBS? ------------------------------------------Chapter 2:2 There is no shortage of good jobs. But with limited resources, each job that is done prevents another from being done. Good jobs should give way to better ones. Audit organizations must meet many requirements. Decisions must be made on what to do first and over time. Many factors influence those decisions. A good planning system can help ensure good choices. KEY FACTORS IN PLANNING ------------------------------------------Chapter 2:3 While audit organizations share the need to plan, no single planning system likely meets the needs of each. But answering some key questions can help develop quality plans: What are the interests and/or needs of the legislative (or other) body that the audit organization reports to? How effective are planning efforts in meeting longer-range legislative requirements and in addressing current issues as they arise? How good is the framework within which plans are developed? Does the planning system provide a good basis for making choices within and among programs for which the organization has auditing responsibility? In all but the smallest audit organizations, work focuses on many governmental programs and subprograms and on a range of objectives to make audits better and cost less. Sorting this out within a framework makes cross-comparisons easier and helps to focus what should be done. What is the planning horizon? How far does planning reach? A longer-range perspective helps in setting significant audit objectives or issues to be addressed that may be beyond the reach of individual assignments and are attainable only by a series of related jobs. Within available resources, how are individual assignments selected to best meet multiyear objectives? Is there a vehicle for integrating "must do" jobs to help meet longer-range objectives? A FRAMEWORK FOR PLANNING ----------------------------------------Chapter 2:3.1 Responsibilities included in mission statements are broad; planning to meet them requires a sharper focus. Planning works best when it is focused within a framework. Governmental programs or subprograms could provide that focus. Should each program or subprogram be a planning area within which economy, efficiency, effectiveness, accountability, and other objectives are sought? Should the framework encourage cross-cutting issues? Does it permit work that evaluates management and accountability across programs to be arrayed and evaluated in relation to other planning objectives? The planning framework and areas it comprises could vary. However, the one selected should represent top management's judgment of how best to address the areas of responsibility. Once approved, planning areas will likely be the focus of work for a considerable period. While an approved plan is an achievement, it should not be viewed in concrete; instead, it should change when managers consider it necessary. MULTIYEAR PLANS ----------------------------------------Chapter 2:3.2 A good framework provides planning focus--helping to determine the most productive jobs in a planning area--and getting the most out of "must do" jobs. Planning works best when it covers a period of years in which longer-range objectives can be sought. Individual assignments--with their own current accomplishments--can be planned as building blocks to broader, more significant accomplishments. The length of a multiyear planning cycle depends on the area that it covers, e.g., when programs are volatile, a shorter planning period is more appropriate. But even when the areas include volatile programs, planning beyond a single year is beneficial. The objectives sought by assignment building blocks need time to develop. KEY STEPS --------------------------------------Chapter 2:3.2.1 Key steps in multiyear planning include the following: Understanding the Area--An Overview. To plan for an area, the planner should know a great deal about it. He or she should be able to answer questions such as: - What programs and subprograms does it include? What are their objectives? - What are the national goals to which the programs contribute? What is their contribution and how do they relate to those of other program contributors? - How are the programs viewed by the legislature, the agency, the public, and other stakeholders? - What are the present and foreseeable issues? - What are the economic, technological, political, and social trends? - What is the dollar significance of the programs? What is the potential for savings? - What effect do the programs have on people? - Do potential problems of stewardship or accountability exist? Setting Objectives. Analyzing the planning area, along lines suggested by the overview, will likely suggest a large number of worthy objectives--with the potential for significant accomplishments. Although the potential for accomplishment may seem virtually limitless, available resources are not. A good multiyear plan can sort out alternative objectives and prioritize those that offer the greatest benefit given available resources. Developing Strategy. How should each objective included in the multiyear plan--culled out from other possible objectives--be approached? What strategy should be employed? Will building blocks be used? What is the role of each? How do they relate to each other? Is there work that must be done? Can mandatory jobs be designed to help meet other planned objectives? The strategy provides a roadmap for assignment planning. It identifies principal building blocks to achieve longer-range objectives. Providing a Basis for Measurement. How will you know when planned objectives are accomplished? Have significant results been identified and will progress be tracked and measured against them? Does the plan clearly establish what will happen when objectives are reached? Are checkpoints built into the process to help correct the course when things are not going as planned? COOPERATIVE DEVELOPMENT --------------------------------------Chapter 2:3.2.2 Responsibility for multiyear planning should be clearly defined. Getting the input of all who can make a contribution should also be unequivocal. The needs and interests of the legislature should be appropriately included. Does the plan have all the input needed to ensure that it has an organizationwide perspective? Is it based on a high level of subject matter knowledge and expertise? As appropriate, have legislative staff, agency officials, outside experts, stakeholders, think tanks, and interest groups contributed to the richness, vitality, and usefulness of the plan? Does the plan as developed represent the objective and independently derived judgment of the audit organization? Was that judgment enhanced by a comprehensive knowledge of issues and the factual basis for differing points of view that are seemingly inherent in connection with major national programs? TOP MANAGEMENT INVOLVEMENT --------------------------------------Chapter 2:3.2.3 Approved multiyear plans represent major organizational decisions about resource usage for an extended period. They set basic directions. Top management involvement is essential. This normally includes (1) providing guidance on plan development, (2) setting resource levels for each planning area after considering the needs of all planning areas, (3) reviewing plan proposals and approving them, and (4) evaluating progress and proposed updates. UPDATING THE PLANS --------------------------------------Chapter 2:3.2.4 Effective plans provide the overall necessary direction for the audit team. However, as time passes, the plans should be reviewed. As part of this assessment, the progress and the overall contribution of the work should be examined. Any significant factors requiring changes to the plans or the overall strategy should be identified. If necessary, resources may be shifted. SHORTER-TERM PLANNING ----------------------------------------Chapter 2:3.3 Individual assignments should logically flow from the multiyear plan and contribute to the mosaic structure. The shorter-term work plan identifies the specific assignments that the unit plans to perform and the resources they plan to use. Assignments that the organization decides to undertake should constitute a balanced portfolio, including jobs the organization must do, those it selects to meet established multiyear objectives, and those it sees as targets of opportunity. Targets of opportunity are jobs which were not included in the multiyear plan that offer immediate payoff. These assignments represent the organization's judgment on the best use of available resources to meet the various objectives. In considering jobs, staff should ask questions such as: Will the proposed job meet user needs? Does it fit into the organization's priorities? Is it part of a longer-range plan? How does it contribute to the plan's objectives? Why is this job the best choice? What benefits will it achieve? Will expected benefits exceed likely costs? At this stage, knowledge about job costs and benefits will probably be limited. But with limited resources available, jobs that are most likely to achieve the greatest benefits should receive priority. How sensitive is the job? What is the climate in which its findings and recommendations will be judged? By and large, assignments are performed to meet particular user needs and to obtain results. Some matters, however, are so significant that they must be pursued regardless of unyielding opposition or great sensitivity. That kind of decision should be made before the job is begun. Is the job "doable?" A job\'s viability should be considered as the job is planned. If a job is not doable, it may be possible to modify its objectives and still realize significant, worthwhile results. But a job should not be initiated or kept alive with the hope that things will fall into place later. Could another organization do the job? If there is a choice, an audit organization should do those jobs for which it is most clearly suited. In larger audit organizations, work plans can help alert field offices to upcoming work. They help to communicate planned work throughout the organization, encouraging cooperation and avoiding duplication. THE GAO EXAMPLE ------------------------------------------Chapter 2:4 GAO has broad audit, evaluation, and investigative authority covering federal agency operations, activities, and functions and those that are federally assisted. It also has legislatively defined responsibility to perform congressionally requested work. A high percentage of GAO's work is done in response to specific requests of congressional committees and members. Appendix II illustrates GAO\'s planning system. DOING THE RIGHT JOB Chapter 3 PURPOSE ----------------------------------------- Chapter 3:1 This chapter illustrates ways the quality assurance system can help in designing the job to meet significant objectives,implementing the design effectively, andpreparing the product to obtain desired results. JOB DESIGN ----------------------------------------- Chapter 3:2 The reasons for the job--its objectives--and the way it will be done are set by job design. In large measure, the job\'s success depends on the quality of the assignment\'s design. While changes can and should be made when they are needed as work progresses, they may be avoided or at least minimized by quality plans. SETTING OBJECTIVES ----------------------------------------Chapter 3:2.1 Objectives are the questions or issues that the audit will address. Clearly defined objectives are a must. Without such clarity, the scoping, planning, and performing become more difficult, if not impossible. It is during the assignment design that objectives should be focused. If they are vague or excessively broad at the start of work, time and money can be wasted by inappropriate methodology, scope, and resultant redirections. A useful approach is to state assignment objectives as questions and subquestions. This approach helps preclude the perception of prejudged outcomes, fosters discipline and precision, facilitates clarity, helps focus data gathering activities, helps establish underlying logic, helps in segmenting work, guides design, and helps structure the presentation of assignment results. SCOPING ----------------------------------------Chapter 3:2.2 Scoping an assignment normally involves matters such as the number of sites or field locations to be visited, time frames to be covered, and the type and depth of work needed to ensure that assignment objectives are accomplished and that all applicable audit standards are met. In establishing assignment scope, the team should consider questions such as: What elements of a finding are required? Is disclosing a particular condition enough or is it necessary to establish cause and effect? Will it be necessary to evaluate the condition against criteria? Will assignment findings relate only to the cases reviewed or will it be necessary to generalize them to a larger universe? What are the relevant sources of data? Who/what holds the data (people, data tapes, files)? Will the data be available? Are they likely to be reliable? What kind of information will be required? For example, will a judgmental sample be acceptable, or will a statistically-projectable random sample be required? When will work results be required? Determining scope may require tradeoffs. For example, a more narrow scope may result in a less powerful message. But it may be the best that can be done considering available resources and time. Is the more narrow scope acceptable? ESTABLISHING METHODOLOGY ----------------------------------------Chapter 3:2.3 Designing a job requires selecting the methodologies to be used in collecting, verifying, and analyzing data. A wide variety of methodologies exist, and the choice can be critical. You cannot choose what you do not know about. Staff must learn what methodological tools are available. They must know what methodologies can be used for what purposes and the strengths and weaknesses of each. They need to answer questions such as: What kinds of comparisons are required? Which methodology best meets assignment objectives? How will data be analyzed? Will the results that it produces be persuasive and meet applicable audit standards? Can it be done in the time available? Does its application require expertise or skills that are not available to the audit team? Can they be obtained from elsewhere in the audit organization or through the use of outside consultants or experts? TESTING ----------------------------------------Chapter 3:2.4 Before the more time-intensive data collection and analysis phase of the job begins, testing of sample transactions of key systems to ensure that the systems work as designed should be performed. More extensive testing will generally be performed during implementation. Testing helps to establish the extent to which systems and data sources can be relied on to meet assignment objectives. Through testing, staff should be able to answer questions such as: Is the system(s) working as intended? Have agency self-evaluations disclosed system weaknesses that affect planned work? To what extent can agency internal controls be relied on? Is the agency complying with laws and regulations that affect assignment objectives? If the work of others or computer-processed data is to be used, can it be relied on? Staff should also pretest the reliability and validity of data collection instruments. Models to be used by the assignment design should be verified and validated by testing. Testing of data sources and methodologies helps to determine whether the planned design will be excessively costly or time-consuming to operate. DETAILED JOB PLANNING ----------------------------------------Chapter 3:2.5 The job plan makes the design operational. Points that should be covered include the following: Will the selected methodology used meet objectives? Has the task analysis identifying the steps needed to accomplish objectives been done? Has the source, the type of data to be collected, and the method of analysis been chosen? Has the work been segmented? Have individual responsibilities been assigned? Does the work sequencing and established time frames meet customer needs? Have key decision points and management progress assessments been established? Have adequate supervision and monitoring been built into the plan? Have applicable auditing standards been considered and are steps sufficient to address them? Will the requester or auditee be kept apprised of progress through periodic briefings? What type of auditee comments (oral or written) are most appropriate? How will assignment results be communicated? An assignment plan should address these questions and will guide staff on what should be done at each stage as well as how it is to be done. JOB STAFFING -------------------------------------------Chapter 3: Highly skilled and well-trained staff are essential to effective assignment performance. Staff should have the range of skills collectively needed to meet assignment objectives. Depending on objectives, this could include auditors, accountants, financial specialists, social science analysts, attorneys, and evaluators as well as experts in a wide range of disciplines (e.g., transportation specialists, engineers, natural resource managers, psychologists, economists, actuaries, physical and computer scientists, mathematicians, and statisticians). Few audit organizations will have staff with all these disciplines and expertise in house, but they should be available through contracting when needed for a particular assignment. EFFECTIVE IMPLEMENTATION ----------------------------------------- Chapter 3:4 With the course set by the assignment design, the next step is for staff to do the work--to collect and analyze data. The design was carefully developed and tested. It should work--but will it? Were there unforeseen problems? What effect do they have on assignment success? Do team members really understand what they need to do? Do they know why it is important? Are they motivated to do it right? PROGRESS MONITORING--COURSE CORRECTION ------------------------------------- Chapter 3:4.0.1 Staff should follow the plan but not as if it is in concrete. Requesters' or auditees' needs should be kept in focus at all times. Team members should be alert for changing priorities -be proactive and not simply reactive. Assignment objectives should be renegotiated and the assignment plan changed when necessary to meet those needs. High-quality performance includes proper supervision and reassessments of the continued appropriateness of objectives and progress made to achieve them. Answers to these questions would serve the team well: Is progress on schedule? Should the methodology be changed or should objectives be redefined in cooperation with the requester? Is the evidence being collected reliable, competent, and sufficient? Should the questions that are being asked be modified to better accomplish objectives? Do data access problems require special attention? Performance quality is a shared responsibility. Staff evaluate their own performance and progress on a day-to-day basis. The auditor-in-charge or evaluator-in-charge oversees the assignment and keeps abreast of progress and problems in meeting milestones. Problems should be revealed promptly to permit the team to address and resolve them. Scheduled progress assessments with the team by supervisors or managers who have assignment responsibility are helpful to keep assignments in focus and address problems that arise and could derail the plan. For example, such meetings could be scheduled at the following points: At the start of the assignment. Agreement at that time helps define objectives and provides an understanding of the charge--what is to be accomplished? When the assignment moves from design to implementation. Is there a reportable message? Has the assignment been well thought out and planned? Are the scope and methodology adequate to achieve assignment objectives and meet requesters' or auditees' needs? Is the assignment still worth the expected resource investment? Questions such as these should be answered before extensive resources are used to gather and analyze additional data. When enough work has been done to assess how well the plan is working and whether assignment objectives are being met. In GAO, this meeting is held when one-third of the calendar days between the completion of job design and the estimated date for message agreement have elapsed. It reviews progress and determines whether modifications should be made in assignment scope, methodology, or resources. When fieldwork has been completed. The assignment's message should have been considered throughout the life of the assignment. It is developed early in the assignment but is constantly refined as additional evidence is gathered. A message conference promotes agreement on the message consistent with the evidence, the product type, timing, and format that best meets requester's or auditee's needs. MOTIVATION AND SUPERVISION --------------------------------------- Chapter 3:4.1 Job success is highly dependent on the team's commitment--both collectively as a team and individually as a team member. Assignment design sets the course. Assignment progress reviews and plan modifications help stay the course. However, they do not guarantee desired results; they do not ensure that results will be achieved. Fundamental to quality performance is a motivated team which understands how their work relates to overall assignment objectives. Team members and supervisors should ask themselves questions such as: Am I enthused about the job? Do I respect other team members, am I willing to communicate, and am I prepared to listen to others' views? Am I doing the segments that I am best qualified for? Did I, as supervisor, consider the staff's interests and strengths in assigning tasks? Were task descriptions clear? Did I and team members ask questions when necessary for a complete understanding of expectations? Am I and team members demonstrating personal pride and professional care? Am I and others requesting help and asking questions when necessary and making suggestions where possible? Is the team's supervisor getting firsthand knowledge of work being done--praising good performance and providing guidance when needed? Are team members receptive to guidance and counsel? Is the team's supervisor objective and fair in appraising performance? Are team members receptive and open-minded? PRODUCT PREPARATION ----------------------------------------- Chapter 3:5 All the hard work and contributions of motivated staff during the design, data collection, and analysis phases culminate in the assignment's product. The product is the part of the assignment that is perhaps most visible. It is the primary vehicle for obtaining desired results. Developing the product's message is a continuing process. It begins with assignment initiation and should be modified as facts are gathered and analyzed. Tentative findings and conclusions should be tested against accumulated evidence and discussed with the auditee to get valuable input and to validate both correctness and completeness. Even recommendations should be discussed with the auditee to help ensure that they are doable and will achieve desired results. When all or most of the evidence is in, the team and managers should meet to reach agreement on the product's message, timing, and format. This can be done formally in a message conference or more informally on simpler issues. The basic purpose of reaching such agreements is to get everyone's buy-in and keep surprises to a minimum. Such action should also facilitate timely reporting. The team and managers directly responsible for the assignment should be able to respond affirmatively to questions such as: Are the findings and recommendations based on competent, reliable, and sufficient evidence? Are they responsive to assignment objectives? Were all applicable auditing standards followed? Is the message clear, coherent, and persuasive? Can the recommendations be implemented? Will they correct the situation identified? Have the auditees' views been adequately considered? Has the requester been informed of the overall message so that no surprises occur when the product is issued? FINAL QUALITY CHECKS --------------------------------------- Chapter 3:5.1 At this point in the life of an assignment, those responsible for it are convinced of its quality. Is that enough for the organization? Two other quality tests--an independent verification of the evidence supporting the product (referencing) and product review--can help ensure quality. REFERENCING ------------------------------------- Chapter 3:5.1.1 GAO uses a process called referencing to provide an objective look at the support for the product--a careful evaluation of the evidence. It requires that the person performing this careful evaluation--the referencer--be independent, objective, and experienced. Referencers should have analytical ability and a thorough knowledge of applicable standards and audit organization policies and procedures. Based on their review, referencers should be able to answer questions such as: Have the workpapers received appropriate supervisory review? Are facts and figures correctly reported as determined by satisfactory evidence in the workpapers or by independent mathematical or other checks? Are findings adequately supported by the facts in the workpapers? Do conclusions and recommendations flow logically from the findings? Was the report coordinated with key officials within the audit organization (e.g., in GAO the Assistant Comptrollers General for Policy and Planning and Reporting, the Office of the General Counsel, the Chief Economist, and the head of other organizations may have an interest in the subject matter)? Were their comments adequately addressed and documented? Have the auditee's views been accurately reported and are points made in rebuttal accurate and adequately supported? Has a qualified person who is not involved in the assignment examined highly technical data? Are the results of that examination documented in the workpapers? Points, questions, and suggestions made by the referencer may disclose flaws that cannot be permitted in a high-quality product. They should be documented and carefully reviewed by the team leader and, where necessary, by higher level managers. Action taken on the referencer's comments should also be documented. The referencer should also be alert to pertinent evidence in the workpapers that either contradicts or calls into question facts or statements in the report (negative assurance). Such observations should be noted for management consideration. PRODUCT REVIEW ------------------------------------- Chapter 3:5.1.2 Product review is a quality check to provide assurance that higher level managers are satisfied with the overall quality of the product; the message is sound, addresses the objectives, and meets the customers' needs; the message is consistent with prior positions (if not, did the appropriate officials approve the change); key units of the audit organization had an opportunity to review the product and agreed with the message; and the auditee's views are appropriately reflected and key differences have been adequately addressed. GETTING RESULTS Chapter 4 PURPOSE ----------------------------------------- Chapter 4:1 Recommendations are made with expectations of getting prompt corrective actions that achieve desired results. The quality control system should provide a sound basis for getting prompt and satisfactory action from recommendations. Key elements for enhancing such action include high-quality recommendations, commitment, effective monitoring and followup, and special attention to key recommendations. This chapter provides guidance on this important aspect of audit effectiveness. How to Get Action on Audit Recommendations (GAO/OP-9.2.1) discusses the subject in greater detail and includes case studies. HIGH-QUALITY RECOMMENDATIONS ----------------------------------------- Chapter 4:2 The report demonstrates the problems in need of action, presents the needed corrective action, and identifies who can take it. High-quality recommendations pinpoint needed changes. In developing recommendations, the team should consider questions such as: Is it clear why a change is needed, why current conditions should not be allowed to continue, and what the underlying cause of the problem is? Does the recommendation include specific actions that should be taken and results that should be achieved? Is the recommendation convincing? Will it correct the root cause of the problem? Have alternatives been adequately considered? Is the recommendation feasible and cost-effective? Is it clear who should take the corrective action? COMMITMENT ----------------------------------------- Chapter 4:3 Commitment is believing in the recommendation and in its importance. Individual members of the team can test their commitment by answering questions like: Am I willing to devote the extra effort to demonstrate the merits of implementing the recommendation? Do I fully understand the environment in which the recommendation will be received and evaluated? Has the team explored options and alternatives to achieving intended results without sacrificing the goals being sought? Have we created a climate of helpfulness and cooperation that generates respect and confidence? Have I made the commitment to work with the audited agency to ensure successful implementation of the recommendation? Staff commitment to results is heightened when management emphasizes results as a major objective (measuring outcomes, not outputs); tangible results are highlighted and identified as successful outcomes; followup is emphasized as a significant and valued responsibility; appraisal and award/reward systems recognize and reinforce the value of audit outcomes and reward accomplishments and proactive, innovative, and creative approaches in achieving desired outcomes; and training programs reinforce these values and encourage staff to prepare action plans to get recommendations accepted and effectively implemented. MONITORING AND FOLLOWUP SYSTEMS ----------------------------------------- Chapter 4:4 Monitoring and followup systems should answer two key questions: What improvements were made as a result of the work? Did those improvements achieve the desired result? Determining actions that were taken on recommendations require active status monitoring of recommendations. Individual responsibilities should be defined and ground rules established that describe minimum required actions, documentation required, and the like. They should leave room for staff initiative to tailor followup actions to particular recommendations and the results they seek. How far should staff go in verifying not only that action was taken but, more importantly, that it achieved desired results? Staff should be able to answer questions such as: Are implementation plans and time frames consistent with recommended actions? Have the steps to carry out the planned actions been adequately documented? Are the auditees' estimates of dollar savings or other benefits reasonable? Generally, some limited testing of the implementation would be expected to ensure it is working as intended. For key recommendations, these tests are essential. SPECIAL ATTENTION TO KEY RECOMMENDATIONS ----------------------------------------- Chapter 4:5 While all recommendations should be aggressively pursued, some are so significant that added steps are needed to get them implemented: The significance of a recommendation should be communicated to the auditee as early as possible during an assignment. Key recommendations should be identified and highlighted in reports. If action on a key recommendation is not progressing, this should be communicated to auditee management. The head of an audited organization cannot be expected to focus on every recommendation; therefore, the auditor must carefully choose those matters that deserve this special attention. DEMONSTRATING CONSISTENT QUALITY Chapter 5 PURPOSE ----------------------------------------- Chapter 5:1 The foregoing chapters give guidance on how to build and use a system that promotes quality in assignment selecting, planning, performing, reporting, and following up. This chapter helps determine whether the quality control system has resulted in consistent quality throughout the organization measured by outcomes rather than outputs. It also discusses peer reviews and how they can help provide valuable feedback and ensure consistent quality. INTERNAL QUALITY REVIEWS ----------------------------------------- Chapter 5:2 How consistent is quality among products and product types within and among organizational units? Does the quality control system help ensure that quality is maintained each time, every time? Answering these questions requires reviewing and testing policies and procedures. It also requires assessments of the quality of individual audits after they have been issued. Over time, all organizational units should be reviewed and their products tested. Reviewers should be independent of the unit being reviewed. This postaudit quality review provides top managers with an independent assessment of the extent to which the organization complies with Government Auditing Standards and its own policies and procedures. In reviewing compliance with Government Auditing Standards and policies and procedures, questions should be answered such as: Are policies and procedures consistent with Government Auditing Standards? Are policies clearly stated and are they doable? Do they cover key matters on which guidance would be helpful? Are policies unnecessarily prescriptive or do they leave room for using initiative and objectivity in meeting assignment objectives? Are policies and procedures readily accessible by staff? Have staff been adequately trained in the organization's policies and procedures? How is compliance with policies and procedures assessed? Reviewing individual assignments provides valuable feedback to managers on how well selected organizational units consistently achieve the expected quality. The number and type of assignments selected for testing should provide a reasonable basis for making this assessment. In reviewing individual assignments, questions should be answered such as: Was the team collectively qualified for the tasks required? Did individual staff members meet applicable continuing professional education requirements? Do the workpapers indicate any unresolved questions concerning external or personal impairments to independence? Was there adequate evidence that a determination was made of applicable standards and that they were complied with? Were assignment objectives clear and responsive to requesters' or auditees' needs? Was the assignment scope adequate? Was methodology appropriate? Were data sources, methodology, and data collection instruments tested? Was a detailed audit plan prepared? Was the assignment plan effectively implemented? Were deviations from the plan consistent with Government Auditing Standards and appropriate to assignment objectives? Were the workpapers adequately documented, summarized, indexed, and reviewed? Was there evidence that supervision was timely, adequate, and responsive to staff needs and professional development? Was the reliability of computer-processed data established when their use was significant to assignment objectives? Was the reliability of significant data supplied by others appropriately established? Were applicable internal controls identified, tested, and appropriately relied on? Was compliance with laws and regulations applicable to assignment objectives appropriately tested? Were findings and conclusions supported in the workpapers and was the evidence relevant, competent, and sufficient? Were auditees' positions on findings and recommendations obtained and appropriately handled in report development and presentation? Was the report timely? Did the report meet Government Auditing Standards and organizational policies and procedures, assignment objectives, and requesters' or auditees' needs? Did conclusions follow reasonably from the findings? Were recommendations responsive to the root cause of deficiencies detected? Were they clearly doable and cost-effective? Was there adequate evidence that the facts in the report were independently referenced? Were referencer's questions appropriately handled? Was the report reviewed for logic and consistency of positions taken? AUDIT EFFECTIVENESS --------------------------------------- Chapter 5:2.1 An effective quality control system needs to do more than ensure the quality with which work was performed. It also needs to determine what the work accomplished and how the result was viewed by customers and stakeholders. This can be done by system approaches such as: Surveys of customers and stakeholders. Did they consider the work to be timely and responsive to their needs? Did they find it convincing and useful? Were they able to use the results--implement the recommendations? Do they believe that the recommendations achieved their intended results? Recommendation tracking and reporting systems. Is there a system in place to followup on all recommendations and to determine what action was taken with respect to them? How were implementing actions and their results tested? Was special attention given to key recommendations? Were accomplishments reported and major contributors identified? Performance measurement and award/reward systems. Do those systems measure and emphasize outcomes or do they give inordinate emphasis to factors such as report production? PEER REVIEW ----------------------------------------- Chapter 5:3 The fourth general Government Auditing Standards states: "Audit organizations conducting government audits should . . . participate in an external quality control review program." The following are some organizations that help audit organizations comply with Government Auditing Standards: The President's Council on Integrity and Efficiency (PCIE). PCIE publishes standards and detailed guidance for conducting reviews of the audit operations of the Offices of the Inspector General (OIGs). It operates a peer review program for the OIGs. The National State Auditors Association (NSAA). Peer reviews are conducted by NSAA in accordance with policies and procedures developed by the NSAA Peer Review Committee and approved by NSAA members. The program is administered by the National Association of State Auditors, Comptrollers, and Treasurers. National Association of Local Government Auditors (NALGA). NALGA has prepared a guide to help auditors meet the requirements of Government Auditing Standards. The NALGA Peer Review Committee responds to questions about the guide and helps to identify potential reviewers. The committee receives reports issued using the NALGA guide and will serve as a mediator or arbitrator of unresolved disputes between an audit organization and a review team. The Institute of Internal Auditors (IIA). IIA will arrange peer reviews using Government Auditing Standards or its own standards. The American Institute of Certified Public Accountants (AICPA). AICPA has defined requirements for the peer review of member firms. The AICPA \"Peer Review Manual\" includes guidelines covering audit engagements of state or local governmental entities, including those receiving federal financial assistance. The guidelines include questions for use when the engagement is subject to Government Auditing Standards. LISTING OF GAO\'S TECHNICAL GUIDANCE PUBLICATIONS (GRAY BOOKS) Appendix I Publication Number Title1 -------------------- 1 Gray books include more detailed guidance on technical subjects that are included in the GPPM and the CM. AFMD-2.1.1 ----------------------------------------------------- Appendix I:0.0.1 A Glossary of Terms Used in the Federal Budget Process (Rev. Jan. 1993) AFMD-2.1.2 ----------------------------------------------------- Appendix I:0.0.2 Critical Factors in Developing Automated Accounting and Financial Management System (Jan. 1987) -------------------- 2 Under revision. AFMD-4.1.0 ----------------------------------------------------- Appendix I:0.0.3 Establishing Government Auditing Standards (Aug. 1990) AFMD-4.1.1 ----------------------------------------------------- Appendix I:0.0.4 Government Auditing Standards (July 1988) OP-4.1.2 ----------------------------------------------------- Appendix I:0.0.5 Assessing Compliance With Applicable Laws and Regulations (Dec. 1989) AFMD-4.1.3 ----------------------------------------------------- Appendix I:0.0.6 Standards For Internal Controls In The Federal Government (1983) OP-4.1.4 ----------------------------------------------------- Appendix I:0.0.7 Assessing Internal Controls in Performance Audits (Sept. 1990) AFMD-4.1.5 ----------------------------------------------------- Appendix I:0.0.8 Evaluating Internal Controls in Computer-Based Systems: Audit Guide (June 1981) OP-4.1.6 ----------------------------------------------------- Appendix I:0.0.9 An Audit Quality Control System: Essential Elements (Aug. 1993) OP-4.1.7 ---------------------------------------------------- Appendix I:0.0.10 Performing GAO's Work: Where to Find Guidance and Help (Rev. Aug. 1993) OIMC-6.1.1 ---------------------------------------------------- Appendix I:0.0.11 Mission and Assignment Tracking System (MATS) Users' Manual (Aug. 1991) OP-6.3.1 ---------------------------------------------------- Appendix I:0.0.12 Message Conferences: A Guide to Improving Product Quality and Timeliness (Rev. June 1992) AFMD-8.1.1 ---------------------------------------------------- Appendix I:0.0.13 Guide for Review of Independent Public Accountant Work (Dec. 1988) AFMD-8.1.2 ---------------------------------------------------- Appendix I:0.0.14 Guide for Evaluating and Testing Controls Over Sensitive Payments (Rev. May 1993) OP-8.1.3 ---------------------------------------------------- Appendix I:0.0.15 Assessing the Reliability of Computer-Processed Data (Sept. 1990) IMTEC-8.1.4 ---------------------------------------------------- Appendix I:0.0.16 Information Technology: An Audit Guide for Assessing Acquisition Risk (Dec. 1992) IMTEC-8.1.4SW ---------------------------------------------------- Appendix I:0.0.17 Quick Reference for Automated Audit Guide (Dec. 1992) -------------------- 3 Software to accompany GAO/IMTEC-8.1.4. IMTEC-8.1.6 ---------------------------------------------------- Appendix I:0.0.18 Information Technology: A Model to Help Managers Decrease Acquisition Risks (Aug. 1990) OP-9.2.1 ---------------------------------------------------- Appendix I:0.0.19 How to Get Action on Audit Recommendations (July 1991) PEMD-10.1.2 ---------------------------------------------------- Appendix I:0.0.20 The Evaluation Synthesis (Rev. Mar. 1992) PEMD-10.1.3 ---------------------------------------------------- Appendix I:0.0.21 Content Analysis: A Methodology for Structuring and Analyzing Written Material (Mar. 1989) PEMD-10.1.4 ---------------------------------------------------- Appendix I:0.0.22 Designing Evaluations (May 1991) PEMD-10.1.5 ---------------------------------------------------- Appendix I:0.0.23 Using Structured Interviewing Techniques (July 1991) PEMD-10.1.6 ---------------------------------------------------- Appendix I:0.0.24 Using Statistical Sampling (Rev. May 1992) PEMD-10.1.7 ---------------------------------------------------- Appendix I:0.0.25 Developing and Using Questionnaires (July 1986) PEMD-10.1.9 ---------------------------------------------------- Appendix I:0.0.26 Case Study Evaluations (Nov. 1990) PEMD-10.1.10 ---------------------------------------------------- Appendix I:0.0.27 Prospective Evaluation Methods: The Prospective Evaluation Synthesis (Nov. 1990) PEMD-10.1.11 ---------------------------------------------------- Appendix I:0.0.28 Quantitative Data Analysis: An Introduction (June 1992) IMTEC-11.1.1 ---------------------------------------------------- Appendix I:0.0.29 Preparing, Documenting, and Referencing Microcomputer Data Base Applications (Apr. 1991) IMTEC-11.1.2 ---------------------------------------------------- Appendix I:0.0.30 Planning, Preparing, Documenting, and Referencing SAS Products (Aug. 1992) IMTEC-11.1.3 ---------------------------------------------------- Appendix I:0.0.31 Preparing, Documenting, and Referencing Lotus Spreadsheets (Nov. 1987) OSS-11.1.4 ---------------------------------------------------- Appendix I:0.0.32 Security Highlights (Apr. 1991) OIMC-12.1.2 ---------------------------------------------------- Appendix I:0.0.33 Writing Guidelines (Sept. 1986) OIMC-12.9.1 ---------------------------------------------------- Appendix I:0.0.34 TextFrame: Policies and Instructions for Producing Presentation Materials (Oct. 1988) OIMC-12.14.1 ---------------------------------------------------- Appendix I:0.0.35 Preparing Publications for Typesetting (Rev. Apr. 1993) OIMC-12.14.3 ---------------------------------------------------- Appendix I:0.0.36 Publishing Survival Guide (June 1990) AFMD-12.19.1 ---------------------------------------------------- Appendix I:0.0.37 How to Avoid a Substandard Audit: Suggestions for Procuring an Audit (May 1988) AFMD-12.19.3 ---------------------------------------------------- Appendix I:0.0.38 Guide to Federal Agencies' Procurement of Audit Services From Independent Public Accountants (Apr. 1991) AFMD-12.19.4 ---------------------------------------------------- Appendix I:0.0.39 The Chief Financial Officers Act: A Mandate for Federal Financial Management Reform (Sept. 1991) AFMD-12.19.5A ---------------------------------------------------- Appendix I:0.0.40 Financial Audit Manual (June 1992) OGC-15.1.1 ---------------------------------------------------- Appendix I:0.0.41 Guidance on Employee Ethics and Conduct (June 1986) OSI-16.1.2 ---------------------------------------------------- Appendix I:0.0.42 Investigators Guide to Sources of Information (Jan. 1992) OCE-17.1.1 ---------------------------------------------------- Appendix I:0.0.43 Discount Rate Policy (May 1991) DOING THE RIGHT JOB--THE GAO EXAMPLE Appendix II LEGISLATIVELY DEFINED ROLE --------------------------------------- Appendix II:1 The Budget and Accounting Act of 1921 gave GAO broad authority and responsibility to audit federal agencies and to report on all matters related to the receipt, disbursement, and use of public money. Through amendments to the 1921 act and other GAO-specific legislation, GAO audits federally assisted programs and government corporations to determine the extent to which accounting and financial reports fully disclose the financial operations of departments and agencies; assess whether financial transactions have been conducted in accordance with laws, regulations, or other legal requirements; evaluate whether public funds have been economically and efficiently administered and expended; assess the extent to which programs are achieving their intended purposes; and ensure consistent operation of financial accounting systems and the application of accounting principles, standards, and procedures. While granting the Comptroller General broad discretion to decide which programs to audit, the Congress retained the right to request specific GAO assistance. GAO has legislatively defined responsibility to respond to the requests of House and Senate Committees and to Joint Committees having jurisdiction over programs and activities. As a matter of policy, GAO assigns equal status to requests from Ranking Minority Members and to requests from committee chairs. To the extent practical, GAO also responds to individual members' requests. At present, about 80 percent of GAO's resources are used in response to congressional requests. THE TOUCHSTONE FOR PLANNING ---------------------------------------- Appendix II:2 In carrying out its legislatively defined responsibilities, GAO establishes quality as a driving force. The following is the mission statement on which its planning and performance is based: "We seek to achieve honest, efficient management and full accountability throughout government. We serve the public interest by providing members of the Congress and others who make policy with accurate information, unbiased analysis, and objective recommendations on how best to use public resources in support of the security and well-being of the American people. "Commitment to quality is the single most important principle governing our work." PLANNING FRAMEWORK --------------------------------------- Appendix II:3 To fulfill its comprehensive audit responsibilities, GAO's planning framework has a wide-ranging perspective. It fosters jobs that (1) make individual programs work better and cost less; (2) highlight those things that work best in particular programs or management areas so that they can be considered for application in others; and (3) determine whether programs are overlapping, duplicative, working at cross-purposes, or effectively contributing to defined national goals. GAO's planning framework is built around issues and issue areas. Typically, an issue area is a functional or major responsibility area that affects a major segment of American society and is tied to large government expenditures. An issue, on the other hand, is a significant topic or question of national importance within an issue area. Currently, GAO's planning is built around 36 issue areas. They evaluate effectiveness, economy, effici |
|
