Public Service Benchmarking - A Leadership Dialogue

Public Service Benchmarking - A Leadership DialogueM was held May 27, 2004 in Toronto, Ontario. The forum, co-sponsored by the American Society of Quality and the Institute for Citizen-Centred Service, offered participants the opportunity to hear leading practitioners from American, Canadian, and British jurisdictions explore their successes and challenges with public service benchmarking. It also provided an opportunity to identify critical issues and establish shared strategies for advancing the benchmarking agenda.

The Agenda for the Day

• What are the issues you run into in implementing a public sector benchmarking program?


• What are the most important performance indicators to benchmark across public organizations?


• What common measurement tools are available to measure these indicators? What new common measurement tools might be needed?


• What, if any, organizational arrangements will be needed to make public sector benchmarking a reality across North America, and internationally?


• How do we move the public sector benchmarking agenda forward from here? What are some concrete next steps?

The Presentations

The day began with Noel Landuyt speaking about his group's work with the State of Texas and beyond. Working within a legislated mandate, they use a "tripod" approach to public service review and development, namely: employee attitudinal surveys, customer service assessments, and leadership training.

Their measurement of employee satisfaction is at a mature stage of development and is conducted using the Survey of Organizational Excellence (SOE). Customizable for the individual organization, its administration is almost entirely electronic, with on-line survey invitations being e-mailed to employees. Agencies receive real-time reporting of aggregated data upon completion of the designated survey time period, with the reports offering both internal and external comparisons.

Their measurement of customer satisfaction is at a less mature stage of development but still very impressive. Building on their experience with employee satisfaction, their Customer Service Assessment process uses a combination of "point of service" and sampling-based surveys and, as with the SOE, real time reporting is offered. They have also developed a method of automatically analysing and routing on-line complaints, compliments, and suggestions to the appropriate state departments, allowing for immediate response to e-mail and online input.

Finally, Noel shared information about the Leadership Excellence survey tool (3600 Feedback Instrument) and noted their practice of convening an annual forum on Best Practice.

Noel's presentation generated lots of questions and discussion. In particular, the group learned that the potential to politically punish agencies with the results has not been an issue to date. To the contrary, results are having a positive impact on agencies. In further discussion, Noel noted that survey results are aggregated each year to provide state-wide assessments for the Texas legislature.

The UK Experience - Ian McDonald, Office of Public Services Reform, Cabinet Office

Ian McDonald spoke about his government's customer-focused public service reform efforts. Operating for three years now, one of the key drivers behind the project has been what Ian described as the "Perception Gap", the observation that citizens tend to be satisfied with their personal service experiences with government, yet dissatisfied when asked to make general ratings about government.

Ian reviewed the customer-focused principles that they have developed and promoted across departments, and the principles of good customer feedback processes that have guided the project. Among the principles cited, he noted the importance of looking at segmentation issues. For example, the UK has observed lower levels of satisfaction with the young, black and minority ethnic groups, which has lead to questions of social justice issues that may need to be addressed.

In terms of overall progress, Ian noted two areas of headway: the increasing acceptance of the benchmarking agenda, with clear interest being expressed by politicians, and the increasingly robust nature of the data that is being collected. The main challenges to date: making the data simple to understand, effectively disseminating findings, ensuring findings are commonly owned, and effectively measuring areas involving complex relationship issues.

Ian closed his presentation with a review of the challenges they have identified, in becoming a truly customer focused government. He emphasized the importance of better understanding the customer's needs, so as to be able to design services and align organizations and process to meet them, and the importance of focusing on targets that are meaningful.

Ian’s presentation generated considerable discussion regarding the Perception Gap issue. It was noted that various political factors will influence people's response when asked to comment on their confidence in government's ability to improve public services, and that ideally it would be good to tease out those various influences in a survey. This led to a further discussion about the importance of marketing and branding government services.

OMBI: Partnering for Service Excellence - Ron Gibson, Director, Ontario Municipal Chief Administrative Officers Benchmarking Initiative

Ron Gibson spoke about the shared approach of 16 municipalities in identifying and benchmarking performance statistics and operational best practices. To date OMBI has mainly focused on objective service measures and is now beginning to look at customer satisfaction measures.

Ron reviewed each of OMBI’s major work plan elements, noting their activity-based costing model, their communication strategies, their benchmarking methodology, their relationship management activities, their training and best practice forums, and their data and web page management.

As to early outcomes being achieved by OMBI, Ron noted that most of the major achievements have been coming from the unintended consequences of adopting shared approaches. For example, municipalities have used benchmarking to help them with their blue box contract negotiations and to achieve more efficient Emergency Medical Service deployment.

Ron reviewed a number of challenges that OMBI has experienced in its benchmarking efforts. In particular, he noted the challenges of ensuring relevance of the benchmarking to the issues facing the CAOs, and of nurturing partnerships, maintaining focus, and generating comparable, reliable data. He also stressed the challenge of providing adequate monitoring and follow-up.

Finally, Ron shared a list of lessons learned through the OMBI experience, stressing the importance of top management support, the need to make a long-term commitment, and the importance of linking benchmarking to the organization's overall strategic direction. He also underlined the importance of focusing on process (over targets), of starting small projects on things that really count, and monitoring progress carefully.

Next steps for OMBI: expanding the project to cover all other municipal services, extracting betting meaning and value from the data, and better integrating performance data into the budgeting and planning cycle.

Keynote Address - Jonathan Walters, Governing Magazine

Jonathan Walters shared his thoughts on the issues confronting benchmarking initiatives and what some of the solutions might be. He introduced his presentation by noting how little interest and attention there seems to be in the notion of holding government accountable for providing high quality and efficient services. He presented his case that it seemed a reasonable goal to have data at least inform government decision making, acknowledging that the realities of politics preclude the likelihood that decisions will always be based on data.

He expressed his opinion that, contrary to the belief of some, performance measurement is not a fad that has run its course, noting that citizens, for example, will always care about the state of their neighbourhoods and how their children are doing in school. He noted, too, that politicians are often moved to action by tragic events resulting from poor services, such as child deaths, and that the challenge is to find an every day set of standards and indicators that will work for them.

Jonathan identified three roadblocks to the adoption benchmarking strategies, elaborating on each of them with case examples.

• Politics - Ideology gets in the way of a data-driven approach.


• Conspiracy of Convenience - We tend to measure what we can control.


• A Simplistic Approach to Data Interpretation - not understanding, misusing data

Jonathan then offered suggestions on how to overcome these roadblocks, citing three broad approaches:

• To develop better communication strategies (e.g. we tend to use too much jargon and indecipherable flow charts)


• To educate as many people as possible, using leverage points (e.g. leverage points might include national associations of cities and municipalities)


• To publicly shame legislators, by comparing them with other jurisdictions. (Jonathan noted his belief that this has to be come from outside government and offered his example of the now discontinued Governing Magazine comparative performance methodology.

Jonathan's presentation generated considerable discussion as to what structures or organizations may be necessary to provide the comparative performance role from outside government. Jonathan challenged the group to figure out who might pick up that ball.

Common Measurements Tool: Benchmarking Client Satisfaction - Charles Vincent, Institute for Citizen-Centred Service

Finally, Charles Vincent provided an overview of Canada's Institute for Citizen-Centred Service, described the development of the Common Measurements Tool (CMT), and outlined the Institute's current efforts to provide a benchmarking service for participating public service agencies.

Developed 6 years ago, and recently revised to allow measurement across multiple channels of access, the CMT offers a ready-made tool that can be used and adapted by public service agencies, and which allows like-organizations to compare results and avoid common survey mistakes. The CMT contains core questions that all public services are encouraged to ask, and offers a number of other questions that agencies may choose from when designing their own surveys. The core questions relate to known drivers of satisfaction and are the focus of the benchmarking initiative.

A CMT benchmarking database has been built and partners are beginning to submit data. While all benchmarking reports to date have been produced manually, the Institute is now developing an automated report capacity, which will allow partners to easily compare themselves anonymously against peer organizations.

Charles noted a number of challenges encountered in operationalizing the benchmarking function. Among them: establishing take-up and buy-in across a variety of public service organizations, ensuring consistent use and acceptability of data, ensuring confidence in the confidentiality of data sharing, and being able to offer acceptable comparators (especially in the early stages).

One discussion point arising out of Charles' presentation was whether or not specific satisfaction targets should be set. Charles noted that the primary value of the satisfaction measurement exercise is that it identifies areas of focus for service improvement efforts. He suggested that it can be strategically useful to set satisfaction improvement targets, but that they should not necessarily be the focus of the exercise.

Discussion

Having completed the formal presentations, the next task was to summarize key learnings and establish some practical next steps. To that end, Brian Marson facilitated a group discussion, asking participants to respond to the questions identified at the beginning of this report, namely:

• What are the issues you run into in implementing a public sector benchmarking program?


• What are the most important performance indicators to benchmark across public organizations?


• What common measurement tools are available to measure these indicators? What new common measurement tools might be needed?


• What, if any, organizational arrangements will be needed to make public sector benchmarking a reality across North America, and internationally?


• How do we move the public sector benchmarking agenda forward from here? What are some concrete next steps?

What are the issues you run into in implementing a public sector benchmarking program?

Presenters and participants generated the following list.

1. It is necessary to measure the right things.
This point had been emphasized by a number of speakers. It is important to measure things that really matter, not the things that may simply be easy to measure or politically attractive to measure.

2. It is important to communicate benchmarking results in a way that achieves understanding by those who read them.
Participants discussed the challenge of communicating results in ways that are simple to understand but which do not lead to a simplistic interpretation and misuse of the data.

3. There are challenges presented by ideology and politics.
It was noted that politicians are tempted to measure things they can control or that will give them the results they want to hear, not necessarily the things that are important. As such, it may be unreasonable to expect full political support for benchmarking strategies. It was suggested that benchmarking may be most effective when undertaken by third-party organizations.

4. The "I'm Different" syndrome is a common reaction among benchmarking participants, and makes buy-in difficult to achieve.
Especially in the early stages of a benchmarking, participants can react defensively to low scores, believing the proposed comparators to be inappropriate. The challenge: How do we find "apples to apples" comparisons that people will buy into?

5. We must remember - the purpose of benchmarking is to meet clients' and citizens' needs, to improve services.
Several presenters and discussion participants noted the importance of keeping focus on the main objective in all of this - that of meeting citizen and client needs.

6. It is difficult to set useful targets when you have achieved a good level of satisfaction.
Two points were noted with this challenge, namely: how does one help high achieving organizations continue to move forward and not become complacent; and are there points of diminishing returns that need to be recognized?

7. It is a challenge to maintain a constancy of purpose and commitment over time.
For there to be results from benchmarking, one has to be in it for the long haul and give it the support it needs.

8. There are special benchmarking challenges presented with evolving service channels.
With the rapid evolution of new access channels, there may be no established bodies of practice with which to benchmark. The increasing variety of service challenges also adds to the comparability challenge.

9. We must learn how to "get past the fear" to promote buy-in by service managers.
Benchmarking can create anxiety. Managers may feel that the survey tool or questions don't fit their situation, that their data won't be managed with the necessary level of confidentiality, or that their programs will be placed at risk.

10. Public service organizations need to have a clear strategic direction, to which they can link their benchmarking initiatives.
It was noted that private sector organizations are typically clear about where they want to be. For benchmarking to be effectively implemented, public service organizations need to identify their strategic direction and attach the benchmarking initiative to it.

11. We must use benchmarking approaches that recognize the "law of unequal development" across organizations.
It was noted that there is great maturational variation across organizations, and that any overall strategy must recognize and accommodate that reality.

What are the most important performance indicators to benchmark across public organizations?

Again, through a facilitated discussion, participants generated the following four categories of priority indicators.

What common measurement tools are available to measure these indicators? What new common measurement tools might be needed?

Again, through facilitated discussion, the group produced the following ideas.

What, if any, organizational arrangements will be needed to make public sector benchmarking a reality across North America, and internationally?

The tight time frames did not permit this question to be fully explored. It was suggested, however, that there might be ways of leveraging the work of the ICCS and the ASQ Government Division to create new organizational arrangements or establish communities of practice. It was also suggested that further attention be given to exploring possibilities for a third party benchmarking structure (as per the past Governing Magazine role).

How do we move the public sector benchmarking agenda forward from here? What are some concrete next steps?

Finally, through one final piece of facilitated discussion, participants generated a list of tentative actions.

Proposed Follow-Up Actions

1. Post a summary of today's event on a website.

2. Create a link between ICCS and ASQ websites.

3. Explore development of a message board and / or a listserve, to support the development of a community of interest.

4. Initiate a data benchmarking exercise among some of today's group members. (It was suggested that we take an indicator for specific service area, and benchmark)

5. Explore ways of aligning core metrics across jurisdictions (Can we check whether our respective approaches are compatible with each other's and can we establish some agreements?)

6. Establish more formalized partnership strategies to nurture emerging relationships. (For example, establish bilateral or trilateral arrangements, and take formal steps to build a "Community of Interest"

7. Undertake a more detailed discussion on the identification of benchmarking barriers and how to overcome them. Included among the issues to discuss, ways of engaging the support of elected officials.

8. Convene a future working session of this group, to advance the initiatives discussed at today's meeting.