Resource Manual for Customer Surveys Part 2

Archive

_________________________________________________________________________________
Document Name: Resource Manual for Customer Surveys Part 2
Date: 10/01/93
Owner: OMB
_________________________________________________________________________________
Title: Resource Manual for Customer Surveys Part 2

Author: OMB

Date: October 1993

Status: RO

3. ACTIVITIES IN CONDUCTING A

CUSTOMER SATISFACTION SURVEY

3.1 Introduction

This section provides a step-by-step guide to mounting a

customer satisfaction survey. The guide focuses on the

methodology of designing, conducting, and analyzing a survey,

not on administrative, clearance, or funding issues. (OMB

clearance is covered in Section 5.)

The survey-taking tasks have been divided into twelve stages --

beginning with early planning activities and ending with the

presentation and analysis of results. (See Figure 1)

While much of what is said applies to surveys generally, there

are other features that are unique to customer surveys. For

example, asking about customer attitudes and opinions requires

more careful consideration of the underlying concept that should

be measured than asking factual questions. In addition, to be

useful to managers, customer survey results must be related to

actions that the agency might take to alter customer perceptions.

3.2 Top Management Role

Top management needs to provide leadership in the activities set

out here, particularly at --

- Step 1, where the scope of the customer

measurement program is set.

Heavy involvement is also needed by top management at --

- Step 2, where an attempt is made to identify

the factors and characteristics that contribute

to customer satisfaction.

After Step 2 there may be a need for top management leadership

in refining the scope of the measurement program. In fact, an

extensive back and forth effort may occur between Steps 1 and 2

and ample time should be allowed for this. These first two

steps are far and away the most important and the most in need of

top management attention. On the other hand, while many of the

next steps in the survey-taking process will need monitoring by top

management, they can be carried out, for the most part, by

technical staff. In particular --

- Step 3, where decisions on scope are translated

into the activities needed to identify the

particular customer populations targeted for

study.

- Step 4, where information on how to contact the

targeted customers is developed. (This is also

referred to as constructing the sampling

frame.)

- Step 5, where the data collection method is

chosen (telephone, mail, face-to-face).

Direct top management action may again be advisable at --

- Step 6, where decisions on who will actually

conduct the survey are made (e.g., contract it

out or do it in-house).

The remaining steps, except the last, are largely operational in

nature and need to be implemented by a team of in-house survey

experts or an outside contractor. These are --

- Step 7, developing and pretesting the

questionnaire.

- Step 8, designing the statistical sample.

- Step 9, implementing techniques to achieve high

response rates.

Decisions on the overall sample size and the response rate goals

of the survey have cost implications, and top management needs

to be involved in those decisions.

- Step 10, monitoring and obtaining high uqality

data.

- Step 11, processing the survey data and

preparing it for analysis.

Obviously top management will be on the receiving end of the

survey results, as will many other levels in your agency. For

the last step. . .

- Step 12, summarizing and delivering results

to work well, top management needs to spell out beforehand how

it wants to see the results and, then, to refine its requests

based on what is discovered.

One of the key actions after a survey is completed is to

document lessons learned and recommendations for future sutdies.

No one survey of customers will suffice; an on-going measurement

program is needed so changes in customer perceptions can be

tracked over time. However, improvements in the way

measurements are made should be emphasized early in an on-going

effort ,even if partially at the sacrifice of consistent trend

data.

3.3 Presentation of Steps

Each of the twelve steps covered here is discussed in a more or

less consistent fashion: definitions of the activities are

given, suggestions on who should carry out the step are made,

considerations to think about and some cautions are provided,

and summarizations are included in separate boxes of the skills

needed for each step and the outcomes anticipated. To make the

ideas concrete, examples are interspersed throughout.

Figure 1. The Steps in a Customer Survey

__________________________________________________________

Step 1 Determining the Scope of the Survey Measurement

Program

Step 2 Identifying the Factors and Characteristics that

Underlie Customer Satisfaction

Step 3 Identifying the Target Customer Population for the

Survey

Step 4 Developing a Sampling Frame of the Target Customers

Step 5 Choosing a Data Collection eMthod Best Suited to

Your Customer Survey

Step 6 Choosing Who Will Collect the Survey Data From

Customers

Step 7 Developing and Pretesting the Survey Questionnaire

Step 8 Constructing the Statistical Design of the Sample

of Customers

Step 9 Designing Procedures to Achieve High Response Rates

in the Customer Survey

Step 10 Ensuring Quality While the Survey Data are Being

Collected

Step 11 Processing the Survey Data and Preparing Them for

Analysis

Step 12 Analyzing the Data, Summarizing the Results, and

Presenting the Fidnings

__________________________________________________________

Step 1. Determining Scope

As in all large-scale activities, an extensive planning effort

is needed to mount a customer satisfaction survey. The first

planning task is to answer the questions:

- What are your agency's products/services?

- Who are the customers of your agency?

These sound so basic, but prior experience teaches that coming

to an agency consensus on the answers may be difficult.

Top and middle management should participate, along with your

agency's front-line staff, in answering these questions. Input

from all levels of the organization is necessary, since the

products and customers look different depending on one's

position in the organization. The Executive Order addresses

programs that serve the public directly, but you may also identify

other agencies, private organizations or your own agency staff as

customers. It may be useful to distinguish between internal and

external customers.

Another major aspect in determining overall scope is deciidng --

- What should the specific survey goals be?

Part of the answer here is to look at how the survey information

will eventually be used to develop service standards. Usually

it will be necessary to make hard choices about which key issues to

measure. You should be preparde to rethink these choices once

initial survey results become available.

____________________________________________________________

To Determine the Scope of the Survey Measurement Program

____________________________________________________________

You Should Have:

* knwoledge of the agency's products and services

* experience with attitudinal measurement issues

____________________________________________________________

You Should Produce:

* identification and definition of agency's

products/services and customer groups

* statement of survey goals, keeping in mind the

eventual use of the data is to set service standards

____________________________________________________________

Step 2. Defining Customer Saitsfaction

Customer satisfaction is a many faceted concept; it is clearly

not just the answer to the single question, "How satisfied are

you?"

- How then should customer satisfaction be defined?

The agency should listen to customers and front-line employees

to obtain their defintiions of the agency's services and the

attributes that determine satisfaction/dissatisfaction. The

agency also needs to know the criteria used b ythe customers to

evaluate the various products and services. It is miportant to

talk to as diverse a group of customers as possible, so that no

major perspective or point of view is omitted.

The ingredients of satisfaction for agency customers must be

identified. Overall satisfaction is affected by customer

expectations, their ideal or required performance, and actual

performance of the agency. The agency must identify activities

that might affect the satisfaction of its customers and what

aspects of the service are important. For some agencies this

may require separate measurements of satisfaction for

- the agency's products (e.g., nature and size of benefit),

and

- the service concerning those products (e.g., helpfulness of

staff).

Important activities must be measured along with satisfaction,

so that the relationship between agency actions and customer

satisfaction can be observed. For example, in an agency

processing benefits, the timeliness of the process (regardless

of outcome) may be a major determinant of satisfaction. Ideally,

the agency should be able to show, quantitatively through the

survey results, the degree to which changes in agency actions

affect customer satisfaction.

____________________________________________________________

To Identify the Factors that Underlie Customer Satisfaction

____________________________________________________________

You Should Have:

* knowledge of the agency's products/services

* experience with attitudinal measurement issues

____________________________________________________________

You Should Produce:

* lists of agency activities that affect customer

satisfaction

* descriptions of components of satisfaction as seen by

customers

____________________________________________________________

Step 3. Identifiyng Target Customers

This step is where decisions on the scope of the survey program

are translated into the activities needed to identify the

customer populations targeted for study. Federal agencies serve

distinct and diverse groups of customers. For example, some

serve individuals and families; others serve organizations.

- Who are the specific groups of customers you want to survey?

The customer types and the products/services have already been

identified in Step 1, and the criteria for measuring customer

satisfaciton have been agreed upon in Step 2. Now, that work

done in Step 1 has to be translated t othe context in which the

survey is conducted.

Key segments of the customer base and other characteristics of

the customers should be identified for measurement in the

survey. For example, the agency may already have records of

contacts with some customers but may have to develop new

recordkeeping systems to survey other customers.

Ideally, the agency will choose for initial studies those

combinations of products and services, on one hand, and

customers, on the other, that will contribute most to the

overall performance of the agency. It may be prudent, though,

to begin the customer survey program with studies of those on

whom records already exist, while other information soruces are

being developed.

____________________________________________________________

To Identify the Target Customer Populations for the Survey

____________________________________________________________

You Should Have:

* knowledge of products and services key to agency

success

* knowledge of sampling statistics and of what kinds of

customer lists are appropriate for survey use

____________________________________________________________

You Should Produce:

* descriptionso f key customer groups and

products/services to be studied in the survey

* ideas for expanding the set of customers that are

accessible for later surveys

____________________________________________________________

Step 4. Developing the Sampling Frame

Having decided upon your target customer population, you can now

identify some means to select a sample of them. To do so, you

should develop a "sampling frame."

- What is a sampling frame?

It is a list o fcustomers with addresses, telephone numbers, or

some other information indicating how customers can be

contacted. This information can be distilled from various sources.

For instance, reviewing administrative records; collecting names,

addresses, etc., at point of contact; or locating indirect

customers by contacting direct customers. Bear in mind that

frames vary in quality -- completeness, rececny, inclusion of

information for contacting the customers, etc.

Certain subclasses of customers that may be of special interest

in a satisfaction survey may be difficult to enumerate. For

example, it ca nbe challenging to contact discouraged or

infrequent customers, or the customers of agencies that have no

formal documentation of services provided. The ability to make

general statements about your agency's customers based on the

survey results is linked to how complete and accurate your frame

is.

A frame can include information on the transaction per se or on

the customer per se. This (whether the frame includes transaction

data or customer data) will have statistical repercussions that

must be taken into consideration. For instance, if you are using

a frame that contains contact/episode/transaction data, single

customers can appear in the frame many times, leading to problems

of multiple records.

In many cases, the information required to construct a frame is

inaccessible or difficult to obtain because it is considered

confidential. In other cases, once the frame is constructed, the

mix of information can be such that the frame itself becomes

confidential. It is important to separate administrative uses of

the frame from survey uses of the frame.

____________________________________________________________

To Develop a Sampling Frame of the Target Customers

____________________________________________________________

You Should Have:

* knowledge of applied statistical sampling techniques

* knowledge of administrative systems that generate

information on the agency's customers

____________________________________________________________

You Should Produce:

* description of existing customer lists

* documentation on how to access existing customer data

bases

* documentation on limitations of alternative customer

lists for selecting the custome rsurvey sample

____________________________________________________________

Step 5. Choosing the Data Collection Method

There are a number of different approaches to conducting a survey

-- self-administered mail questionnaires, telephone surveys and

face-to-face surveys.

- Which data collection method is best suited for your customer

survey?

The type of frame developed for the survey will determine, or

limit, the choice of data collection methods. For instance, some

frames have only customer addresses, making mailed questionnaires

the most feasible choice. Other frames come from individual

telephone or face-to-face contact of customers with the agency.

In any case, the nature of the service may affect the type of

data collection methodology that can be used.

Each type of data collection method comes with its associated

costs. Self-administered mail surveys are the least costly, with

telephone surveys running second, and face-to-face interview

surveys being the most expensive. Data collection methods also

vary in the amount of time they take to conduct. Telephone

surveys are the fastest, face-to-face surveys run escond, and

mail questionnaires are the slowest (because of the time required

to return questionnaires and complete follow-up activities, using

the mail). In addition to cost and time, data collection methods

vary along other aspects that can ultimately affect the quality

of survey results -- response rates, quality of answers provided,

and credibility of confidentiality pledges. Overall survey

quality is a complex mix of these and other features connected to

the data collection method.

____________________________________________________________

To Choose a Data Collection Method Best Suited to Your

Customer Suvrey

____________________________________________________________

You Should Have:

* knowledge of alternative survey methodologies

* experience in survey design for diverse populations

____________________________________________________________

You Should Produce:

* documentation of the various methods considered

* description of data collection approach to be used

____________________________________________________________

Step 6. Choosing the Survey Team

Next you need to decide what group should be charged with

collecting the customer survey data. In particular --

- Having decided on the basic data colleciton method, how do

you implement it?

Data collection can be undertaken by agency staff who have direct

contact with the customer, yb agency staff removed from customer

contact, by another government agency, or by an independent

contractor. Whatever choice your agency makes, the manner of

contacting the customer and the interviewer asking questions must

strive to elicit responses in an objective manner, giving the

sampled customers freedom to express satisfaction or

dissatisfaction. For some contexts this objectivity will be

difficult to obtain if the staff who provide the service are

also used to collect the survey data.

When interviewers are used, they should receive training in those

aspects of the agency's services that should be known in order to

record customer feedback accurately. This step may be needed

even if in-house staff is used in the interviewing, to assure

uniform interviewing behavior. Regardless of who does the

interviewing, they should be trained in traditional survey

interviewing techniques.

In addition, whoever does the data collection should have a good

track record for designing and conducting attitudinal surveys

with high cooperation rates. Remember that prices of data

collection of equal quality can vary across units, and the agency

needs to be alert to judging the best value among alternative

sources. Also, as the survey process can take several months to

more than a year, continuity in data collection should be

considered.

____________________________________________________________

To Choose Who Will Collect the Survey Data from Customers

____________________________________________________________

You Should Have:

* ability to make cost and quality tradeoffs

* knowledge of skills, flexibility, and quality of work

of independent data collectors

____________________________________________________________

You Should Produce:

* description of characteristics of data collector

* if data collection is contracted out, description of

procedures for financial and technical oversight

____________________________________________________________