Resource Manual for Customer Surveys Part 3

Archive

_________________________________________________________________________________
Document Name: Resource Manual for Customer Surveys Part 3
Date: 10/01/93
Owner: OMB
_________________________________________________________________________________
Title: Resource Manual for Customer Surveys Part 3

Author: OMB

Date: Oct 1993

This is part 3 of 7 sections of the new OMB Customer Survey Manual.

Text follows:

Step 7. Developing and Pretesting the Questionnaire

Perhaps one of the most critical components for obtaining useful

feedback on customer satisfaction from surveys is the

questionnaire or survey instrument.

- How do you ask questions in the survey that best illuminate

levels of customer satisfaction?

The key is careful development and pretesting of questions so

that you can be reasonably sure that the survey answers reflect

the customers' true opinions. For instance, when answering

satisfaction questions, respondents have a tendency to answer in

a positive way. The questionnaire should, therefore, deal with

both attitudes and experiences, and include multiple questions

for key dimensions of service quality (e.g., courtesy,

competence, reliability, communication), each with a slightly

different focus. Asking about an important service attribute in

several ways provides more complete and reliable measurement.

Because respondents may be affected by the wording and ordering

of questions, tracking satisfaction over time requires that the

questionnaire remain as constant as possible for key indicators.

However, most organizations find that initial satisfaction

measurement techniques can be improved. Thus, there is a tension

between desire to measure change and the desire to improve the

quality of the survey.

The following methods are often used to pretest a questionnaire:

unstructured one-on-one interviews; investigations of

respondents' comprehension of terms in proposed questions; focus

group discussions of questions; use of cognitive laboratories.

After a questionnaire is developed, it should be pretested under

real conditions. This kind of field testing is to a survey what

a dress rehearsal is to a play. The questionnaire is tested

using a small scale version of the entire survey, with all design

features in place.

____________________________________________________________

To Develop and Pretest the Survey Questionnaire

____________________________________________________________

You Should Have:

* survey questionnaire design experience

* applied attitudinal measurement experience

____________________________________________________________

You Should Produce:

* documentation of changes in question wording,

question order, and questionnaire structure

through the developmental process

* quantitative and qualitative data as evidence

supporting changes in the questionnaire

* draft of the final questionnaire

____________________________________________________________

Step 8. Constructing the Statistical Design of the Sample

There are many options for the design of a sample for a customer

survey.

- How do you design the sample for the customer survey?

It is the strongly held view of most experts that results of

customer surveys cannot be interpreted with confidence unless

probability sampling is used. In general, the larger the sample

size the more precise are the survey results, but other aspects

of the sample design also affect survey quality. For instance,

stratification of customers into subgroups or classes is a

commonly used technique to ensure that the customers in the frame

represent the agency's actual pool of customers.

Oversampling of a geographic area or population subgroup may be

desirable to permit the presentation of regional statistics, or

characteristics by different age or ethnic groups. If your

agency has area or regional offices, separate samples of the

areas served by different offices may be useful. Each of these

sample design features affects the statistical properties of the

survey results.

The ideal respondent is usually the person who most directly

receives the service. In some cases the respondent may be the

person who contacted the agency on behalf of someone else, e.g.,

a wife calling about her husband's claim. If an organization,

rather than an individual, is the customer, the choice of

respondent becomes more complex, and there may be multiple

customers of interest in each organization. In these situations,

identifying and sampling several different staff members in each

client organization may be desirable.

____________________________________________________________

To Construct the Statistical Design of the Sample of

Customers

____________________________________________________________

You Should Have:

* knowledge of probability sampling theory

* experience in sampling from complex sampling

frames

____________________________________________________________

You Should Produce:

* description of sample design

* description of sample selection procedures

* estimated levels of precision of survey estimates

____________________________________________________________

Step 9. Achieving High Response Rates

Regardless of how well the survey is designed, there will be some

customers selected for the survey who will not respond.

- What can you do to achieve adequate response rates?

Low response rates can lead to misleading survey results, since

the people who did not respond may be unlike the survey

participants. The following methods usually prove useful in

ensuring high rates of response. Each of them also may have

implications for the cost efficiency of the survey.

- Advance notification: The respondent is contacted ahead of

time and informed about the survey, including its goals and

its provision of confidentiality, and is encouraged to

participate.

- Ease of answering questions: Customers will more likely

answer questions that are easy to comprehend and to answer.

- "Friendly" questionnaires: For mail questionnaires use a

personalized graphical design that is simple, attractive and

easy to read -- booklet style may make it look smaller and

easier to complete.

- Repeated follow-up: Those not immediately responding are

sent postcard reminders and additional questionnaires in

mail surveys. Repeated calls are used for noncontacts in

telephone and face-to-face surveys.

____________________________________________________________

To Design Procedures to Achieve High Response Rates in the

Customer Survey

____________________________________________________________

You Should Have:

* knowledge of how nonresponse error relates to

overall survey quality and survey objectives

* knowledge of alternative survey strategies to

reduce nonresponse and likely reactions of agency

customers to such strategies

____________________________________________________________

You Should Produce:

* documentation of survey design features to reduce

nonresponse

* estimation of survey cost implications of efforts

to reduce nonresponse

____________________________________________________________

Step 10. Ensuring Quality During Data Collection

Now you are ready to conduct the survey. However, one of the

biggest management tasks remains...

- What can you do to ensure quality while the data are being

collected?

There are a number of indicators of survey quality that should be

sought throughout the entire survey process. The specific

indicators that are available will depend on how the survey is

conducted. Special attention should be paid to:

- the percentage of sampled customers contacted

- interviewer response rate performance

- extent of questions read as worded

- extent of use of non-directive probing

- questionnaire completion rates as the survey progresses

- response rates of individual survey items

- daily records on problems arising in the data collection and

calls coming into the agency regarding the survey.

If interviewers are used, they must receive high quality training

and continuing supervision, including checks of a sample of their

work. The questionnaire should be administered with no variation

from instructions. Any variation at this stage could give rise to

biased answers from sample customers.

____________________________________________________________

To Ensure Quality While the Survey Data Are Being Collected

____________________________________________________________

You Should Have:

* experience in survey data collection management

* knowledge of techniques for intervening in ongoing

data collection to resolve problems and improve

quality

____________________________________________________________

You Should Produce:

* report on technical quality of data collection

* documentation on limitations of data that may

affect conclusions

____________________________________________________________

Step 11. Processing the Survey Data

Once the survey is conducted, the survey answers need to be

placed in a form useful for analysis.

- How do you convert the data to a useful form?

Transforming survey answers into data usually entails coding

(i.e., grouping into categories answers given in the respondents'

own words) and data processing steps (e.g., key entry of numeric

data into computer files). To be processed and analyzed, the data

must be put into computer readable form. The data set is

formatted to be compatible with a statistical software package.

It is important to include in the data set information on the

probabilities of selection and other design features

corresponding to each interview record.

The questionnaires will seldom be "perfect," in that rarely are

all questions answered in their entirety. Some responses will be

missing, and the survey results can be affected if those missing

answers are for distinctive types of customers. In attitudinal

surveys, the answer "don't know" can be informative and should be

maintained as a response in its own right.

____________________________________________________________

To Process the Survey Data and Prepare for Analysis

____________________________________________________________

You Should Have:

* knowledge of data entry designs

* knowledge of computer file design for statistical

analysis

* knowledge of statistical software packages

____________________________________________________________

You Should Produce:

* computer data set ready for analysis in

statistical software

* documentation of editing and coding procedures

____________________________________________________________

Step 12. Summarizing and Delivering Survey Results

Once a survey data set is ready, the data must be analyzed to

learn what they can tell you about customer satisfaction.

- What analytic techniques should be undertaken?

- How can survey information be effectively communicated to

management?

The statistical analytic techniques useful at this stage depend

on the nature of the data collected. Customer satisfaction

statistics sometimes are simple percentages of respondents who

chose a particular response category. At other times complex

statistical models are used to summarize the answers to many

questions simultaneously.

In all cases the presentation of survey statistics should be

accompanied by available measures of their quality, such as

standard errors, confidence intervals, levels of missing data,

response rates, and statistical comparisons of multiple

indicators of the same concept. Graphical presentations of

results often communicate more clearly than numbers and text.

The survey results should be disseminated to all levels of staff

that were involved at the initial stages described in Step 1,

with the form of presentation tailored to the audience. Reports

should be designed for the concerns of the given level.

Presentations should strive for clarity and brevity. Remember

that certain key facts, such as question wordings and the

population on which the statistics are based, should be included

in all reports. Information that is operationally useful

(actionable) should be part of the presentations. Most

importantly, the survey reports should be timely, for the value

of the data diminishes as time passes.

____________________________________________________________

To Analyze the Data, Summarize the Results, and Present

the Findings

____________________________________________________________

You Should Have:

* knowledge of statistical data analysis

* ability to write clearly on technical matters

* knowledge of agency organization and activities

____________________________________________________________

You Should Produce:

* multiple reports on survey results for alternative

agency groups

* commentary on actionable items for future

improvement of customer satisfaction

* feedback on how survey results inform the setting

of service standards