Public Housing Authority Accreditation July 14, 1998

Table of Contents

Standard Setting for Public Housing Authority Accreditation............................................... 2
Introduction ............................................................................................................................ 2

Methodology .......................................................................................................................... 2

Chapter One: PHA Evaluation Models and Accreditation..................................................... 3
PHMAP.................................................................................................................................. 3

Standard & Poor's Public Housing Authority Evaluation Criteria ........................................... 4 National Community Housing Forum ..................................................................................... 5 Comparison of the Three Models ............................................................................................ 6

Chapter Two: Variables in Setting Accreditation Standards ................................................. 7
Introduction ............................................................................................................................ 7

Quantitative vs. Qualitative Standards..................................................................................... 8 Result vs. Process Measures ................................................................................................... 8 National vs. Local Standards................................................................................................... 9 Weighting Factors................................................................................................................. 10 BOOZ·ALLEN & HAMILTON

Chapter Three: Setting Accreditation Standards for Public Housing.................................. 11
Designing a Comprehensive Accreditation Program ............................................................. 11

Setting the Standards ............................................................................................................ 14 Examples: Issues in Standard Setting ................................................................................... 15 Next Steps ............................................................................................................................ 16 Conclusion............................................................................................................................ 17

Appendix I: Comparison of Model Standard Areas ............................................................ I-1

Appendix II: Comparison of PQA Standards and Potential Accreditation Standards ...... I-1

Table of Exhibits

Exhibit 1: PHMAP Indicators: Relative Weights......................................................................... 4
Exhibit 2: Comparison of Models ............................................................................................... 6
Exhibit 3: Comparison of Quantitative and Qualitative Standards ............................................... 8
Exhibit 4: Result vs. Process Measures ....................................................................................... 9
Exhibit 5: National vs. Local Standards .................................................................................... 10
Exhibit 6: PQA Evaluation Factors ........................................................................................... 11
Exhibit 7: Possible Examples of Accreditation Standards............Error! Bookmark not defined.
Exhibit 8: Possible Examples of Accreditation Standard Grading.............................................. 14
Exhibit 9: Steps For Creating Accreditation Standards.............................................................. 15

Standard Setting for Public Housing Authority Accreditation

Introduction

In a previous study¹, the Council of Large Public Housing Authorities (CLPHA) demonstrated that it was
financially feasible to create a Public Housing Authority Accreditation Agency. This body would improve
the quality of public housing service provision and evaluation through a peer-driven, outcome-oriented
accreditation process.

CLPHA members support the idea of accreditation and recognize that accreditation offers benefits over existing models of evaluation. Chief among these benefits is the fact the review process would be directly responsive to PHA industry concerns:

• Areas for measurement would be selected by PHA professionals and stakeholders, not by external
  organizations;
• Standards for those areas would be proposed and ultimately established by PHA professionals and
  stakeholders with hands-on understanding of relevant issues;
• Low performance results could be met with technical assistance rather than sanctions or exclusion from
  funding.

1 _{Feasibility Study on the Establishment of a Public Housing Accrediting Board, prepared for the Council of Large Public Housing}
Authorities by Booz·Allen & Hamilton, December 1997.

Booz·Allen & Hamilton Inc. ii 2/18/99

• Is designed by the PHA industry and its stakeholders;
• Is easily understood (process, results) by the PHA industry and its stakeholders;
• Measures all significant aspects of a PHA's operations; and
• Uses a variety of indicator types to measure compliance with the standards.

Chapter One compares the existing models of public housing evaluation. Chapter Two describes issues
surrounding standard setting. Chapter Three presents notional examples of how a comprehensive public
housing accreditation system might look, based on the discussions in the previous two chapters, and
provides examples of other accrediting agencies' experiences with developing their standards. While the
process and decision points are common among agencies, each accrediting agency ultimately faces different
issues. This report thus is only an introduction to the process that CLPHA will undertake.

Methodology

The study methodology involved benchmarking several accrediting bodies and a review of findings from literature and professional associations. Organizations with which we conducted benchmarking interviews and/or standard reviews included the National Committee for Quality Assurance (NCQA); the National Community Housing Forum (NCHF); the Council on Accreditation of Law Enforcement Agencies (CALEA); the Continuing Care Accreditation Committee (CCAC); and the Council on Accreditation of Services for Families and Children (CAFSC). Additional material was developed from other agencies interviewed for the Feasibility Study. We also reviewed a number of secondary sources including Public Productivity and Management Review, National Institute of Standards and Technology, American Society of Quality Control, and materials from other accrediting bodies. Finally, we applied added perspective from in-house Industrial/ Organizational Psychologists obtained during previous accreditation, measurement, and standards projects.

Booz·Allen & Hamilton Inc. 2 2/18/99

Chapter One: PHA Evaluation Models and Accreditation

This chapter compares the emphases, strengths and weaknesses of three public housing performance measurement models. The three models include:

• HUD's current Public Housing Management Assessment Program (PHMAP) system;
• Standard & Poor's Public Housing Authority Evaluation Criteria; and
• National Community Housing Foundation (Australia) draft accreditation standards for private, community-
  based low-income housing providers.

Implicit in this comparison of the three models is a comparison to an ideal accreditation program for public
housing. Therefore, our review of the strengths and weaknesses of the existing models leads to a proposed
direction in Chapter Three for CLPHA to follow in designing a comprehensive accreditation program.

PHMAP

The purpose of PHMAP is to assist HUD in its oversight of local housing authorities' ownership and operations of public housing.2 PHMAP is meant to be objective, with scores to be consistent across all PHAs, so PHMAP indicators largely are quantitative or "yes/no" measures. For example, indicators include:

• Achievement of certain vacancy rates;
• Amounts of unexpended funds and unobligated funds;
• Number of findings related to adequacy of contract administration;
• Rents uncollected as a percent of total dwelling rent to be collected;
• Number of work orders completed within a certain number of days or number of days by which order
  completion has been reduced; and
• Annual inspections of units and systems.

According to the PHMAP regulations, there is an "implicit" measure of the adequacy of the PHA's
management systems in these indicators. The On-Site Confirmatory Review Guidebook (74605.G) guides
the HUD reviewer through a series of questions on the adequacy of systems. The final score, however, is
based on the achievement of the objective measures.

As shown in Exhibit 1, PHMAP emphasizes asset management, followed by financial management. Indicators of tenant service compose only 9 percent of the total PHMAP score. This weighting may ensure that PHA managers focus on safeguarding HUD's investment in the physical stock but--insofar as the management principle "what gets measured, gets done" is true--it undervalues the social mission of public housing.

2 _{HUD is revising PHMAP. The new standards are not available for review at this time.}

Booz·Allen & Hamilton Inc. 3

In summary, the strength of PHMAP is that it allows clear measurement and comparison across PHAs on certain outputs and results that are of importance to property and asset managers. The weaknesses of the model are that it does not adequately account for the underlying management systems in the scoring, nor does it include all standard areas that might be of importance to the PHA industry and its stakeholders.

Standard & Poor's Public Housing Authority Evaluation Criteria

While Booz·Allen was unable to analyze the specifics of Standard & Poor's (S&P) Public Housing Authority Evaluation Criteria because the methodology is proprietary, we can make some general comments nonetheless. The S&P Public Housing Authority Evaluation model focuses on many issues of concern to public housing managers. These include3:

• Quality of management--the organizational structure and experience of management, the presence of a
  focused business strategy and sound budgeting and control systems;
• Property management-- cost efficiencies, maximization of rental income, control of vacancies to minimize
  rent losses, cost and quality of maintenance services, experience and training of personnel;
• Asset management--evaluation of written policies and procedures, quality of financial and information
  systems, property surveys and budgeting for repairs, comparison of financial measures to other federally
  assisted multifamily housing and PHAs;
• Development programs--use of HOPE VI, acquisition and rehabilitation, relationship with local
  redevelopment authority;
• Condition and marketability of the housing stock--capital needs, tenant income mixture, curb appeal,
  market conditions;
• Financial flexibility--adequacy of maintenance and reserve levels, other sources of income, budget
  accuracy, banking relationships; and

3 _{Source: S&P briefing materials provided to CLPHA}

Booz·Allen & Hamilton Inc. 4 2/18/99

• Relationship with federal, state and local governments--plan to address impact of welfare reform,
  relationship with HUD, relationship with municipal services.

The value of the S&P criteria is two-fold: first, the evaluation does focus on important issues of property
management, asset management, and owners and developers of low-income housing; second, the result of
the evaluation allows capital markets to assess PHAs as credit risks. Nonetheless, while many of the
indicators provide useful information to the PHA, the model lacks the transparency and focus needed for
PHA stakeholders to understand, or provide input to, the evaluation design and process.

National Community Housing Forum

Community housing in Australia serves a similar purpose to public housing in the United States--it is rental housing for low- and moderate-income households with rents set in keeping with the tenant's ability to pay. There are differences in structure, however. First, although the housing is publicly funded, it is managed by community-based organizations, and tenants often are members of the organization. Second, the housing may be owned either by the government or by the housing organization. Third, housing may take the form of apartments or rooming houses, and may be targeted toward tenants with special needs. The purpose of NCHF's standards is both to set common standards for the community housing sector nationwide and to provide guidance on best practices to community housing organizations. The intent is to provide these organizations with standards to enable quality improvement and accountability, as well as to demonstrate the effectiveness of community housing. NCHF designed the accreditation program to reflect the underlying principles and values of the community housing sector; thus, the categories that follow strongly reflect the organization's social mission:

*	Tenancy management;
*	Asset management;
*	Tenant rights and participation;
*	Working with the community;
*	Organizational management;
*	Evaluation, planning and service development; and
*	Human resource management.

Moreover, because the standards must apply to a variety of organizations, NCHF intentionally designed them to be qualitative and process-based. Examples of standards include:

• Does the organization have clear guidelines on the different types of responsive repairs with set time frames
  and systems for responding?
• Does the organization have a system for reviewing staff performance (with certain defined elements)?
• Is the organization involved in local area planning on housing matters?

The strength of this model is the flexibility and comprehensiveness it provides. A significant weakness,
however, is that there are no quantitative targets for vacancy rates, financial reserves, etc.; instead, each
housing organization only needs to track the data and act accordingly. Therefore, the evaluation of how well
the organization's processes meet the needs may be a subjective one.

Booz·Allen & Hamilton Inc. 5 2/18/99

Comparison of the Three Models

A high-level comparison of the three models suggests that they share many standard areas but treat them differently. PHMAP emphasizes physical and financial conditions; S&P emphasizes financial viability; and NCHF emphasizes organizational effectiveness and mission achievement. Again, each model brings strengths and weaknesses, as shown in Exhibit 2. Exhibit 2: Comparison of Models

Booz·Allen & Hamilton Inc. 6 2/18/99

Chapter Two: Variables in Setting Accreditation Standards

Introduction

This chapter describes the most important variables and decision points to be considered during development of accreditation standard: Quantitative vs. Qualitative Standards

• Qualitative standards are numeric and measurable;
• Quantitative standards require judgment of more subjective factors.

Result vs. Process Measures

• Result measures are measurements of the intended end result of a program;
• Process measures are interim measurements of the process which yields the end result.

National vs. Local Standards

• National standards apply equally everywhere;
• Local standards provide local flexibility against a general standard but the specifics are tailored to
  community needs and market conditions

Weighting Factors

• Accreditation standards can have equal weights, or can be assigned different weights to reflect differences in
  relative importance.

In identifying the accreditation variables and identifying the key issues for each, Booz·Allen obtained
information from several organizations that have established accreditation standards. These included the
Council on Postsecondary Accreditation, the National Institute of Standards and Technology, the National
Council for the Accreditation of Teacher Education, and the American Society of Quality Control.
Additionally, we drew from our proprietary database of related projects to identify accreditation best
practices.

Booz·Allen & Hamilton Inc. 7 2/18/99

Quantitative vs. Qualitative Standards

Quantitative and qualitative standards measure different
things, but both contribute to a holistic understanding of an
issue or program. There is value to each kind of
measurement. Exhibit 3 defines the terms, identifies the
benefits of each type of standard and provides examples.
Exhibit 3: Comparison of Quantitative and Qualitative Standards

Clearly, there is no "correct" answer that applies in all cases. Once the items to be measured have been identified, both qualitative and quantitative standards must be considered.

Result vs. Process Measures

Result measurements are nearly always used in accreditation and program evaluation, the National Community Housing Foundation of Australia being a notable exception. Exhibit 4 defines the terms, identifies the benefits of each type of standard and provides examples.

Booz·Allen & Hamilton Inc. 8 2/18/99

National vs. Local Standards

National standards are those that apply to all jurisdictions in exactly the same way, meaning that all jurisdictions are expected to conform to the same standards. Local standards are those which can be established by each local authority (e.g., Housing Authority). Determining whether a given standard (accreditation criteria) should be national or local is determined by considering several test questions as shown in Exhibit 5.

Booz·Allen & Hamilton Inc. 9 2/18/99

Weighting Factors

Most evaluation and accreditation schemes use
multiple rating criteria. Thirty criteria are not an
unusually high number. Clearly, with such a large
number of criteria, not all will carry equal weight.
Some factors are more important than others in
evaluating quality and overall performance. The
benefits of factor weighting include:
* Allows focus on the "critical few" measures;

• Recognizes the fact that some factors are upstream, or causal, to others; and
• Permits organizations to prioritize benchmarking and improvement efforts.

The generally accepted practice in assigning weighting factors is to have a panel of subject matter experts
independently assign weights and statistically confirm agreement.

Example: the Presidential Quality Award

The President's Quality Award (PQA) Program recognizes federal organizations for excellence in performance and customer service. The Office of Personnel Management presents the PQA in recognition of public organizations that provide high quality products and services to their customers. In making its determinations, OPM evaluates federal organizations against seven categories, as shown in Exhibit 6.

Booz·Allen & Hamilton Inc. 10 2/18/99

Since the focus of the PQA standard is what the organization does in each area and how the organization evaluates and improves performance, these elements measuring quality could serve as the review areas for an accreditation program, regardless of the program's focus (housing, education, etc.). Moreover, because the PQA is a federally recognized quality assessment system, CLPHA might consider incorporating these elements into its accreditation program as a means of building the program's credibility.

Chapter Three: Setting Accreditation Standards for Public Housing

Designing a Comprehensive Accreditation Program

Given the relative strengths (and weaknesses) of the models described in Chapter One, a comprehensive accreditation system based on these models would have the following characteristics:

• It would address the full range of standard areas to reflect the mission of the PHA industry and its
  stakeholders, with appropriate--but not necessarily equal--weights on each indicator;
• It would look for systems and policies to achieve nationally recognized best practices in organizational
  management, real estate and property management, and tenant service; and
• It would measure quantifiable performance indicators that reflect both national benchmarks and the local
  market, to assess the effectiveness of those systems and policies.

Using the list of standard areas drawn from the earlier Feasibility Study, examples of standards that a public
housing accreditation program could include are shown in Exhibit 7 on the following page. Other areas for

Booz·Allen & Hamilton Inc. 11 2/18/99

• 
  
• Three-year average vacancy rate. This indicator would be compared to a (weighted) average for other
  assisted multifamily housing developments in the market area. It would be a quantitative, locally derived,
  and result-oriented indicator whose benefit is to demonstrate the PHA's effectiveness at meeting market
  need.

Booz·Allen & Hamilton Inc. 12 2/18/99

Booz·Allen & Hamilton Inc. 13 2/18/99

• 
  Grading Based on the Standards: As described in the Feasibility Study, accreditation status generally
  includes unconditional (full) accreditation, conditional accreditation (for a shorter period than the full
  accreditation cycle), and denial of accreditation. Thus--again, depending on what the accreditation program
  is meant to achieve--evaluators could grade a standard on unit inspections and repairs in a more meaningful
  way than PHMAP does, as illustrated in Exhibit 8.

Exhibit 8: Possible Examples of Accreditation Standard Grading

Again, it is up to the public housing industry to select the areas for measurement, set the standards and indicators, and decide how to grade them.

Setting the Standards

Accreditation agencies have used a similar procedure for setting program standards. The four organizations interviewed on this question all prepared draft standards--either through an industry panel or by employees who were assisted by an industry panel; released the draft for comment and made adjustments; field tested the standards; released the field test results for comment and adjusted the standards; and then released the standards. Three of the four organizations also created a formal process for regularly updating the standards. See Exhibit 9.

Booz·Allen & Hamilton Inc. 14 2/18/99

3= Yes º= No

NCHF's experience may be instructive for CLPHA. NCHF's process was as follows:

• Initial standards were developed in one state:
  -Housing providers held a workshop to discuss different approaches to accreditation and standards.
  -The state government agreed to fund the accreditation project; the housing providers appointed a
  consulting firm to draft the standards.
  -Consultants held focus groups with tenants.
  -Consultants held a two-day workshop with representatives of stakeholder organizations (housing
  managers, government, other service agencies and tenant groups) to identify the agreed-upon aims of
  housing services and to discuss approaches to accreditation.
  -A small group of stakeholders and the consultants spent a week developing the first draft of the
  standards.
  -The draft was distributed for comment; three workshops were held to obtain further comments.
• The draft standards were revised and then were brought to housing managers and tenants in the other states,
  and discussed with other government administrators.
• Finally, the standards were revised again and then presented to a national workshop of stakeholders for
  ratification.
• The standards will be piloted in 11 housing service providers.

Lessons Learned: Issues in Standard Setting

Interviews with accrediting agencies surfaced four significant issues for consideration. Iterative nature of standard setting. First, there is the recognition that standard-setting is an iterative process involving comment by both facilities themselves and their stakeholders (e.g., consumer groups,

Booz·Allen & Hamilton Inc. 15 2/18/99

2. Stringency. Second is the question of how exacting the standards should be--i.e., "how high to set the
   bar." Initially, benchmark targets for each indicator may be developed by compiling operating statistics
   from all PHAs (internal benchmark) or other multifamily housing projects (external benchmark) to
   determine what is reasonable for the industry. Then, however, the accrediting agency must decide
   whether the standards should be set to a minimum that most PHAs can reach now, to a mean standard
   that half of the PHAs can achieve, or to a high standard which many PHAs may not achieve in their first
   attempt at accreditation. Both CCAC and NCHF reported that they adopted a maximum standards
   approach.

3. Focus on results. An accreditation program such as CALEA's does not include quantitative results
   indicators. CALEA specifically does not want to mandate that law enforcement agencies make a certain
   number of arrests in a year, for example. NCHF similarly does not call for an explicit measure of results.

NCQA recently concluded that it should include more results indicators in its accreditation program, however. Agency members realized that health care providers were scoring well in accreditation based on their processes but were not having the outputs and outcomes those processes were meant to achieve.

4. Weights. Finally, there is the question of weighting the standards. Weighting is a necessary process for
   standard setting; where no weight is assigned, each standard carries equal weight. CLPHA may choose
   to weight standards for social goals equally with standards for property and asset management goals.
   This would mean that CLPHA, like NCHF, places more importance on the achievement of social goals
   than does a system like PHMAP.

Accrediting agencies handle this question differently. Some of CCAC's members argue against attaching weights in their standards because they feel that weights prescribe a level of importance to one standard over another. The American Corrections Association does not use weighting either, although it does differentiate between mandatory standards (those related to life and safety issues) and non- mandatory standards; ACA requires the facility to meet all mandatory standards and 90 percent of the non-mandatory standards. Other accreditation agencies find weighting to be a valuable way of focusing management attention on higher-risk or higher-value areas, however. NCQA, for example, uses weights to focus management attention on certain areas, and it develops its weights through an iterative process based on informed opinion.

Next Steps

There is a generally accepted practice in developing accreditation standards, which is to convene an expert panel of industry members to decide what aspects of a program or operation should be included in the accreditation program and how those aspects should be measured. Thus, CLPHA's next step would be to:

Booz·Allen & Hamilton Inc. 16 2/18/99

• Convene an internal panel of representatives from recognized "high performing" PHAs to take a first cut at
  identifying those aspects and measures. These representatives should have:
  -Direct experience in implementing and executing best practices
  -A full awareness of industry trends
  -A breadth of experience in function areas across housing organizations of different sizes and
  geographical locations
  -An ability to commit to a multi-week series of meetings

We expect the standards development committee to be comprised of 6-8 members who will be engaged at a 50-60% level of effort.

• The panel will propose a set of standards to serve as the starting point for discussions with a broader range
  of stakeholders. They might use the following questions to direct their formulations:
  -What are the hallmarks of a high-performing PHA?
  -How are these hallmarks best measured?
  -Should standards be set against minimum quality, "best-in-class" or in-between? What impact will
  that have on PHAs' ability to be accredited in the first cycle?
  -How should PHMAP be reflected in accreditation?
• Finally, the panel should identify stakeholders and conduct a stakeholder analysis (i.e., what does each
  stakeholder understand and agree to about the direction to take for each issue?) This analysis will enable the
  panel to anticipate stakeholder concerns and develop a strategy for obtaining the stakeholders' support.

The panel will sit for a number of meetings, on a schedule to be determined by CLPHA. A typical schedule
includes at least 2 meetings per week, with face-to-face meetings strongly recommended due to the
interactive nature of the development process. During the pilot phase, involvement of the committee would
be reduced to a 15-25% level of effort. There will likely be additional support from CLPHA and/or
contracted staff for support throughout this process.

In total, we estimate that developing accreditation standards for CLPHA will take at least six months, with pilot testing requiring an additional six months; other organizations have taken as long as two years.

Conclusion

As noted before, the standard setting process itself is fairly straightforward. It is the actual identification of the standards and selection of indicators that is complicated. CLPHA has three models of public housing evaluation, and an internal set of goals, from which to begin developing an accreditation program. Moreover, there are many resources in the form of existing accreditation agencies with which CLPHA representatives can confer as they get into the process. Ultimately, however, the benefits that accreditation offers the public housing industry are those that the stakeholders create themselves through the standard setting process.

Booz·Allen & Hamilton Inc. 17 2/18/99

Appendix I:
Comparison of Model Standard Areas

Issues that PHMAP, Standard and Poor's Public Housing Authority Evaluation Criteria* and the National
Community Housing Forum's Draft Accreditation Standards address in common (albeit through different
indicators) include:

*	Property management					-	Case reserves
		-	Rents uncollected	*	Modernization
		-	Vacancies and turnaround time	*	Tenant programs
		-	Work order timeliness
		-	Unit and system inspections	*	Relationship with HUD

*Financial management

Issues that S&P* and/or NCHF address that PHMAP does not address, include:

*	Organizational management			*	Office environment
		-	Board structure	*	Asset and property management
		-	Board experience			-	Impact of vacancies on rent losses
		-	Information management			-	Cost of work orders
		-	Internal control systems/accountability			-	Property development
		-	Policies and procedures

*	Financial management			*	Tenant management -Tenant selection
		-	Budgeting systems			-	Tenant retention
		-	Financial benchmarks			-	Tenant rights and responsibilities
		-	Financial stability				(including eviction)
*	Planning					-	Mixed income
		-	Strategic planning	*	External relationships
		-	Capital finance planning			-	State and local government
		-	Portfolio evaluation and improvement			-	Community organizations and service
			program				providers
*	Personnel					-