” Background States have implemented a variety of modernization initiatives to improve access and delivery of the Supplemental Nutrition Assistance Program (SNAP) in order to manage increasing caseloads under budgetary constraints. Modernization initiatives can include the following types of activities: policy simplification, restructuring organizational and administrative functions, partnering with other organizations and implementing new technology. As these efforts have multiplied and expanded, so has the Food and Nutrition Service’s (FNS) interest in how States implement and monitor their efforts. Similarly, States have an interest in seeing how other States are performing. This exploratory study was conducted to determine the measures States have in place across eight modernization initiatives, including call centers, online systems, document imaging, kiosks, partnering, waiving the face-to-face interview, shortened interviews and online expedited applications. A review of SNAP performance standards and reporting requirements as they pertain to these modernization initiatives was conducted. The effort resulted in two reports. The first report integrates and summarizes findings across the 45 study States (44 states plus the District of Columbia) on how they monitor and measure performance. It also includes suggestions for performance measures and standards for FNS and States to consider. The second report delves into details on how each of the 45 States measures performance and describes the performance standards in use. Methods The study was conducted in two-phases. In the first phase, a broad group of stakeholders were interviewed for their perspectives on performance measurement activities in the States. During the second phase, data were collected from the 45 study States, more than 100 local agencies within those States, and more than 50 community and business partners who work with the States and local agencies. Findings States and local offices are collecting a large number of measures. Across the 8 modernization initiatives that were examined, States and local offices reported using 89 different performance measures to assess their activities (Table1). Measures are most commonly used for technology-related initiatives. Table 1: Number of States and Number of Measures for the Eight Initiatives Initiative Number of States With Initiative Number of Different Measures for Initiative Call Center 28 34 Online System 33 27 Document Imaging 24 8 Kiosks 14 1 Partnering 34 3 Waiver of Face-to- Face Interview 40 12 Shortened Interviews 12 2 Online Expedited Applications 20 2 Total 89 Data were collected between August 2010 and April 2011 from the 45 study States States do not always establish standards for the performance measures they collect. Of the 89 measures collected, States developed PERFORMANCE MEASUREMENT FOR SUPPLEMENTAL NUTRITION ASSISTANCE PROGRAM MODERNIZATION INITIATIVES Office of Research and Analysis December 2012 Nutrition Assistance Program Report Series Office of Research and Analysis Family Nutrition Programs Performance Measurement for Supplemental Nutrition Assistance Program Modernization Initiatives: Individual State Findings Final Report United States Food and Department of Nutrition Agriculture Service December 2012 Non-Discrimination Policy The U.S. Department of Agriculture (USDA) prohibits discrimination in all of its programs and activities on the basis of race, color, national origin, age, disability, and where applicable, sex, marital status, familial status, parental status, religion, sexual orientation, political beliefs, genetic information, reprisal, or because all or part of an individual’s income is derived from any public assistance program. (Not all prohibited bases apply to all programs.) Persons with disabilities who require alternative means for communication of program information (Braille, large print, audiotape, etc.) should contact USDA’s TARGET Center at (202) 720-2600 (voice and TDD). To file a complaint of discrimination, write: USDA, Director, Office of Adjudication, 1400 Independence Ave., SW, Washington, DC 20250-9410 or call (866) 632-9992 (Toll-free Customer Service), (800) 877-8339 (Local or Federal relay), (866) 377-8642 (Relay voice users). USDA is an equal opportunity provider and employer. Performance Measurement for Supplemental Nutrition Assistance Program Modernization Initiatives: Individual State Findings Final Report Authors: Kathleeen Keefe Emily Sama-Miller Laura Castner Maura Bardos Elizabeth Clary Sarah Wissel Larry Vittoriano Submitted by: Submitted to: Mathematica Policy Research Office of Research and Analysis 1100 1st Street, NE, 12th Floor Food and Nutrition Service Washington, DC 20002-4221 3101 Park Center Drive Telephone: (202) 484-9220 Alexandria, VA 22302-1500 Facsimile: (202) 863-1763 Project Director: Project Officer: Laura Castner Rosemarie Downer This study was conducted under Contract number AG-3198-D-08-0105 with the Food and Nutrition Service. This report is available on the Food and Nutrition Service website: http:\/\/www.fns.usda.gov Suggested Citation: U.S. Department of Agriculture, Food and Nutrition Service, Office of Research and Analysis, Performance Measurement for Supplemental Nutrition Assistance Program Modernization Initiatives: Individual State Findings, by Kathleeen Keefe, Emily Sama- Miller, Laura Castner, Maura Bardos, Elizabeth Clary, Sarah Wissel, and Larry Vittoriano, Project Officer: Rosemarie Downer, Alexandria, VA: 2012. United States Food and Department of Nutrition Agriculture Service Family Programs Report December 2012 http:\/\/www.fns.usda.gov\/ iii ACKNOWLEDGMENTS This report was prepared by Kathleen Keefe, Emily Sama-Miller, Laura Castner, Maura Bardos, Elizabeth Clary, Sarah Wissel, and Larry Vittoriano of Mathematica Policy Research under contract to the U.S. Department of Agriculture, Food and Nutrition Service. Additional staff played a critical role in collecting data and conducting interviews during telephone interviews and site visits: Lauren Akers, Kevin Conway, Elizabeth Laird, Alexander Millar, Antoniya Owens, and Amy Wodarek O’Reilly. We thank Scott Cody for thoroughly reviewing each portion of this report. Bea Jones and other database programmers were instrumental in retrieving and reporting the extensive survey data used to produce the appendices. Alfreda Holmes carefully prepared the manuscript and adjusted the formatting on hundreds of pages of appendix tables, and Molly and Jim Cameron provided editorial assistance to optimize the text. We also extend our gratitude to the many individuals outside of Mathematica that contributed to the success of this report. Project Officer Rosemarie Downer provided helpful comments on report drafts and invaluable guidance throughout the duration of the study. State, local, and partner offices that participated in this study shared their staff time and resources during data collection and site visits. This report would not have been possible without their cooperation. v CONTENTS EXECUTIVE SUMMARY …………………………………………………………………………………….. xi I STUDY BACKGROUND, OBJECTIVES, AND ANALYTICAL APPROACH ……………… 1 A. Overview of Data Collection …………………………………………………………… 2 1. Preliminary Assessment and Instrument Development ………………….. 2 2. Respondent Selection and Data Collection ………………………………….. 3 3. Results of Data Collection Efforts ……………………………………………… 5 B. Data Analysis Approach ………………………………………………………………… 6 1. Constructing State Profiles ………………………………………………………. 6 2. Identifying Challenges and Performance Across States …………………. 6 II HOW ARE STATES PERFORMING? ………………………………………………………….. 9 A. Observed Results on Measures Across States ……………………………………. 9 1. Call Centers ………………………………………………………………………… 10 2. Online Systems ……………………………………………………………………. 12 3. Document Imaging ……………………………………………………………….. 14 4. Kiosks ………………………………………………………………………………… 16 5. Partnering …………………………………………………………………………… 17 6. Interview Initiatives ………………………………………………………………. 19 7. Application Tracking …………………………………………………………….. 20 B. Setting Standards and Benchmarks ……………………………………………….. 21 1. Preventing Corrective Action ………………………………………………….. 21 2. Encouraging Staff Productivity ………………………………………………… 22 3. Monitoring and Sustaining Performance …………………………………… 22 4. Responding to Legal Action ……………………………………………………. 23 C. Conclusion ……………………………………………………………………………….. 23 III LIMITATIONS TO USE OF MEASURES AND STANDARDS ……………………………. 25 A. Contextual Limitations ……………………………………………………………….. 26 1. Initiative Scope and Features ………………………………………………….. 26 2. System Limitations ……………………………………………………………….. 31 3. Terminology Differences ……………………………………………………….. 33 4. Varying Measurement Motivations …………………………………………… 37 Contents Mathematica Policy Research vi III (continued) B. Data Limitations ………………………………………………………………………… 38 1. Accuracy …………………………………………………………………………….. 38 2. Validity ……………………………………………………………………………….. 42 3 Reliability ……………………………………………………………………………. 44 C. Conclusion ……………………………………………………………………………….. 46 REFERENCES ………………………………………………………………………………………………… 47 GLOSSARY …………………………………………………………………………………………………… 49 APPENDIX A: STATE PROFILES vii TABLES I.1 Modes of Data Collection for Each Respondent Type ……………………………….. 3 II.1 Average Wait Time and Abandonment for Selected States ………………………. 11 II.2 Measures of Online Systems for Application Tracking for an Illustrative Sample of States ………………………………………………………………. 14 II.3 Quantity of Documents Imaged and Averages for Selected States …………….. 15 II.4 Document Imaging Approach and Time Measures for Selected States ……………………………………………………………………………………………. 16 II.5 Measures of Partnering for an Illustrative Sample of States …………………….. 18 II.6 Reported Performance Measures for All States with Application Tracking Efforts ………………………………………………………………………………. 20 III.1 Variations in Reported Call Center Functions, by Call Center Reach and Number of Functions …………………………………………………………………. 27 III.2 Variations in Reported Online System Functions …………………………………… 29 III.3 Sample of Partners’ Service Variation in Selected States …………………………. 30 III.4 Terminology Variations for Selected Performance Measure Calculations ……………………………………………………………………………………. 34 III.5 Selected State and Local Office Performance Measure Data Entry Methods ………………………………………………………………………………………… 39 III.6 Selected Reports of Methods to Ensure Data Accuracy …………………………… 41 III.7 Validity Concerns for Selected Performance Measures Across Initiatives ……………………………………………………………………………………….. 43 III.8 Selected States and Localities Reporting Standards or Benchmarks on Measures Viewed as Possibly Being Invalid ………………………………………. 44 III.9 Sample of Changes to Performance Measures Affecting Data Reliability ………………………………………………………………………………………. 45 ix FIGURES I.1 Number and Combination of Respondents, by State ………………………………… 5 xi EXECUTIVE SUMMARY The number and breadth of efforts across federal and state agencies to improve technology and streamline service delivery supporting access to the U.S. Department of Agriculture Food and Nutrition Service’s (FNS) Supplemental Nutrition Assistance Program (SNAP) and other public benefits have risen over the last decade. Technology and policy changes often operate together in the states to offer clients new ways to access benefits; such changes also offer staff and partner organizations improved procedures for conducting business. As modernization efforts have multiplied and expanded, so has FNS’s interest in how states implement and monitor their efforts in an environment of increased demand and restricted resources. States, too, have an interest in seeing how well other states are performing. This report is second that stem from the same data collection effort. That effort sought the participation of all states and the District of Columbia, and the reports are based on the responses of the 45 that were able to participate. This report specifically delves into states’ experiences with measuring their modernization initiatives, while the other report focuses more broadly on experiences across the nation. The overall effort focused on a subset of eight modernization initiatives of greatest interest to FNS: call centers, online applications, document imaging, kiosks for application submission, partnering with non-SNAP organizations, online expedited applications, waiver of face-to-face interviews, and shortened interviews. This report (1) describes each state’s performance measures and standards in detail (particularly in state-specific appendices that constitute most of this report), and (2) explains how these measures are implemented and how states use the results. For this report, we sought specific examples of the situations that lead to varied performance measurement approaches by examining the information that states, localities, and SNAP partner organizations provided us in surveys, interviews, and on-site observations. (Throughout, we use performance measures to refer to measures, as well as the aggregate data and data elements that make up the measures (see box).) Diversity in State Measures States track a wide range of performance measures, and how staff use those measures also varies considerably (as the detailed appendices to this report illustrate). No specific performance measurement activities are undertaken uniformly across those states implementing a given modernization initiative. Even performance metrics that track application receipt and processing vary; some states differentiate between applications and recertifications while others do not. However, some similar approaches to measurement exist across states. Except for the shortened interview procedure and kiosks, the states are involved in notable efforts to measure their performance on the modernization initiatives. Most of the common measures across initiatives relate Performance Measurement Terminology 1. Data Element is a field in a database that stores an instance of an activity or characteristic and is the basis of aggregate data and performance measure calculations. 2. Aggregate Data are counts of data elements, such as the number of observations. 3. Performance Measures are calculations using related pieces of aggregate data to assess an activity and typically are displayed as percentages and averages. 4. Performance Standard includes a standard and a benchmark. A standard is the desired outcome of an activity\u2014for example, answering a call within three minutes, while a benchmark is the desired rate of success\u2014for example, answering at least 80 percent of calls within three minutes. Executive Summary Mathematica Policy Research xii to the proportion of an activity completed and the time taken to complete a given task, although the specific variable definitions and calculations states use to measure them tend to vary. Some common measures for each modernization initiative exist across the states interviewed, except for the shortened interview protocol and kiosks. However, within these common measures, we found a variety of measure names and definitions. While the measures could be used to track a state’s performance over time and identify anomalies, the variations across measures and initiatives prevent many specific cross-state comparisons. Challenges of Comparing Measures Due to the variation in the structure of the states’ modernization initiatives and measurement needs, comparing outcomes or even measurement activities across states is difficult and, in some cases, inappropriate. Two aspects commonly affect cross-state performance measure comparisons: contextual limitations and data limitations. Contextual limitations, such as differences in what the initiatives do and who they serve, affect the calculation and comparability of performance measures. Data limitations (specifically those that affect the accuracy, validity, and reliability of data) influence calculations of performance measures, and perhaps their utility and comparability. Meaningful comparisons of performance outcomes across states require an understanding of the contextual and data limitations that each state faces. In addition to cross-state comparison limitations, within-state variations in context and data quality may also be present. Local areas serve different populations with special sets of services, face unique system limitations, and have different reasons for measuring select aspects of performance. In some cases, state and local offices reported different methods of calculating performance measures and setting standards. So, we can expect to see a range in data quality across local offices within a state, and across states. (An exception to this is states where all decisions about data entry systems and training are made at the state level and strictly observed at each local office.) Data Collection Our two-phase research approach began with interviews of a broad group of stakeholders to obtain their perspectives on state performance measurement activities. We refined the number of modernization initiatives to include in the study and developed the data collection instruments, building from an inventory of initiatives and measures that we knew to be active in some core states. The second phase built on the first phase interviews by soliciting survey and interview input from all states and a range of local SNAP offices and partner organizations. 1 The information in this report is the result of an analysis and synthesis of the data collected in that second phase. We first collected data from state SNAP agencies (asking them to respond to a survey, participate in a telephone interview, and submit 12 months of data on their performance measures). Then, with their assistance, we identified and contacted local staff to approach for additional interviews (using the same procedures we used with the state). Finally, using feedback from both state and local SNAP staff, we constructed a list of partner organizations to approach 1 Additional details regarding the methods employed during the study can be found in the companion report: Castner, Laura, Amy Wodarek O’Reilly, Kevin Conway, Maura Bardos, and Emily Sama-Miller. Performance Measures for Supplemental Nutrition Assistance Program Modernization Initiatives. Washington, DC: Mathematica Policy Research, 2012. Executive Summary Mathematica Policy Research xiii about telephone interviews (and request 12 months of performance data, if available). We specifically focused on those doing a wide variety of measurement or participating in unique partnering arrangements (for example, contracting out portions of services, such as document imaging, that SNAP staff would otherwise perform). Staff from some states also participated in site visits from Mathematica staff to explain their measurement activities in greater depth. During the nine-month data collection period, from August 2010 to April 2011, we received feedback from 217 SNAP offices or community organizations representing the 45 states (including the District of Columbia) that were able to participate. In total, we completed a survey, telephone interview, or both with 45 state and 117 local SNAP offices, and completed telephone interviews with 55 partners. In 15 states, respondents included the state and local SNAP offices and partner organizations, and site visits with state staff occurred in 10 states. In 19 states, both the state and local offices responded; in the remaining 11, the state office was the only respondent. 1 I. STUDY BACKGROUND, OBJECTIVES, AND ANALYTICAL APPROACH Modernization of public services, including the U.S. Department of Agriculture’s Food and Nutrition Service (FNS) Supplemental Nutrition Assistance Program (SNAP), involves implementing technological advancements in service delivery and changes in program policy intended to streamline the application and recertification processes. Technology and policy changes (sometimes in the context of reorganized business processes) often operate together in states to offer clients new ways to access benefits, and improve staff and partner organizations’ procedures for conducting business. The number and breadth of such modernization efforts have risen over the last decade in response to several interrelated factors. For example, increased economic need among applicants, increased participation, decreased resources (including staff availability), and monetary incentives related to quality have motivated states to find ways to improve the efficiency of their service delivery without affecting the accuracy of benefit determination. Monetary incentives to increase program participation have encouraged states to improve their customer service and program access. Flexibility in setting policies, paired with technological improvements, has given states many options in exactly how to achieve their goals. As modernization efforts have multiplied and expanded, so has FNS’s interest in how states implement and monitor their efforts, as has the interest of states in how other states are performing in the current environment of increased demand and restricted resources. As a result, FNS contracted with Mathematica Policy Research to conduct an exploratory study that captures how states are measuring the performance of their modernization activities. This report is the second that stems from the same data collection effort. While the first report presents findings integrated across states to examine themes and promising approaches for specific modernization efforts, this report delves more deeply into specific states’ experiences. In examining the information that states, localities, and SNAP partner organizations provided us in surveys, interviews, and on-site observations, we sought specific examples of the situations that lead to cross-state variance in performance and performance measurement. In both reports, we typically use the term performance measures to refer to measures, as well as the aggregate data and data elements that constitute them (see box). The broader project has four research objectives, two of which we address specifically in this report: (1) describe each state’s performance measures and standards in detail; and (2) explain how these measures are implemented, the observed results, and how states use the results. In the remainder of this chapter, we provide an overview of how we collected the data used for the study and discuss our approaches to constructing profiles of each state and examining results across states. Performance Measurement Terminology 1. Data Element is a field in a database that stores an instance of an activity or characteristic and is the basis of aggregate data and performance measure calculations. 2. Aggregate Data are counts of data elements, such as the number of observations. 3. Performance Measures are calculations using related pieces of aggregate data to assess an activity and typically are displayed as percentages and averages. 4. Performance Standard includes a standard and a benchmark. A standard is the desired outcome of an activity\u2014for example, answering a call within three minutes, while a benchmark is the desired rate of success\u2014for example, answering at least 80 percent of calls within three minutes. Background Mathematica Policy Research 2 A. Overview of Data Collection A two-phase approach comprised the core of our data collection effort, with results of the first phase informing the second. First, we interviewed a broad group of stakeholders (including SNAP staff at the national, regional, and state levels, and advocates) to obtain their perspectives on state performance measurement activities, refined the number of modernization initiatives to include in the study, and developed the data collection instruments. We then solicited survey and interview input from all states and a range of local SNAP offices and partner organizations. In this section, we provide a brief description of the specific activities and respondents in both phases, and summarize the response to our data collection effort.2 1. Preliminary Assessment and Instrument Development Before constructing a survey instrument for this exploratory study, it was essential for us to better understand the basics, especially how some states with prominent modernization efforts measure their performance for SNAP. We began with telephone interviews of stakeholders from FNS, national human services organizations, state SNAP agencies, and community-based organizations and business partners. These respondents were selected specifically for their perspectives on state modernization and measurement activities. We asked them about their knowledge of ongoing modernization initiatives and their measurements and standards, as well as their perceptions of performance measurement gaps. Along with these calls, we conducted a targeted document review of sample performance measurement reports provided by our respondents, as well as relevant earlier reports on modernization efforts (Cody et al. 2008; Rowe et al. 2010). Using the findings from the interviews and document analysis, we drafted a preliminary inventory of state performance measurement activities. The inventory helped us assess the extent of measurement activities for all activities that FNS classified under the heading of modernization. Upon consultation with FNS, we then identified a subset of eight initiatives, based on their wide use across states and importance to FNS. Limiting the number of initiatives was important to balance the time required for state and local staff to participate in the data collection with FNS’s need for thoughtful and thorough answers to questions. The eight initiatives that became the focus of the remainder of the project fall into the following three categories: Technology – Call centers, including their computer phone systems, change centers, interviews, and application processing – Online systems, including the online screening tools, online applications, and online accounts – Document imaging – Kiosks used for application access and submission 2 Additional details regarding the methods employed during the study can be found in the companion report: Castner, Laura, Amy Wodarek O’Reilly, Kevin Conway, Maura Bardos, and Emily Sama-Miller. Performance Measures for Supplemental Nutrition Assistance Program Modernization Initiatives. Washington, DC: Mathematica Policy Research, 2012. Background Mathematica Policy Research 3 Partnering with Other Organizations Policy Simplification – The waiver of face-to-face interviews – A shortened interview process – The capacity for clients to complete expedited applications online We developed a survey that included a module for each of the eight initiatives. Due to the emphasis that states place on existing FNS measures and the centrality of application processing to SNAP operations, we developed an additional survey module dedicated to performance measures around the receipt and processing of applications and recertifications. This module expanded upon the FNS-required performance measures regarding application approvals and denials by including measures related to the source of the application or recertification. To support our second phase (data collection), we designed instruments for surveys, phone interviews, and in-person interviews and observations to collect detailed information and provide states with an interactive response approach to describe and demonstrate their use of performance measures. We designed an electronic survey for state and local SNAP agencies, and created a secure file transfer protocol (FTP) site through which they could return performance data after completing the survey. We also developed structured interview protocols for use during follow-up telephone interviews with state and local staff and for targeted interviews (with no preliminary survey) with partner organization staff. A final data collection tool was a semi-structured interview guide to support site visits to a smaller number of states. Mathematica data collectors participated in a full day of training to learn about the purpose of the project in general and the data collection in particular, and review and practice using the data collection instruments. 2. Respondent Selection and Data Collection Once we developed the instruments, we collected data from state SNAP agencies. Then, with their assistance, we identified and contacted local staff and partner organizations to approach for data collection. Using feedback from both state and local SNAP staff, we also constructed a list of partner organizations to approach about telephone interviews. Staff from some states also participated in site visits made by the Mathematica team to explain their state’s measurement activities in greater depth. Table I.1 displays the data collection mode used for each type of respondent, with further details on the data collection efforts described in the remainder of the section. Table I.1. Modes of Data Collection for Each Respondent Type Respondent Type Electronic Survey Telephone Interview Performance Data Site Visit (Selected States) State SNAP Offices X X X X County\/Local SNAP Offices X X X X Partner Organizations X X X Background Mathematica Policy Research 4 All state SNAP agencies received an invitation to participate in the study and, in most states, a subset of their county and local SNAP offices and partner organizations received a subsequent invitation (once we had background information from the state survey). We selected local offices across multiple states in an effort to represent the breadth of local-level measurement activities occurring (see Castner et al. 2011 for details). Responses collected from selected county or other local SNAP offices supplemented the state-level picture by determining the extent to which additional performance measurement activity is being conducted at local levels, as well as giving a front-line perspective on performance data generation and usage. At both levels, the electronic survey provided us with information about the presence and functionality of any focal modernization initiatives in a state as well as application tracking, and solicited basic information (such as definition, importance, frequency of collection) about the performance measures the state collects and the standards and incentives it uses. We then followed up with phone interviews to clarify a small number of survey responses and obtain more details about selected performance measures. SNAP office respondents also received a request to submit 12 months of performance data for each of the measures the office calculates. In addition to our efforts to survey and interview SNAP staff, we collected information on the specific activities and performance measurements of three types of SNAP partner organizations\u2014 national businesses, local businesses, and community-based organizations. To contact these organizations, we selected among the outreach partners and contractors providing services that respondents in SNAP offices discussed during their telephone interviews. We specifically focused on those who were doing a wide variety of measurements or participating in unique partnering arrangements. To collect information from partner organizations, we interviewed them about their partnering activities with the state or local SNAP agencies as well as their performance measurement activities, and also requested that they submit 12 months of performance data for each measure they calculate. Finally, our team undertook site visits to 10 states we identified as leaders in their implementation of modernization initiatives and most likely to have developed a wide range of performance measures. After completing surveys and interviews in these states, we visited them to conduct in-person interviews with state and local staff and partners, and observe their use of performance measures. We sought participation from all 50 states and the District of Columbia, but found that 6 said they did not have time available to respond. We accommodated the schedules of respondents so that the request was as minimally burdensome as possible; this included allowing states to complete an interview only and extending the data collection period. However, despite our best attempts, some state and local office respondents still were unable to participate due to their administrative responsibilities. As data collection progressed, we revised our approach to collecting data from local offices. Our initial plan was to conduct surveys and interviews in 19 states\u2014those we identified as having a relatively high number of the key initiatives that together covered the range of initiative functionalities and regions of the country. In county-administered\/state-supervised states, we requested contact information from the state for 13 local offices; in state-administered states, we requested it for 7. During data collection, however, we often found little variation among the local offices in state-administered states regarding their measurement of modernization initiatives because they primarily followed direction from the state. As a result, to enhance the data collection effort, we modified our approach and requested contact information for local offices in all states, as opposed Background Mathematica Policy Research 5 to just the subset of states. The target numbers changed from 13 to 5 for county-administered states and from 7 to 2 for state-administered states. 3. Results of Data Collection Efforts During the nine-month data collection period from August 2010 to April 2011, we received feedback from 217 offices or organizations representing 45 states, including the District of Columbia (Figure I.1). In 15 states, respondents included the state and local SNAP offices and partner organizations, and in 10 of these states, state staff also participated in site visits. In 19 states, both the state and local offices responded; in the remaining 11, the state office was the only respondent. We completed a survey, telephone interview, or both a survey and telephone interview with 117 offices out of 137 originally contacted. Finally, although we collected contact information for numerous partner organizations, it would not have been fruitful to interview all of them. This was because many partner activities involved conducting SNAP outreach and providing application assistance. After conducting a substantial number of interviews with these types of partners and identifying their typical measurement activities, we chose to continue interviewing organizations with activities going beyond outreach and application assistance. We sent participation requests to 71 partners and interviewed 55. Figure I.1. Number and Combination of Respondents, by State Background Mathematica Policy Research 6 B. Data Analysis Approach Having accumulated a great deal of survey and interview information, as well as some details on performance from reports that respondents sent us, we then undertook two analysis efforts to distill the information into a usable form. First, we created a profile of each state that listed the set of measures and aggregate data, along with any performance standards, benchmarks, or incentives each had in place, and a brief narrative description of the state’s activities. Then, using these profiles, the interview responses, and performance reports, we summarized states’ use of these measures (which we report in Chapter II) and identified several common challenges they faced in performance measurement (as reported in Chapter III). We present the results of these two analysis activities in the remainder of the report. 1. Constructing State Profiles For each of the 45 states responding to the study in any form, we created a specific profile that details that state’s measurement activities. These profiles, arranged alphabetically, form the appendices to this report. Each profile opens with key facts about the state: how SNAP is administered, the number and types of respondents contributing, and the initiatives active in the state. Next, we present a table that lists the performance measures and aggregate data each state or its localities or partners reported to us. The table is arranged by modernization initiative and describes the area in which a given initiative is active, specifies whether a given measure is implemented statewide or locally, and the goals the state or locality is trying to meet through use of the measure. Finally, we provide any standards, benchmarks, or incentives the state or locality applies to the measure. Following the table, we supply a brief narrative giving an overview of the initiatives active in each state to provide context for how the performance measures are implemented. In developing these appendices, we strove to report exactly what respondents told us, within reason. For some measures, additional refinement was necessary. For example, if a respondent reported in the survey that no standard was in place, but during their interview described a standard, we added this information as an update to our database and reported it in the appendix. We renamed some measures across states for the sake of continuity\u2014for example, if a state chose to write in a measure of its own and its definition was similar to a measure we had supplied. In cases for which one or more local offices in a state-administered state reported using a measure from reports provided by the state office, we listed the measure as being implemented statewide even if the state had not indicated this to be the case in its survey. That is, we assumed as long as any local offices were using the data from the state, it is accurate to say that statewide data are available, even if not all local offices choose to use the data in this way. 2. Identifying Challenges and Performance Across States We analyzed how states are using the measures they employ and, to a degree, how they report their performance on these measures. In Chapter II, we discuss our findings on these aspects. For each initiative, we report information on how selected states are implementing and using key measures, drawing attention to specific examples that illustrate why cross-state comparisons may be difficult or inappropriate. In some cases, we report the performance of a small number of states on select measures although, as we describe, it is difficult to do so in a more comprehensive fashion because of the complex contextual factors that limit generalizability. The chapter closes with a discussion of approaches states take to setting benchmarks and standards for themselves on the Background Mathematica Policy Research 7 measures they value most, which may be of interest to states or stakeholders seeking to replicate certain modernization initiatives in new locations. Reviewing respondents’ information within and across states revealed some clear challenges that SNAP offices and their partners face when dealing with performance measurement. In Chapter III, we discuss these limitations more specifically. They include contextual limitations (such as varied functionality of modernization initiatives across states, differences in how states refer to similar concepts), and data limitations (that affect the accuracy, reliability, and validity of data in a way that impairs the generalizability of findings). The data limitations may prove particularly cumbersome for state users of data. While they also affect readers who want to understand these data across states, those readers who are interested in applying this information on measures to their states may be most concerned with the contextual limitations that could complicate their interpretation of the data that respondents provided in the course of this study. Throughout this report, we offer examples of individual states’ experiences with performance measurement. Readers seeking greater detail on a specific state’s situation and measures will find this information in the state profile appendices. 9 II. HOW ARE STATES PERFORMING? As state agencies and FNS seek to identify appropriate performance measures for SNAP, information on how all states are measuring their modernization initiatives and performing with respect to those measures is valuable. For each initiative, a number of performance measures could be useful and informative, but we found that only a few are actually in use\u2014and even fewer are used broadly. In some cases, states are using several similar measures to address the same concept. In this chapter, we explore how states are measuring their modernization initiatives, what standards they are setting for those measures, and how these vary across the states. We provide the details on each state’s modernization initiatives and measures in the appendices. Performance on 175 different measures is being recorded by at least one state or local office (89 are tied specifically to the eight modernization initiatives we examined, and 86 relate to application tracking, accuracy, and approval\/denial rates). With the exception of the quality control measures required by FNS, no performance measures are being collected in all states. Among the more common measures, the method of calculation is not always consistent, so they are not entirely comparable (as we discuss in Chapter III). To further complicate attempts at cross-state comparisons, differences in the nature and maturity of modernization activities, state and local priorities, and business process change efforts shape how the states view and use the measures and what performance levels they find acceptable. We begin this chapter with a review of the results we observed across states, handling each modernization initiative in turn (see the glossary for descriptions). The discussions describe the reach of each initiative, performance on frequently reported metrics for illustrative subgroups of states, sources of variation in those metrics, and the situations in which performance measures are reported on less frequently. In the second section, we focus on setting benchmarks, examining how and why states have set benchmarks and standards, and what some have done to incentivize their staff, despite severe fiscal limitations. A. Observed Results on Measures Across States Information in this chapter is based on what state and county or local SNAP offices reported to us through the survey and interview (or, in some states, site visits). After collecting information about their initiatives, we asked specific questions about the performance measures and benchmarks related to those initiatives. As discussed earlier, this is an exploratory study\u2014we focused on gathering a wide range of information, then verifying the details for a subset of the information. Verifying all of the information we received would have placed too much of a burden on SNAP and partner staff. No performance measures, other than those federally mandated for quality control reviews, are being calculated by all of the states implementing a given initiative. We found several common measures across states, although many were calculated in various ways (as a result, this report often looks at measure families, grouping differently named but similarly calculated measures together). Some states also discussed novel measures they have developed around one or more of their modernization initiatives. This section discusses these common performance measures reported for each modernization initiative in turn, beginning with call centers.3 3 Throughout this report, we refer to performance measures by the names listed in the survey or directly provided by the states. How Are States Performing? Mathematica Policy Research 10 1. Call Centers Among the 28 states4 with call centers, a state or local office in 23 states5 reported that they calculate two key types of measures of clients’ access to telephone services: (1) wait time and (2) volume of calls addressed by the call center. This includes states that operate call centers statewide, regionally, or that operate a combination of regional and statewide call centers, and those for which a performance measure may be reported statewide, by a local or county office, or both. Performance measures addressing how long clients must wait to have their concerns addressed include: Average Hold Time Average Waiting Time (to Speak to Agent) Average Queue Time Average Answer Speed Average Time to De-Queue Measures of the portion of calls addressed by the call center include: Percent Calls Abandoned Percent of Total Calls Handled Percent of Queued Calls Handled No measures from either of these groups were reported for the five other states with call centers (however, the District of Columbia does track the number, but not percent, of calls abandoned). Table II.1 displays the benchmark and performance data for selected states for measures of wait time and rate of calls abandoned, as well as key features of their call center(s). As we describe in Chapter III, the challenge in reviewing data from call centers is that many of them serve clients from multiple programs (such as medical assistance and Temporary Assistance for Needy Families (TANF)) and they do not differentiate callers by program in their databases. Arizona was unique in reporting that their call center metrics are differentiated by public benefit program. Table II.1 shows substantial variation in the standards and benchmarks of selected states and their actual results. Some variation can be explained by how a state’s call center fits into other initiatives and how it is operated. For example, due to increasing SNAP applications and resource constraints, Florida cannot fully staff its call centers without compromising the timeliness and accuracy of processing applications, so it reported prioritizing application processing over allocating additional resources to the call centers. Operational capacity within the call centers also impacts these figures. Arizona’s call center serves the entire state and its staff also process changes that come through its 4 Arizona, California, Connecticut, Delaware, District of Columbia, Florida, Georgia, Idaho, Illinois, Indiana, Iowa, Louisiana, Massachusetts, Minnesota, Missouri, Nebraska, Nevada, New York, North Carolina, Ohio, Pennsylvania, South Carolina, Texas, Utah, Virginia, Washington, West Virginia, and Wisconsin. 5 Arizona, Delaware, Florida, Georgia, Idaho, Illinois, Indiana, Iowa, Louisiana, Minnesota, Nebraska, Nevada, New York, North Carolina, Ohio, Pennsylvania, South Carolina, Texas, Utah, Virginia, Washington, West Virginia, and Wisconsin. How Are States Performing? Mathematica Policy Research 11 Table II.1. Average Wait Time and Abandonment for Selected States State Call Center Features Measure Benchmark or Standard Monthly Average Arizona Statewide Automated response unit Change center Average waiting time (to speak to agent) None 16:43 minutes Percent of total calls handled None Not reported Florida Regional Automated response unit Change center Average answer speed 3 minutes 12:52 minutes a Percent calls abandoned 12 percent 17.1 percent a Georgia Statewide Automated response unit Change center Average answer speed <1 minute 8:57 minutes Percent calls abandoned <9 percent 40.38 percent Indiana Regional Operated by a contractor Automated response unit Cannot transfer to agents Change center Average hold time \u22643 minutes 4:14 minutes Percent calls abandoned \u22645 or 7 percentb 6.53 percent Pennsylvania Multiple areas Automated response unit Change center Average answer speed None 9:23 minutes Percent of queued calls handled 80 percent 12.45 percent (abandoned)c South Carolina Automated response unit Change center Average answer speed 5 minutesd Not reported Percent calls abandoned None Not reported Texas Multiple areas Operated by a contractor Automated response unit Change center Real-time work management system Average answer speed 180 seconds 0:56 minutes Percent calls abandoned 10 percent or lesse 4.0 percent a This represents one circuit (Florida regional SNAP office) only. b Local offices in Indiana use one of several models. Those using a hybrid model have benchmarks of 7 percent. Modern offices have a benchmark of 5 percent. See the Indiana appendix for more information. c Performance data submitted showed [percent] of calls abandoned of calls entered. d The South Carolina respondent reported that this is the industry standard for call center wait times. e Texas offices are fined per percentage point by which their rate of abandonment exceeds 10 percent. online system. Supervisors, however, do not have a tool that allows them to assess immediate staffing and support needs and reassign staff across tasks if needed. This lack of resource could result in longer wait times for callers. Some call centers also are limited by system capacities; calls are abandoned automatically either after some designated wait time (25 minutes in Arizona) or when the queue exceeds a certain number of callers (120 in Pennsylvania). Even for states that use the same measure, such as Percent Calls Abandoned, we cannot compare performance across states, as they do not include or exclude these system-abandoned calls uniformly. States also apply different minimum call lengths in labeling a call abandoned in an effort to exclude misdials. Average wait times and call abandonment rates typically are correlated, as indicated by the monthly averages shown in Table II.1. Of the four states in the table reporting average abandonment rates and wait times by month, the peak rates for both measures occurred during the same months for Georgia, Indiana, and Texas (for example, both measures were highest in May for Georgia and How Are States Performing? Mathematica Policy Research 12 Indiana and in September for Texas). For Pennsylvania, the month with the highest wait times had the second highest monthly abandonment rate. The rates of call abandonment in Pennsylvania and Indiana were higher than in Texas, at median monthly averages of 12.45 percent, 6.53 percent, and 4.0 percent, respectively. This is not surprising, given the differences in average hold time, as people are more likely to abandon calls when faced with longer hold times, such as the median monthly averages of 9:23 (Pennsylvania) and 4:14 (Indiana), than when calls are answered in less than a minute, as in Texas. We observed the same pattern when looking at performance data within states. Texas shows the highest call volume of these three states, averaging more than 300,000 calls per month for the 12-month period for which performance data were submitted. Texas and Indiana, which have the lowest wait times and abandonment rates of those states that submitted performance data, employ contractors to operate their call centers. Texas's answer speeds and abandonment rates may be related to how the state penalizes its call center contractor for rates in excess of the benchmark. (For more information, see Section B on Setting Standards and Benchmarks.) Some additional measures are used in call centers, although primarily for staff and task management. For example, Pennsylvania tracks the Average After-Call Work Time by Staff and Average Calls Handled, a per-hour, per-employee measure. Supervisors monitor call length and intervene when calls become excessively long. Such measures were cited by staff in several states as efforts to increase efficiency. 2. Online Systems States' online systems vary and can include simple websites, in which clients can populate and submit an application, as well as more elaborate sites in which data from a screening tool pre- populates an application that can be signed and submitted electronically. Clients in some states can create online accounts to monitor benefit amounts and submit changes. Online submission for SNAP applications is available in 33 states.6 Of these, 23 states reported that the application source is recorded, either through the eligibility determination systems or another system for tracking application receipts, such as clerical logs.7 Workers in some other states may be able to identify the source for individual applications by searching for markers on the applications themselves, such as Uniform Resource Locators (URL) and printer codes, but could not reasonably determine the number of applications received through the online system using aggregate data collected in an automated way. States in which the eligibility systems can differentiate application sources often can report the percentage of applications received through the online system and the percentage of online applications approved; this enables them to document the initiative's impact on clients' program access and staff workloads without any manual counting. None of the states reported tracking submissions of online applications by demographic subgroup. In theory, online application capability should increase program access, as it provides potential clients with an additional means to learn about their eligibility and submit an application. However, this may depend upon the states' communication efforts. New Jersey used radio advertisements to 6 Arizona, California, Delaware, Florida, Georgia, Illinois, Indiana, Iowa, Kansas, Louisiana, Maine, Maryland, Massachusetts, Michigan, Montana, Nebraska, New Hampshire, New Jersey, New York, North Carolina, Ohio, Oregon, Pennsylvania, Rhode Island, South Carolina, Texas, Utah, Virginia, Washington, West Virginia, and Wisconsin. Alaska has an online screening too, but the online system cannot submit applications. 7 Arizona, some California offices, Delaware, Florida, Georgia, Illinois, one Kansas local office, one Louisiana local office, Maryland, Massachusetts, Michigan, some local offices in Montana, Nebraska, New York, Ohio, Pennsylvania, Rhode Island, South Carolina, Texas, Utah, one local office in Virginia, Washington, and Wisconsin. How Are States Performing? Mathematica Policy Research 13 notify potential clients about its online system and recorded the number of online applications to measure success and determine when it was appropriate to discontinue the advertisements. States also expect that the approval rates for online applications may be lower than for paper applications, due to the demographics of the clientele to which an online application might appeal. For example, Massachusetts uses a benchmark of 50 percent approval for online applications, which is lower than the state's benchmark for approval of all applications (70 percent). Several states reported that they calculate the approval rates specific to applications received through their online systems, but we received minimal performance data for this measure. Without these data, we cannot confirm that online application systems attract applicants less likely to be eligible. Two of the states that calculate the approval rates for online applications\u2014New York and Utah\u2014implemented their online systems within the past five years. The third state using this measure\u2014Massachusetts\u2014implemented its online system in 2004. Two early adopters, Arizona (2002) and Pennsylvania (2001), are not measuring approval rates for applications received online. SNAP offices also reported tracking the quantity of online applications and their approval rates to indicate the workload impact of the initiative. All of the states listed in Table II.2 have screening tools available on their websites so that potential clients can gauge their eligibility before applying for programs. Some states8 reported having online systems without screening tools, but they did not report calculating either of the performance measures highlighted in Table II.2. Screening tools also may save staff time, as those potential clients whom the screening tool shows to be ineligible may be less likely to apply. Because so few states reported calculating the approval rates for applications submitted through their online systems, all of which include an online screening tool, we cannot speak to variation in the approval ratings for online applications based on this tool. Maryland, Pennsylvania, and North Carolina track the use of their online screening tool differently. Maryland reports the number of screenings started and completed (34,783 screenings were started and 27,591 completed); Pennsylvania reports that they use it to determine the percent of potentially eligible screenings resulting in application submissions (results not provided by the state); and North Carolina computes a monthly average (results not provided by the state). A few states track online client account activity, although the type of data they track is not consistent across states. Michigan, Utah, and Florida track the number of online accounts created and the number of changes submitted. Florida also tracks the number of customer log-ins. Michigan's online account measures inform its outreach activities. Specifically, Michigan analyzes online account creation to determine how much time field staff should devote to community outreach and whether they should educate community partners about the online system. Other states may be using their measures similarly but were not as explicit as to their intent. A few states employ unique performance measures for their online systems. Six states\u2014or local offices within those states\u2014reported measuring the volume of clients who received help to apply online.9 Several of these states also track the number of clients who did not receive help to apply online or the number who requested assistance. North Carolina has an online application, but applications must be printed and submitted as hard copies. It uses monthly data about the number of hits on the application site as a proxy for the number of clients applying through the online system. Arizona posts a customer service survey on its website. 8 New Hampshire, Oregon. 9 California, Florida, Maryland, New York, Virginia, and Wisconsin. How Are States Performing? Mathematica Policy Research 14 Table II.2. Measures of Online Systems for Application Tracking for an Illustrative Sample of States State Online System Features Percent of Applications Received Online (associated benchmark) Percent of Online Applications Approved (associated benchmark) Arizona Client accounts Change reporting Integrated screening tool E-signature X (none) Indiana Client accounts Change reporting Integrated screening tool E-signature X (none) Massachusetts Client accounts Screening tool E-signature X (40 percent) X (50 percent) Nebraska Client accounts Change reporting Screening tool E-signature X (none) New York Client accounts Integrated screening tool E-signature X (20 percent) X (none) Pennsylvania Client accounts Change reporting Integrated screening tool E-signature Unsigned submission X (none) Utah Client accounts Online chats Integrated screening tool E-signature Unsigned submission X (none) X (none) 3. Document Imaging States' measurement of document imaging activities focuses on the percentage of documents imaged and timeliness of imaging activities. For most of these offices, case files are available only electronically and must be remotely accessed; thus, quick and comprehensive document imaging is critical to the success of their application processing. Offices in eight states10 reported tracking the Percent of Documents Scanned and an additional twelve11 collect the Number of Documents Scanned. Table II.3 does not include performance data for the sample, as very few states submitted data for this measure. SNAP offices in an additional four12 states do document imaging but did not 10 Delaware, Minnesota, Mississippi, Nebraska, New Hampshire, Ohio, Washington, and Wisconsin. 11 Arizona, California, Florida, Idaho, Indiana, New York, North Carolina, Pennsylvania, Rhode Island, South Carolina, Texas, and Utah. 12 District of Columbia (planned but not yet implemented at time of interview), Iowa, South Dakota, and Virginia. How Are States Performing? Mathematica Policy Research 15 Table II.3. Quantity of Documents Imaged and Averages for Selected States State Data Collected by the State, Regional, or Local Offices? Who Images Documents? Percent of Documents Scanned Delaware State Documents are imaged at the local offices Mississippi Local Counties scan documents, and states collect imaging system data Nebraska Regional Imaging is done by two central offices New York State Documents are sent to a central location, although some local offices have their own imaging systems Ohio Local Counties have their own locally developed systems Washington State Documents are sent to one of four statewide indexing sites Wisconsin Local Documents are imaged by the local offices and attached to the case files in the eligibility system Number of Documents Scanneda Arizona State Documents used to be imaged by a contractor, but now are handled by local office staff in a state-run system California Local Documents are imaged by those local offices that implement the initiative Florida State State has a central document imaging center, although some local offices image their own documents Indiana State Documents are imaged by a contractor and performance metrics are outlined in the contract North Carolina Local Local office staff image documents and attach them to case files Pennsylvania State Imaging done by county offices into a state system, although some smaller offices send documents to larger offices Texas State A contractor at a statewide office does the scanning Utah State Documents are sent to a main processing center, although some local offices can image documents into the online system Wisconsin State Documents are imaged by the local offices and attached to the case files in the eligibility system aStates and local offices that track performance measures are noted in the Percent of Documents Scanned section only. report measuring the volume or percentage of documents imaged. The offices report these metrics for documents across multiple programs and generally cannot differentiate those documents specific to SNAP, although Arizona did report this capability. A few offices also track documents by source (such as by mail or fax). An office's performance data may vary by the maturity of the document imaging initiative, particularly the approach to documents that predate implementation. For example, an office in Delaware recently had initiated document imaging and was working to image older documents at the time of the interview. Until the office finished imaging all older documents into the system, its performance on the percentage of documents imaged or average time to image documents would misrepresent its normal capacity and probably would lag behind offices with similar procedures and resources. In such cases, an apparent performance gap should not be misconstrued as reflecting poorly on the office. Other offices may opt not to image documents predating the initiative or may image those obtained only within a certain period prior to the implementation date. How Are States Performing? Mathematica Policy Research 16 Offices' approaches to document imaging vary, and these approaches impact the time required to image documents. In some cases, scanning occurs after case workers have used the hard copies to make an eligibility determination. In others, a scan first model is used, in which all documents are scanned before being made available for caseworkers. Some states may set different standards for scanning time, depending on when the document is used, if both approaches exist within the same state. For example, Washington set a standard that documents should be imaged within one day if the contents of the document have not been used already to update the client file, but set a standard of three business days if the client file already has been updated or worked. Imaging also may take longer when documents must be packaged and sent to central processing sites or to larger local offices. In the case of the scan first models, offices expect documents to be scanned in a very short time period so the scanning process does not delay the application decision timeline. Table II.4 shows the processing time measures for select offices that reported tracking it. Because each state operates in a unique context and may vary as to which records it counts and how it measures processing time, performance over 12 months cannot be compared, and so is not shown. Table II.4. Document Imaging Approach and Time Measures for Selected States State Scans Pre- or Post-Processing? Performance Measure Indiana Pre Average Processing Time Nebraska Pre Average Processing Time Pennsylvania Pre Percent of Documents Attached in a Timely Manner Texas Post Percent of Documents Imaged Same- Day Washington Pre\/Post Percent of Documents Scanned within Standard of Promptness Regardless of the degree to which a state integrates document imaging into the eligibility determination process, staff in multiple states reported that the initiative has obvious benefits. In Utah, document imaging enables its call center both to address a wider range of client questions and allow staff to work remotely. An Arizona respondent stated that, I honestly believe that imaging has saved staff time. And the proof is that we were able to survive even after a 600-person cut [in staff over two years]. While Missouri does not use document imaging, a respondent explained that it would be much simpler if they had document imaging because it would reduce the likelihood of lost case files and save time otherwise spent digging through case records. The larger counties can never find 100 percent of their cases. Missouri indicated that it is exploring options to fund a document imaging initiative. 4. Kiosks SNAP offices in 14 states reported having computer stations or kiosks in their lobbies or elsewhere to enable clients to access online applications or accounts.13 In general, when discussing their kiosks, respondents reported minimal use by clients. They said that clients interested in applying online or accessing their online accounts do so from their own homes, libraries, or other convenient 13 Arizona, Florida, Indiana, Louisiana, Michigan, Nebraska, New York, North Carolina, Ohio, Pennsylvania, Utah, Virginia, Washington, and Wisconsin. How Are States Performing? Mathematica Policy Research 17 spaces. The states are doing little to measure kiosk use. Washington does track the number of applications submitted from a kiosk. A field in the online application allows clients to indicate that they are using a kiosk station to complete and submit the application. The state cannot identify which kiosk was used, nor verify that the client used a kiosk. 5. Partnering The extent to which SNAP offices work with partners to help potential clients access the system varies. Partners are engaged in a range of activities, under formal and informal agreements, and may or may not be compensated for their efforts. In a few cases, to be discussed in this section, the partners' compensation is dependent to a degree on their performance. Some states report data on their partnering arrangements, although most of these performance measures track the nature and cost of the partnering arrangements, not the quantity or quality of services provided; the percent of partner applications approved was the only reported partnering measure to address the success of the partnering initiative in reaching potential clients. States reported minimal or no performance reporting requirements for their partners. In a few cases, noted in Table II.5, states do track and report on the number of clients assisted by partners, the number of applications submitted through them, and the approval rate for applications submitted by them. Performance measurement activities vary, but the states that do the most reporting on their partnering initiatives are those with contractual partners. For example, among the states included in case study site visits, those that have been engaged longest in partnering for application assistance (Arizona, Pennsylvania, and Washington) are all states where partnering arrangements are governed by Memoranda of Understanding (MOUs). Arizona does not appear in Table II.5, as it did not report measuring its partnering activities. Washington, whose partnering is the oldest of these three, dating to the 1990s, reported only a few measures, and at least one of the partners interviewed for this study indicated that it submits performance data to the state. Pennsylvania calculates performance data about its partners, and at least one partner reported submitting data to the state. Table II.5 does not include information on partnering in California, Delaware, Florida, Louisiana, or Ohio. Partnering in these states varies significantly at the county level, and there are no consistent measures at the state level. Within each of these states, counties are engaged in partnering arrangements\u2014some through formal contracts or MOUs\u2014and collecting and reporting performance data on these partners. Counties in other states, including New York and North Carolina, may be engaged in partnering activities beyond those reported by the state. It is also important to note the unique partnering arrangements in Utah. The state outsources its staff to locations such as hospitals and medical clinics, where they assist clients with applications. These staff report to the state office but are exempt from some typical eligibility worker responsibilities, such as fielding calls through the call center, so they can focus their attention on the How Are States Performing? Mathematica Policy Research 18 Table II.5. Measures of Partnering for an Illustrative Sample of States Measures Percent or Number of Partners Providing Application Assistance Percent of Partners Providing Access to Kiosks Percent of Partner Applications Approved or Denied Aggregate Data on Client Services: Number of Partners Track Performance on Additional Measures?a States with Contractual Partnersb Alabama X Applications received per partner No Kansas X Clients assisted per partner, Applications received per partner No Maryland X Yes Massachusetts X X Clients assisted per partner, Applications received per partner Yes Michigan Applications received per partner Yes Nebraska X X No Rhode Island X No Texas X Clients educated, Clients assisted per partner, Applications received per partner Yes States with Partners by MOU Georgia X Clients assisted per partner, Applications received per partner, Outreach events, Clients served No Nevada X X Clients assisted per partner, Applications received per partner No Pennsylvania X Applications received per partner Yes Washington X Yes States with Informal Partners Illinois X Applications received per partner Yes Wisconsin X Clients assisted per partner, Applications received per partner Yes Note: Indiana relies on contractors to operate its call center, mail service, and document imaging but does not partner for application assistance. Indiana's measurement of its call center is reported in the Call Center section. California, Delaware, Florida, Louisiana, Ohio, and Utah also are omitted from the table and will be discussed later. Other states reported having community partners but did not report any performance measurement of their partnering. a According to partner reports of what state or local SNAP offices require. Project staff did not interview partners in all states, nor all partners in a state. States may be requiring data from partners with whom we did not speak. bStates listed as having contractual partners also may have partners by MOUs or no agreements. States are listed only once, based on the most formal arrangement noted. How Are States Performing? Mathematica Policy Research 19 on-site clients. These arrangements were initiated in 2007 and constitute an alternative approach to the partner staff application assistance we observed in other states. A few states, such as Washington and Pennsylvania, have established performance-based contracts with their partners. Washington reimburses partners for their application assistance, but at least half of all of their submitted applications must be approved, with a further incentive that payment for approved applications exceeds that for denied applications. For each of its partners, Pennsylvania sets standards for the number of applications submitted monthly, based on the partner's size and service area demographics. Beginning in the second quarter of the year, Pennsylvania's quarterly payments are dependent upon the partner showing progress towards its specific annual goals. Partners that assist with client applications have special access to the SNAP eligibility determination system (COMPASS) so they can check the eligibility status of submitted applications. This enables them to calculate the percent of submitted applications that are eligible, which sometimes is required by their MOU. Partners' reporting capacities often exceed state requirements. They may track other data based on requirements from their stakeholders, or to manage their own workload. For example, one Illinois SNAP partner was compensated by funds available through the American Recovery and Reinvestment Act (ARRA) of 2009. Based on the requirements of that funding source, the partner collected information about applicant demographics, particularly poverty status. When the period of ARRA funding ended, so did the requirement to submit this data to the SNAP office. The partner still collects it for its own use, however. 6. Interview Initiatives Of the state and local SNAP office staff interviewed, staff in all but five14 states we spoke to reported using or planning to use a waiver of face-to-face interviews. Maine and the District of Columbia had not yet implemented the policy waiver at the time of the interviews but had it in place during a later phase of the data collection. SNAP is county administered in Ohio and California, so the waiver is not implemented universally. A few states applied for a waiver for recertification interviews only, and continue to conduct in-person interviews for initial applicants. Others use the waiver to allow for client choice; many states default to a telephone interview unless the client requests to meet face to face. Most offices record the interview type in the case notes or use some other method not conducive to automated reporting. Ten offices use systems that enable them to determine how many interviews were conducted over the phone versus in person; half of these states do not regularly report the percentage of interviews by method. Other states do report information about interview methods, although some is pulled from quality control reports on case samples (and so may not be available for the entire caseload). The error rate by interview source is another common measure related to the waiver of face-to-face interviews. Again, much of these data reflect a sample of cases. The counts of telephone interviews that contribute to these statistics include some clients who would have been eligible under the hardship waiver; states did not specifically report on the number of interviews conducted over the phone using the policy waiver. States also reported the percentage of requests for telephone interviews honored and the percentage of interviews conducted via the automated phone system, although the latter was employed 14 Iowa, Mississippi, Missouri, Montana, and South Carolina. How Are States Performing? Mathematica Policy Research 20 only in Illinois. None of the states were able to report on the eligibility determination by interview type or the demographics of those who elected one interview type over another. 7. Application Tracking Not surprisingly, the most common performance measures relate to tracking the progress and outcomes of initial and recertification applications, a task undertaken by all states. Thirty-six states, listed in Table II.6, reported tracking approval and denial rates. Tables II.6 indicates which states report on approval and denial rates separately, or have only a single overall measures for total applications. (States often declined to define their total application measure, so it is not clear whether the total number sums applications and recertifications, applications only but from all sources, or something else.) Table II.6. Reported Performance Measures for All States with Application Tracking Efforts Percent Approved Percent Denied State Total Applications Applications, Recertifications Differentiated Total Applications Applications, Recertifications Differentiated Alabama X X Arizona X X Alaskaa X X California X X Connecticut X Delaware a X X District of Columbia X X Georgia X X Hawaii X Idaho X Indiana X Iowa X X Kansas a X Kentucky a X Louisiana X X Maryland X Massachusetts X X Michigan X X Minnesota X X Missouri X X Mississippi X X Nebraska X Nevada X X New Hampshire X X New Mexico X X New York X North Carolina X Ohio X X Pennsylvania X X Rhode Island X Texas X X Utah X X Virginia X X Washington X X Wisconsin X Wyoming X X aMeasurement reported at the local level only. How Are States Performing? Mathematica Policy Research 21 States measure application processing time in one of two key ways: the percentage of applications processed within a certain timeframe, and the average processing time for those completed within the reporting period. Most states reporting the percentage processed timely use the federal guidelines that applications and recertifications be processed within 30 days. Some states, however, expect applications to be processed more quickly. Massachusetts, for example, set a target that applications be processed within 17 days, and Idaho set a benchmark that 70 percent be processed on the day of receipt. Kansas and Missouri also have benchmarks stating an aim that 95 percent of applications be processed on time. Indiana is in a unique situation. Due to a lawsuit, the state must process 90 percent of applications and recertifications timely, higher than the internal goals of 80 percent and 85 percent, respectively. However, the state operates under a waiver that enables it to extend the processing time to 60 days in cases in which the client has not submitted sufficient documentation during the first 30 days to make an eligibility decision. While most states would close out the application as ineligible due to incomplete information, Indiana can keep it open for an additional 30 days, in the interest of collecting the missing information without requiring the client to resubmit the entire application. A respondent from Indiana noted that the state's timeliness has improved since it increased efforts to track application processing. Several other application processing measures were common or addressed particular concerns: applications by source, average benefit amount, percent outside of business hours, and the percent of duplicate applications. A few states track reasons for application denial, but each office does so differently. More than half of states calculate the average benefit amount.15 A California respondent explained this as a way to assess program costs, particularly the impact and feasibility of proposed policy changes. States that can report the average benefit amount by household characteristics, such as New Mexico, also use this measure to indicate whether certain populations might be underserved. In tracking the number of applications received, some states reported being able to differentiate by source or isolate one or more sources, such as the percentage submitted through their online systems or postal mail. New York and Illinois monitor the percent of online applications received outside of normal business hours. Illinois explained this as an effort to identify and investigate anomalies when comparing across its local SNAP offices. Some local offices in California also track the average number of online applications submitted and the number of repeat or duplicate online applications. Massachusetts occasionally reviews the percentage of duplicate online applications and is working with its vendor to screen out duplicate cases. B. Setting Standards and Benchmarks States did not report extensive use of performance benchmarks and standards, and several states indicated a deliberate choice not to set them. State and local offices reported setting performance benchmarks or standards for several reasons: (1) to help prevent the need for corrective actions, (2) to encourage staff productivity, (3) to monitor and maintain output levels during periods of modernization, and (4) in response to legislative and judicial decisions. 1. Preventing Corrective Action States performing below the national average for timeliness and accuracy risk are being required to create a Corrective Action Plan. In an effort to avoid this, states may carefully monitor their 15 Alabama, Alaska, Arizona, California, Connecticut, District of Columbia, Delaware, Georgia, Hawaii, Iowa, Louisiana, Massachusetts, Michigan, Minnesota, Mississippi, Missouri, Montana, Nevada, New Hampshire, New Mexico, New York, North Carolina, Ohio, Rhode Island, Texas, Utah, Virginia, Washington, and Wisconsin. How Are States Performing? Mathematica Policy Research 22 performance on timeliness or quality measures. In some cases, states established guidelines more stringent than the FNS guidelines, such as the benchmark in Idaho that 70 percent of applications be processed the day of receipt. In an effort to improve performance and discontinue a Corrective Action Plan, Oregon instituted a similar policy that the counties' error rates with telephone interviews must be below the state average or the local office would be required to submit a Corrective Action Plan. Since implementing this program, Oregon's staff reported that the error rate has improved; it is the lowest in 10 years, and is below the national average. Oregon also conducts a quality assurance review to look at agency-caused error rates and cites the counties with best accuracy rates in a monthly newsletter. It issues certificates each month to workers whose reviewed cases all were found to be accurate. 2. Encouraging Staff Productivity Incentives for meeting benchmarks have been used in some states and local offices to push staff to continually improve performance. These incentives also reward high performers and foster friendly competition both within and between offices. As one respondent in Florida explained it, We benchmark against ourselves in terms of what's possible, where we can push the envelope. In some cases, benchmarks and standards are established with input from office staff, as one local office in Minnesota reported. For example, in setting a 15-day processing standard for all applications, one of California's county offices determined that this standard would be challenging but realistic, given its historical data on processing time. Pennsylvania increased its goal for Number of Calls Handled after meeting its previous goal. The Oregon example cited in the section above also illustrates the use of staff rewards for accuracy. In many cases, initiatives meant to reward staff for their performance on certain measures have been discontinued due to financial strain. A number of states provided examples of fiscally neutral awards for strong performance: Governor's Awards are given to call centers when any of the center's standards are met, including a one-minute average answer speed, less than a nine-percent rate of call abandonment, or an average call time between four and six minutes (Georgia) Top-performing staff are offered special assignments or advisory roles (New York, Washington) Executive Directors of the local office(s) with top performance, once adjusted for rudeness complaints and positive feedback, are recognized monthly (Pennsylvania) Despite an inability to offer monetary rewards, at least one respondent noted that the recognition and praise still benefit staff morale and have some positive impacts on office performance. 3. Monitoring and Sustaining Performance As SNAP offices modernize or merge, they use performance data to monitor how well they are integrating new and existing activities and workloads, or even to justify a change in processes. In these cases, standards set to the performance levels prior to the change can help an office ensure that its clients are not adversely affected by this process. After Illinois implemented its online application system, it noticed that applications were still submitted during normal office hours, with low volume outside of those hours. It began to track the percent of online applications that clients submitted outside of normal hours while initiating a campaign to educate clients about the new opportunity to submit applications at their convenience. They reported that the campaign was a success, noting an increase in Percent [of online applications] Completed After Hours (at 27.3 percent during the year How Are States Performing? Mathematica Policy Research 23 from February 2009 to January 2010). Once the performance reached an acceptable level, the state discontinued use of the measure. Two county offices in Wisconsin undergoing changes in their workflow design provided additional examples of how SNAP offices can use performance measures and standards during the transition. The first office, after an initial transition period, plans to establish performance benchmarks slightly higher than the levels met prior to the redesign. The second office had achieved and Average Answer Speed of 30 seconds or less for 95 percent of its calls, so it established this as a benchmark as it expanded to serve callers from several additional counties. Other state and local SNAP offices reported similar uses of performance measurement to maintain or increase client services while implementing new or expanded initiatives. Arizona's decision to operate its document imaging center internally, rather than continue to employ a contractor, relied on establishing and maintaining benchmarks for a performance level comparable to or better than that of the contractor, and at a lower cost. This proof of concept benchmarking in Arizona represents a use of benchmarks and standards to justify and initiate a change in process. 4. Responding to Legal Action Benchmarks and standards also have been set through legislative or judicial action, typically as a result of notably poor performance. Lawsuits resulting from lost mail have led to benchmarks for prompt and comprehensive document imaging. Cases of significantly delayed benefits also have led to stricter standards for average processing time or benchmarks for percentage of applications processed on time. Maryland must process 96 percent of applications in a timely manner, per a court order resulting from the Thompson v. Donald case. Similarly, the Nevada legislature imposed standards for call center wait times in response to complaints of extremely lengthy wait times. Not all states set standards or benchmarks. In many cases, respondents described their decision not to set them as central to maintaining high levels of customer service and program access. For this reason, Michigan chose not to establish any performance standards or benchmarks beyond those required by FNS. More often, however, the states identified certain measures as inappropriate for benchmarking. For example, a respondent in Rhode Island indicated that setting standards for the application source\u2014online, delivery to local office, partner submission, or other methods\u2014would limit client choice. Similarly, Alaska and Delaware, which can calculate the percentage of interviews conducted over the phone, have not set benchmarks in the interest of protecting client choice. For these states, the telephone interviews are seen as an option to increase program access, rather than as replacing the traditional in-person interviews as a means of increasing efficiency. Arizona noted that call center traffic peaks shortly after Notices of Missed Interview are sent, so establishing an appropriate standard for wait time would be difficult, given the variability in call volume. C. Conclusion No performance measurement activities are undertaken uniformly across the states implementing a given modernization initiative. Even performance metrics that track application receipt and processing vary, with some states differentiating between applications and recertifications while others do not. With the exception of the shortened interview procedure and kiosks, states are involved in notable efforts to measure their performance and do so in similar ways. Most of the common measures across initiatives relate to the proportion of an activity completed and the time taken to complete a given task. However, within these common measures, we found variety in measure names How Are States Performing? Mathematica Policy Research 24 and definitions. While the measures could be used to track a state's performance over time and identify anomalies, the variations across measures and initiatives prevent many specific cross-state comparisons. States' reasons for reporting\u2014or not reporting\u2014specific measures also vary. In general, states do not record the characteristics of the clients benefitting from or served by the modernization initiatives, nor do they collect other information to assess how well a given initiative or activity is improving service to a given clientele. States' use of performance benchmarks and standards also vary, with some abstaining entirely. Benchmarks and standards are applied to performance metrics for four reasons: (1) to prevent against low performance that would warrant corrective action, (2) to encourage staff productivity, (3) to monitor performance levels during periods of change and sustain customer service, and (4) in response to judicial or legislative requirements. In the current fiscal climate, states are very limited in giving monetary rewards to high-performing staff or local offices, but they reported other efforts to help encourage staff productivity and reward high achievers who meet benchmarks. 25 III. LIMITATIONS TO USE OF MEASURES AND STANDARDS States face an array of challenges in collecting data and calculating performance measures, many of which affect data quality and hinder efforts to compare existing measures across states. In this chapter, we address two aspects that commonly affect cross-state performance measure comparisons: contextual limitations and data limitations. Contextual limitations, such as differences in what initiatives do and who they serve, affect the calculation and comparability of performance measures. Data limitations (specifically those that affect the accuracy, validity, and reliability of data) influence calculations of performance measures, and perhaps their utility and comparability. An awareness of the challenges inherent in collecting and interpreting performance measures supports a firm understanding of how states use their performance measures and standards. National performance measures and standards may face the same contextual and data limitations and similarly may create challenges to measuring performance of modernization initiatives. While we examine the challenges and limitations of calculating performance measures that states reported to us, our own analysis also encountered the following four main challenges: 1. This is an exploratory study\u2014the first to seek any information on how states are measuring their initiatives. Our focus was on obtaining a wide range of information, which limited our ability to collect some details related to the measures. 2. Information was reported by staff in state and local SNAP offices and partner agencies during our surveys and interviews. We asked respondents what they do measure\u2014not what they can measure\u2014but some respondents nonetheless reported things they could (but do not yet) measure. In some cases, we identified their answers as prospective by clarifying aspects of their responses during interviews. However, to seek that detailed level of clarification on each measure would have required more time from the states than we could reasonably request. 3. We asked local offices how certain measures are calculated and, in instances in which they are collected for the state office, respondents gave us the best information they had available, although they may inadvertently have given us incorrect information. 4. We requested additional information about certain measures and standards from some states and localities, based on educated guesses about the measures most important to them. However, due to our need to limit the details for which we asked, we have little information on performance standards, which is reflected by the unequal treatment they receive in this chapter. We have tried to convey the information obtained in a manner consistent with how it was reported to us and to represent the information as accurately as possible, but some unexplained inconsistencies remain for exploration in more focused studies. Here we address limitations in two parts. In the first section of the chapter, we discuss the contextual limitations that impact states' use of specific performance measures and standards, including (1) the scope of initiatives, (2) system limitations, (3) terminology differences, and (4) varying measurement motivations. The second section addresses the challenges reported in collecting data for these measures and standards, including the data's accuracy, validity, and reliability. Limitations Mathematica Policy Research 26 A. Contextual Limitations State and local offices operate and implement modernization initiatives in different contexts, which influence their choice of performance measures and standards. This also affects which outcomes can be observed and complicates cross-state comparisons. Common contextual limitations include the following: Initiative Scope and Features. Initiatives differ in their size and breadth, especially whether they serve the entire state or just a region, and what services are provided through them. System Limitations. Eligibility and reporting systems are limited in their ability to capture and report measures of interest to each state. Terminology Differences. Variations in measure wording and definitions across states mean that states are measuring different performance aspects. Varying Measurement Motivations. Motivations for using measures and standards also vary across states, with some choosing measures to manage staff workloads and others setting standards to comply with legislative or other mandates. 1. Initiative Scope and Features An initiative's scope (its size, breadth, and functionality) plays a role in determining performance measures and their associated outcomes, and likewise affects the performance standards that guide an initiative. It may be misleading to compare two states' initiatives if one serves an entire state and the other serves just a local area. For instance, states that run statewide call centers may set different standards for Average Number of Calls Handled than do local call centers. Functionality also matters, as call change reporting centers may have different measures and standards than call centers through which customers call in for interviews. In this section, we highlight the variation in scope across states, by initiative, and illustrate the limitations of cross-state performance measure and standard comparisons. Call Centers. Call centers are particularly prone to comparison limitations because of the diversity of their functions and the breadth of their service areas across states. Even when call centers serve the same area, they may provide different amounts and types of services (Table III.1). Seven states reported serving multiple areas through their call center initiatives. For example, Pennsylvania has a statewide call center used in all but one county that we interviewed\u2014that county established its own regional call center. New York reported multiple call centers, some of which are county based and one is a regional call center in New York City. Service area, functionality, and the interview waivers a state may have in place all affect performance outcomes for call centers. For example, results for Average Number of Calls Handled will be different in small, mostly rural areas than in a call center covering an urban, multicounty area. In terms of service features, some call centers share a phone line that rings at multiple desks, while more complex centers have software to queue and assign calls to agents. These features impact how many people can be served, regardless of the number of staff available. Also, outcomes for measures such as Average Answer Speed will differ by call centers' service features. Phone systems' ability to route calls to specific geographic locations (such as in Pennsylvania) or the use of specialized teams to handle certain types of calls (Utah), or to suggest that clients seek services online (Arizona) also influence outcomes. Limitations Mathematica Policy Research 27 Table III.1. Variations in Reported Call Center Functions, by Call Center Reach and Number of Functions State Has Computer Phone System Transfer to Agent Tele-phone Screening Change Center Accept Change Reports Initial Application or Interview Recertification Application or Interview Number of Functions Statewide Missouri Xa 1 Louisiana Xa 1 Georgia X X 2 Delaware X X X 3 District of Columbia X X X 3 Illinois X X X 3 Nevada X X X 3 South Carolina X X X 3 West Virginia X X X X 4 Idaho X X X X 4 Iowa X X X X 4 North Carolina X a X X X X X 5 Pennsylvaniab X X X X X 5 Texas X X X X X X 6 Arizona X X X X X X X 7 Utah X X X X X X X 7 Washington X X X X X X X 7 Wisconsinc X X X X X X X 7 Regional or Local Minnesota X a X 2 Virginia X X 2 Connecticut X X X 3 California X a X X X 4 New York X X X X X 5 Massachusetts X X X X X X 6 Nebraska X a X X X X X 6 Florida X X X X X X X 7 Indiana Xa X X X X X X 7 Ohio X X X X X X X 7 Number 23 21 14 22 17 13 15 Percent of Call Centers with Functionality 82 75 50 79 61 46 54 aIndicates that the phone center is totally automated, with no option to transfer to an agent. bThe Pennsylvania state office operates a call center staffed by state staff and used in all but one county interviewed. cWisconsin has established locally-run call centers in each county office. The state and county offices collect data on the local call centers. Additionally, whether a state or locality has a waiver of face-to-face interviews and a call center that handles these interviews has an impact on the data that feed into performance measures. The time a caller waits to speak to a staff member and the length of the call may differ in states that conduct phone interviews (Arizona, Pennsylvania, Utah, and Washington) than in those that have more limited services available by phone (District of Columbia, Iowa, Nevada, and Virginia). There may be further variation among states according to the type of waiver they have in place. For example, Idaho's and Utah's call centers accept telephone applications or interviews at both initial application and recertification, while Illinois's and West Virginia's accept only recertification applications or interviews by phone. Variations in call center scope also impact performance standards. The reported performance standard for the Average Calls Handled is 40 calls per day per worker in West Virginia, where the call center provides four functions, while for Utah, the standard is within five percent of a team Limitations Mathematica Policy Research 28 average for a call center that provides seven functions. The New York City call center, which places outgoing calls only for interviews, did not report a performance standard for Average Calls Handled, but if it did, it would reflect the number of interviews it desired a worker to complete in a day, regardless of the size of the area served. Average Answer Speed standards may be impacted by scope, but not always. States and localities reported performance standards for Average Answer Speed that ranged from 95 percent of calls answered within 30 seconds in a Wisconsin local office (six service functions) to 5 minutes in South Carolina (three service functions), Washington (seven functions), and a Minnesota local office (two functions). A local Washington office reported having an informal standard of approximately 25 minutes for Average Answer Speed for people who call the office directly, to parallel the wait times experienced in the lobby. (This is separate from the state's call center and its more formal standards, as described above.) Online Systems. Service offerings through online systems also vary and affect both access and state performance and standards. Online systems, with their open access nature, vary little in their service areas, but the services states offer through this initiative do vary (Table III.2), thus affecting the aspects of performance states choose to measure and their outcomes on those measures. For example, states with integrated online screening tools may have a lower count of Number of Applications: Online each month if, before submitting their application, prospective applicants can discover that they are not eligible or will receive only a small benefit amount. The ability to submit applications online and to print applications and mail them could affect states' performance on other measures of application submission, such as Percent of Applications: Paper Submission to Local Office, and Percent of Applications: Mailed, and could increase the number of duplicate applications if people choose to submit both ways. Interpretations of online systems' outcomes across states thus should consider any variations in available services. Although few states reported setting performance standards for their online system measures, these variations in service provision also could limit the comparability of standards. Document Imaging. The services provided by a document imaging initiative, including whether some tasks are automated, affects states' choices about performance measures and standards. How SNAP offices perform document imaging tasks once the initiatives are in place also affect these measures and standards. Barcodes facilitate automated linking of images to case files in some offices but not in others, and this may affect measures of document imaging volume and speed. Where barcodes are absent or manual indexing otherwise is needed, systems setup and worker cooperation may impact performance. For example, in one Nebraska office, so many index categories exist that workers reported using just those few codes easiest to remember so they can work more quickly than if they had to look up a specific code from a list of more than 300. Measures such as Average Processing Time, Percent of Documents Imaged Same-Day, or Percent of Documents Imaged Next Business Day (the latter two measures are used in Texas) rely on dedicated staff being responsible for imaging documents, although the staff members that are responsible for these tasks vary by office and can impact the measures. Document imaging measures also are impacted by other initiatives. States that employ online systems allowing users to scan and upload their own documents online could see a decrease in the Percent of Documents Submitted through mail or fax. If this system is used heavily, less scanning would be required but indexing and attaching documents to cases still would be necessary. 29 Table III.2. Variations in Reported Online System Functions State Screener Integrated Online Screening Tool and Application Application Download and Paper Submission Online Unsigned Sub-mission Online Signed Sub- mission Application Integrated with SNAP Eligibility System Check Account History or Benefit Status Report Changes Partner Access Alaska Xa Xa Xa X Arizona X X Xa Xa X X X X X California Xa X Xa X Xa Xa X Xa Delaware X X X X X X Florida X Xa Xa Xa X X X X X Georgia X X X X X X Illinois X X X X X X Indiana X X X X X X Iowa X X X X Kansas X X X X Xa X Louisiana Xa Xa X X Xa Xa Maine X X X X X Maryland X X X X X X X X Massachusetts X Xa Xa Xa X X X Xa X Michigan X X X Xa X X X X X Missouri X Xa Xa Xa X X Montana Xa Xa Xa Xa Xa Xa Nebraska X Xa X X X X New Hampshire X X New Jersey X X X X X New York X X X Xa X X X Xa X North Carolina X Xa X Ohio Xa Xa Xa Xa Xa Xa Xa Xa Xa Oregon Xa Xa Xa Pennsylvania X X X X X Xa X X X Rhode Island X X X X South Carolina X X Texas X X X X Xa Xa Utah X X X X X X X Xa Xa Virginia X X X X X X X X Washington X Xa X Xa X X X X West Virginia X X X X X X Wisconsin X X X X X X Number 29 25 30 17 30 25 20 17 15 Percent of States with Online Systems 88 76 91 52 91 76 61 52 45 aReported at the local level only. Limitations Mathematica Policy Research 30 Kiosks. The comparability of performance measures for kiosks is particularly vulnerable to limitations based on the prevalence of these tools, as well as the availability of other initiatives designed to increase access to applications. However, of the 14 states that reported providing kiosks, most did not report measuring their performance. Partnering. Partnering initiatives vary considerably by location, even within a state (for example, partners may be clustered in urban areas rather than being spread equally around a state). This leaves the initiative's measures susceptible to comparability limitations. For example, Feeding Illinois (a core outreach partner that subcontracts with other partners, each of which reports on its performance) reported that the performance standards of its subcontracted partners vary according to their location (urban or rural) and clientele (especially ethnicity). In another instance, a Delaware partner operates a statewide prisoner re-entry program, which obviously targets a limited population of the state. Each state's partnering performance measures reflect the unique contexts in which their partners operate. The types and amounts of services partners provide also vary considerably (Table III.3), which can result in misleading performance outcomes. Partners reported providing services ranging from application assistance to hosting online systems, scanning and submitting documents, and making kiosks available. Although many partners reported some level of application assistance and outreach, the type and amount varied, ranging from helping people fill out and submit paper applications to collecting and copying verification documentation and providing ongoing case management support. For example, Utah's YWCA partner stations an eligibility worker onsite to provide application assistance to the YWCA clientele. While some partners reported providing computers for submitting online applications, few (Utah's Crossroads Urban Center, for example) monitor them to see how frequently the computers are used. The measure Percent of Partners Providing Access to Terminals\/Kiosks may reflect the availability of these machines, but not usage or accessibility. Table III.3. Sample of Partners' Service Variation in Selected States State Application Assistance SNAP- Related Outreach Call Center Online System Document Imaging Kiosks Arizona X X X X California X X X Delaware X X Florida X Illinois X X X X Indiana X X Maryland X X X Massachusetts X X X X Michigan X X Nebraska X X New York X Pennsylvania X X X X X Texas X X X Utah X X X X Washington X X X X Wisconsin X X Number 14 11 8 8 3 2 Note: Not all states have partnering arrangements, and we did not speak with every partner in each state. This table includes services that partners we interviewed reported providing. Thus, this table may under-represent the services available through partners in these states. (Alabama, Alaska, Connecticut, Georgia, Kansas, Kentucky, Maine, Minnesota, Missouri, Nevada, New Hampshire, North Carolina, Ohio, Oregon, South Carolina, Rhode Island, and Wyoming also had partnering arrangements, although we did not speak with partners in those states.) Limitations Mathematica Policy Research 31 Interview Initiatives. The cross-state comparability of performance measures for the Waiver of Face-to-Face Interview initiative is limited both by area served and services provided through that initiative. In states with this waiver, it is possible that it is implemented locally, and not throughout the state. A comparison of performance measures that track the percentage of interviews conducted face to face versus by telephone will yield different results in states where this initiative is implemented statewide versus in only a few localities. Even comparing these measures at the local level raises concerns: (1) rural areas with limited transportation may have higher rates of hardship interviews conducted by telephone, and (2) urban areas with multiple easily accessible offices may have lower rates of telephone interviews but serve more people overall. Finally, as mentioned in the call center section above, variation exists in the waivers themselves, with allowances for interviews at application, recertification, or both. Such differences may affect interview initiative outcomes just as they do call center outcomes. Very few performance standards were reported for this waiver, but any attempt to compare states' performance standards should be cautious due to the underlying regional and service variations. Application Tracking. States' ability to gauge their performance on measures of application tracking is impacted by the presence and variation of all of the initiatives discussed so far. Multiple methods for receiving applications through different initiatives affect how those applications are tracked, what states choose to measure, and where they set their performance standards. The variations in the presence of initiatives and their scope in a state limits cross-state comparisons because of the complexities involved. Controlling for regional variation and service provision for each measure would be necessary to complete a fair assessment of state performance. 2. System Limitations The efficacy of comparing performance measures across states also is impacted by the eligibility and reporting systems the states use. Respondents reported three main limitations to these systems: reporting capacity, system reliability, and capability. Each affects states differently and may be a problem for some states more than others. In this section, we discuss each of these limitations individually, rather than by modernization initiative, because each has potential impacts across initiatives and their performance measures. Reporting Capacity. Performance measures' utility may be limited by a state's reporting capacity. Some states' systems do not have the capacity to report aggregate data or generate reports that can be manipulated by users, while others only produce reports months after data were collected. These systems produce reports on performance measures that are not useful for states and local offices. Also, system-generated reports may not be formatted in a way that answers questions interesting to their readers or may not be dynamic and capable of being manipulated so users can find those answers themselves. Examples of systems that lack useful reporting capacity for states include the following: States do not all track performance measures by program. Indiana tracks the performance measure Average Waiting Time (to Speak to Agent) for all calls that come into its call center, but cannot track performance on SNAP cases versus calls for other benefits. The same is true for its online system and application tracking measures. Some measures available at a point in time are not aggregated to show trends or longer-term performance. Delaware's call center reports are not aggregated at the most useful level for their users. The system calculates Percent of Total Calls Handled based on a point-in-time data collection, but when the staff want to see these data more Limitations Mathematica Policy Research 32 broadly (for instance, for performance reviews), they must aggregate the statistic they need manually. Data may capture only one piece of a broader measure. Alaska SNAP officials reported that they can track only those clients who were approved for SNAP benefits through their food bank partners, not those that applied and were rejected. One county in Wisconsin, which is served by a regional call center, is unable to extract performance measures for itself only, limiting the county's use of any performance measures for that initiative. A local Wyoming office reported receiving aggregate data from the state that did not allow it to locate the data needed to calculate performance measures for its office. In these situations, reports may not address the questions of interest to users or may be too cumbersome to be useful. Data reporting lags also can limit effective responsiveness to performance outcomes. Minnesota's eligibility system requires manual entry before data are transferred to a warehouse. Not all data that Minnesota collects can be entered into this system because the state does not have the resources to add fields to the data warehouse. Its system can produce point-in-time reports of all pending cases by worker and the number of days pending, but not whether the application is new, repeat, or recertification; denial reason; or program. Respondents in Minnesota reported that the system also has a lag of a few months for these data, but did not explain further. System Reliability. Unreliable systems encountering periodic outages limit the usefulness of the data provided through those systems. Arizona staff described an eligibility system that stops working so frequently that caseworkers must keep back-up paper files. A Maryland partner reported a month-long server problem that kept it from logging into the eligibility system as a community partner, limiting the state's ability to track the partner's activities (the partner could assist people with applying but the activities were not credited to the partner). In one New York City office, a machine broke, forcing staff to code documents manually as they were imaged. Even when states and local areas can create manual workarounds for their unreliable systems, such changes introduce the possibility of errors in the data systems. Comparing states' performance on measures relying on data systems that fail regularly could be misleading if all the data are not captured, or are captured inaccurately. System Capacity. States all may report they are implementing an initiative, but varying system abilities can affect what states either can or choose to measure as well as how those outcomes appear, which limits performance measure comparability across states. These types of limitations may apply across all modernization initiatives, including these three examples: The West Virginia computer phone system only records information regarding calls received, requiring caseworkers to self-report outbound calls. Staff reported that this was less accurate than automated computer phone system reports. Pennsylvania's call center capability is limited because the queue capacity cuts off when 120 calls are waiting and does not allow callers into the queue after 4:15 p.m. so as to allow staff to answer all calls remaining in the queue before their workday ends. In Washington, paper applications received from partners around the state come with a barcode attached at the bottom of each so that partners' cases can be tracked and they can be reimbursed for their efforts. However, state scanners are programmed not to scan the bottom margin of the page (where the barcodes are applied), thus forcing Limitations Mathematica Policy Research 33 partners to track their application assistance manually and reconcile each missing case with the state in a process that takes two rounds of review each month to resolve. Given this lengthy reconciliation process, comparing Washington's partner performance in a recent month against that of other states could give the impression that Washington's partners submitted fewer applications than they actually did. 3. Terminology Differences An absence of standardized terminology limits cross-state performance comparisons. For example, multiple states may count the Percent of Documents Scanned, but how they define imaging and what tasks are involved in the imaging process can vary, from making an image of a document to the act of scanning a document, assigning a code, and attaching the document to the proper case. This lack of standardization reflects the differences in service provision and data system functionality, described earlier, and could hinder our ability to compare outcomes across states. Table III.4 shows the variation in how states report defining and calculating selected performance measures. While these differences often are subtle, recognizing them is crucial to accurate cross-state comparisons. Two issues emerge when examining limitations that stem from terminology differences. One is that states measure performance over different time periods (for example 12 months versus one day), which could result in different performance outcomes. The second is that some states provided general descriptions of their measures rather than precise definitions of how to calculate them. In the same way that terminology variations limit cross-state performance comparisons, they also limit cross-state comparisons of performance standards tied to the measures. This does not imply that one method of calculation is better than another, just that the differences can impact the observed values and interpretation of reported performance measures. Call Centers. Call centers vary in several ways in how they define and calculate measures, making it difficult to compare one state's performance against another without recognizing these differences. Where states seem to have similar numerators for their performance measures, differences in denominators and numbers excluded from the calculation reflect varying activities. Georgia and Pennsylvania calculate Average Answer Speed using a common numerator but use vastly different denominators and exclusions. Pennsylvania excludes abandoned calls, but these are included in the same calculation in Georgia, perhaps lowering its Average Answer Speed. Because we were able to collect only limited information about many measures, it sometimes is difficult to tell how certain data are treated from state to state. For example, we cannot always identify how abandoned calls are treated. If the state's queue times are very long, or if it provides a similar service through another avenue, its abandoned call rate could be high, thus potentially skewing the state's Average Call Duration if it includes abandoned calls either in the numerator or denominator, or excludes these calls from the calculation entirely. Broad definitions of certain measures make them particularly difficult to compare against each other. In Arizona and Georgia, for example, Average Call Duration reflects the amount of time a caseworker spends on the telephone with a person, in Indiana it includes the after-call work time spent by caseworkers, and in Pennsylvania it is a factor of all of the time staff were logged into the system and not on breaks. Thus, the results of this calculation could appear higher in states that include more activities in the definition, as in Indiana and Pennsylvania. Limitations Mathematica Policy Research 34 Table III.4. Terminology Variations for Selected Performance Measure Calculations Selected Performance Measure Calculations State Description Exclusions Call Centers Average Answer Speed Georgia Total time from start of queue to answer time divided by total number of calls. Time starts once customer is in queue at the start of greeting, and time ends when the customer comes off the queue, either by the agent picking up the phone or the customer hanging up. Noneb Pennsylvania Total accumulated time to answer for all entered calls divided by total number of calls answered by an agent or calls receiving some treatment (such as cancel, route, or transfer to a voice mailbox). Abandoned calls and queue full excluded Washington Total time calls waited in queue and ringing before an agent answered divided by all calls that get routed to the queue. Excludes abandoned and other dropped calls and direct agent calls Wisconsina Total speed of answer divided by total answered calls. Noneb Average Call Duration Arizona Total number of minutes during which the client speaks to an agent divided by total number of calls answered. Noneb Georgia Total time on the phone with a client post queue divided by total number of answered calls. Noneb Indiana Total talk time plus total after-call work time divided by total calls handled. Noneb Pennsylvania Total login time for day and talk time for the day divided by total number of calls. Time when people are not receiving calls (such as breaks) are excluded Utah Total number of minutes and seconds spent from time of answer to disconnect for all calls in the universe divided by total number of answered calls. Noneb Average After-Call Work Time by Staff Pennsylvania Total system-calculated after-call work time divided by total number of calls handled. Non-handled calls (queue full and abandoned) excluded South Carolina Total amount of time agent is in after call status (after caller hangs up) divided by total number of calls answered. Noneb Utah Total time the phone is set to \"After-Call Work\" mode divided by total number of calls resulting in after-call work. Noneb Average Calls Handled Florida Total number of calls answered divided by total number of workdays. Noneb Limitations Mathematica Policy Research Table III.4 (continued) 35 Selected Performance Measure Calculations State Description Exclusions Ohio a Total number of calls answered by staff in customer service center divided by total number of staff handling calls that day. One local office: Includes all calls: incoming, voicemail, outgoing calls, and when a worker accesses personal voicemail. One local office: Every call that comes in is recorded, including hang-ups\/voicemails. One local office: Wrong numbers and hang ups are screened out by the receptionist before they can reach the customer service center Pennsylvania Total number of calls a worker answered within an hour divided by total amount of time a worker is logged in within an hour. Dropped calls and outbound calls (if the call is dropped and the agent has to call the client back) excluded Utah Total number of calls answered from a call center queue divided by total number of agents answering calls from a call center queue. Dropped\/abandoned calls are excluded Online Systems Percent of Applications Received Massachu-setts Total number of online applications that generated an application in BEACON (the online application system) divided by total number of applications, including all sources (walk-in, mail-in, etc.). Noneb Nebraska Total number of program applications submitted online divided by total number of program applications submitted. System change in July 2010 allows for better count of paper submissions Utah Total number of applications submitted online divided by total number of applications submitted by any means. Applications for programs that closed less than 30 days prior (churners) are excluded from the denominator Percent of Applicants Who Received Help to Apply Online Florida Total number of applicants who answered survey that they needed help divided by total number of applicants who completed the survey. Applicants who chose not to take the satisfaction survey are not included New York Total facilitated e-applications divided by total SNAP applications received. Noneb Virginia Total number who reported receiving help online divided by total number submitting online applications. The calculation excludes all who do not respond to the item Limitations Mathematica Policy Research Table III.4 (continued) 36 Selected Performance Measure Calculations State Description Exclusions Document Imaging Percent of Documents Received by Fax Indiana Total number of documents shown as coming from a fax divided by total number of documents received. Noneb Utah Total number of documents received to a centralized fax line for imaging divided by total number of documents received and imaged. Noneb Interview Initiatives Percent of Interviews Conducted Face-to-Face Alaska Number of face-to-face interviews conducted in the month for approved cases divided by total number of interviews conducted for approved cases. Interviews that result in denials are not part of the performance measure Massachu-setts Total number of face-to-face interviews conducted in a period of time divided by total number of certifications\/recertifications conducted during same period. Noneb Virginia Total number of face-to-face interviews divided by total number of interviews completed. Excludes those with no interview Percent of Interviews Conducted by Telephone California Total number of interviews conducted by phone divided by total number of redetermination interviews. Noneb Michigan Total number of SNAP applications where a worker conducted a telephone interview divided by total number of all SNAP interviews. Interviews for non-SNAP applications are excluded New Mexico Total number of telephone interviews divided by the total number of interviews. Noneb West Virginia Number of cases eligible for telephone interview divided by the number of cases due for redetermination. Excludes any cases not due for redetermination. aDescription was provided by a local office in that state. bEither there are no elements excluded from the calculation, or states did not report any exclusions. Online Systems. Online systems are susceptible to the same cross-state comparison limitations due to their lack of uniform definitions and calculations. Massachusetts calculates its measure Percent of Applications Received by dividing all applications received online by all applications received through any means, but Utah excludes churners (applications from a household whose assistance case was closed less than 30 days before) from the denominator. Document Imaging. Comparing states across document imaging performance also is problematic because of the different ways states define and calculate the data for these measures. Limitations Mathematica Policy Research 37 Indiana and Utah calculate Average Number of Documents Received by Fax in similar ways, but Indiana's denominator includes only documents received, while Utah's includes documents received and imaged. If different types of documents are imaged in each office, this measure may inflate performance differences. Partnering. While states reported calculating fewer performance measures for partnering than some other initiatives, the differences in terminology around those measures still impact measure calculation. For example, the Percent of Partners Providing Application Assistance is calculated differently in three states. Florida calculates the number of partners that provide this type of assistance and divides by the total number of partners. Massachusetts calculates only the number of formal partners (excluding the informal partners that may provide the same assistance) that have submitted active applications during a particular timeframe. Rhode Island calculates the percentage of agencies in the state that provide application assistance through its web application. Across states, the types of agencies, the types of application assistance the agencies provide, and the method of providing that assistance (web applications, paper, etc.) are all variables that introduce limitations to cross-site comparisons of this measure. Interview Initiatives. Three states reported different ways of defining and calculating the Percent of Interviews Conducted Face to Face. Alaska's numerator for this measure is the number of face-to-face interviews for approved cases, excluding all denied cases. Massachusetts's numerator is the number of face-to-face interviews at initial application and recertification during a specific timeframe. Finally, Virginia looks at the number of face-to-face interviews and divides by all cases, including not just those that received telephone interviews, but also those not interviewed at all. 4. Varying Measurement Motivations States may calculate and use the same performance measures and standards, but their motivations for doing so may vary. A performance measure designed to meet a legislative mandate in one state could be calculated differently from one used primarily for staffing purposes, and any performance standards associated with those measures likely will differ as well. This is not problematic in and of itself, but it can complicate comparisons of multiple states because these varying motivations affect the priority that measures receive within an office. As one respondent said, Where you measure is where you get results. Broadly speaking, states have three main rationales for setting and calculating performance measures and standards across any initiative: (1) managing staff, (2) gauging program access, and (3) adhering to requirements dictated by lawsuits or legislative action. Performance measures are valuable tools for managing staff and workloads. Numerous local offices reported calculating and using performance measures across initiatives for staffing purposes. In call centers, measures such as the Average Call Duration, Percent of Total Calls Handled, and Average Answer Speed help staff to monitor workflow and make adjustments as needed. Offices in Nebraska, Washington, and Wisconsin use Average Call Duration for workload management purposes. In Delaware, this measure is used to identify program and technical knowledge deficits among staff. In Wisconsin, the Percent of Total Calls Handled by the Spanish-speaking eligibility worker identifies which times of day and days of week are busiest. Texas and Florida also reported using online systems measures to monitor performance because they had the potential to be more efficient than processing face-to-face or telephone applications. Wisconsin reported using several measures of document imaging for staffing purposes, including Average Processing Time and Percent of Documents Scanned. The challenge for most offices is managing staff workloads with the resources available, and the performance measures on which they focus are those from which Limitations Mathematica Policy Research 38 they tend to see useful results. This may mean that other performance outcomes suffer if they are not considered valuable. Several states and local offices reported using measures across modernization initiatives to estimate clients' accessibility to programs. A Washington office uses Average Answer Speed in its call centers to ensure that these clients do not have longer waits for service than those who walk into its centers. Florida, Pennsylvania, Washington, and Utah use the online system measures Average Number [of online applications] Submitted per Month and Number of Changes Submitted per Month to assess whether clients are using online application systems to apply for and manage their benefits. Performance on these measures is client driven, and it may be more accurate to compare states' outreach and promotion strategies for directing clients to one service or another. Performance measures and standards also can be inspired by lawsuits and legislative mandates. Indiana and Maryland track their performance on certain measures for these reasons, setting performance standards around Average Processing Time of Applications and Recertifications because of a lawsuit and court order, respectively. States involved with lawsuits and court orders may face penalties for failing to meet certain outcomes, and thus have strong incentives to ensure compliance. B. Data Limitations While contextual limitations impact cross-state analysis, the data used in these calculations also pose limits to performance evaluations and can jeopardize interpretations of individual performance measures. Performance measures are subject to three main types of data limitations: (1) accuracy, meaning whether data are entered and measures are calculated correctly; (2) validity, or whether states actually measure what they intend to measure; and (3) reliability, defined as stability of performance measures and their calculations over time. Limitations to the accuracy, validity, or reliability of measures hinder cross-state comparability and performance analyses. To gauge these limitations, we analyzed survey responses that identified limitations and asked staff for their impressions of measure accuracy. 1. Accuracy When data are used to calculate performance measures, how they are entered impacts the quality of the measure. Procedures to ensure that data are entered correctly also influence the dependability of measures. Table III.5 describes whether states and local offices entered data manually, automatically, or by both methods across an illustrative sample of performance measures. Data entered automatically are likely subject to fewer errors and subjectivity (assuming the systems automatically generating those data were programmed correctly). Overall, state and local offices often reported using automatic data entry performance measures, leading to a higher degree of accuracy. However, this tended to be true more often at the state than at the local level, where staff typically have less control over generating reports. Also, little consistency was reported between states and local offices regarding which measures use data that are entered manually versus automatically, even among initiatives with performance measures that lend themselves to automated calculations (such as call centers and Limitations Mathematica Policy Research 39 Table III.5. Selected State and Local Office Performance Measure Data Entry Methods Data Entry Method Measure Manual Automatic Both Call Center Average After-Call Work Time OH a DE, SC Average Answer Speed GA, IN, MN, a SC, UT, WA, WI, a WY Average Call Duration OH, a WI a FL, GA, NC, WI a Average Calls Handled OH, a WI a DE, NY, a SC, WI Average Hold Time IL, IN, OH, a WY Average Waiting Time WI a FL, OH, a SC, WI a Percentage of Calls Abandoned GA, IN, MN, a NE, NC, OH, a UT Percentage of Total Calls Handled OH a DE, VA, a WI a Online Systems Average Number of Accounts Created FL, MI, UT Average Number Started per Month MI, a RI, VA Average Number Submitted per Month CA, a OH a CA, a MA, PA WA a Percent of Applications Received GA a MA, UT, WA CA, a WA a Document Imaging Average Processing Time WI a MN, a NY a DE, a IN, WA, WI a Percent of Documents Received by Fax MN, a OH, a UT Percent of Documents Scanned OH a DE, a MN, a OH, a WI a WA Partnering Percent of Partners Providing Access to Terminals\/Kiosks CA a MI FL a Interview Initiatives Percent of Interviews Conducted by Telephone AK, CA, a OH, a OR, WI a CA,a DE, IL, MA WI a Percent of Interviews Conducted Face-to- Face AK, CA, a OH, a VA NM Percent that are Missed and have Notices Mailed OH, a OR CA, a VA, WI a Application Tracking Average Benefit Amount CA, MN, OH a CA, a DC, UT, WA a MT, NV, NM, NC Average Processing Time DE, a MN a Average Processing Time of Application Decision DC, KS, WA a AK, CA, a DC, RI, WA, a WI a MD Percent of Applications Approved GA, a LA, a MA, a VA a CA, a GA, a LA, a MN,a NYb AZ, CT, a GA,a MA, PA, UT, WA a aLocal office reported. bLocal and state offices reported. online systems, which tend to have data systems that automatically capture a great deal of aggregate data). These factors influenced whether staff thought their data were entered correctly. To help us identify the limitations of these measures, we asked staff to report their level of satisfaction with the accuracy of each measure. In general, respondents reported high levels of satisfaction with the accuracy of their performance measures. Out of 114 performance measures in the survey, respondents reported dissatisfaction with only 31. Some of this satisfaction may be explained by the steps in place to ensure accuracy. Part of ensuring data accuracy is instituting procedures to guarantee that data are entered correctly. In general, states and local offices reported Limitations Mathematica Policy Research 40 three types of procedures to ensure data accuracy: (1) staff training, (2) monitoring and quality assurance, and (3) built-in system prompts (Table III.6). Numerous offices reported training staff as their mechanism for ensuring data accuracy. For example, an office in Wisconsin said that it instituted a training session for call center staff on Average After-Call Work Time by Staff, because staff were not entering all data into the database that calculates this measure, resulting in inaccurate results. Several offices reported using training in conjunction with monitoring and quality control procedures to ensure accuracy. An Ohio office reported concerns that workers were taking calls but not recording them in the phone log, so it held a staff training, and taught supervisors to monitor this activity more closely. In New York, partners of one office are intensely trained on correct data entry procedures and a team of staff conduct site visits to ensure compliance. One California office created program documents to instruct staff on data entry, and its case reviewers check worker input against these regulations. If they find that the processes are not being followed, they conduct special trainings or send announcements reminding staff to follow the formal procedures. A few offices reported creating built-in system prompts for workers to ensure regular and accurate data entry. Massachusetts trains workers to use on-screen drop-down menus to indicate whether interviews are conducted over the telephone or face to face (rather than their having to remember to enter this information in a uniform way). A Wisconsin office has a similar check box but reported no special procedures or training for workers to use this feature. Pennsylvania's eligibility system generates a report on Percent of Total Applications Approved; as long as cases are entered into the eligibility system, no other human intervention is required. Rhode Island created a mandatory field in its screener to indicate which cases are eligible for expedited benefits. A Wisconsin office has a similar field in its online application system that automatically tells staff if a case is eligible for expedited benefits. New Mexico uses a web-based client tracking system to prevent multiple workers from accessing or working on more than one case at a time. All of these different system checks contribute to data accuracy but may not impact every initiative and performance measure uniformly. Call Centers. Call center measures received the highest dissatisfaction ratings among staff we surveyed, spread across nine different performance measures. Utah, Washington, and Wisconsin reported some dissatisfaction with Average After-Call Work Time because they could not determine call outcomes (which affect the amount of after-call work time), and because the measure is related directly to worker action, which is not always carried out under standardized procedure and may be prone to manipulation. For example, one office was concerned that the amount of time may be skewed by workers switching in and out of after-call work time, a manual process that relies on worker judgment. Online Systems. Most dissatisfaction with performance measure accuracy among online system measures stemmed from the ways in which systems capture and generate data. Reports of dissatisfaction with online systems were spread across six performance measures, with only seven state and local offices reporting their dissatisfaction with respect to this initiative. Massachusetts reported concern with its online system's ability to stop duplicate cases from being submitted and so was dissatisfied with its Percent of Multiple Applications measure. Document Imaging. Only one office reported not being satisfied with a document imaging measure. A local Wisconsin office said that workers forget to close out of cases, which can generate inaccurate data for the calculation of Average Processing Time. Limitations Mathematica Policy Research 41 Table III.6. Selected Reports of Methods to Ensure Data Accuracy Performance Measure Training Monitoring\/ Quality Control Built-In System Prompts Call Center Percent of Changes Processed NY, OH NY, OH Average Waiting Time (to Speak to Agent) WI Average Queue Time WI Average Calls Handled OH OH Call Response (percent of time agents are available to take calls) NC Average After-Call Work Time by Staff WI Online Systems Average Number Submitted per Month CA Document Imaging Average Processing Time NY NY Partnering Number of Partners Providing Application Assistance NY Waiver of Face-to-Face Interview Percent of Interviews Conducted by Telephone MA, NY NY MA, WI Percent of Interviews Conducted Face-to-Face MA MA Error Rate for Cases with Telephone Interviews MA MA Percentage of Interviews Conducted by Phone Without Documented Hardship Reason WI Application Tracking Percent of Agency Caused Errors AK, MD Percent of Errors where Policy Incorrectly Applied AK Average Processing Time of Application Decision CA CA Percent of Applications Approved CA CA PA Percent of Applications Approved: Online All NY NY Percentage of Applicants that Appear Eligible for Expedited Benefits RI, WI Note: As reported by a state or local office, or both. Application Tracking. Across all performance measures under application tracking, there were only eight for which states reported any dissatisfaction. These were clustered around performance measures dealing with agency-caused errors, and percents of applications denied. Respondents indicated their concern regarding the ability to determine the fault for errors and distinguish new applications from recertifications or duplicates. All of these data accuracy issues illustrate how difficult it is to ensure that high-quality data are used to calculate performance measures. If data are inaccurate, then measures are poor reflections of state performance and not good bases for comparison. While states have enacted measures to ensure the accuracy of their data, these processes vary from state to state, and even within initiative, further complicating the ability to make useful cross-site comparisons. Setting standards for these measures can create incentives for workers to perform better or force them to find workarounds that may compromise data accuracy further. Limitations Mathematica Policy Research 42 2. Validity Validity refers to whether states actually are measuring what they intend to measure, or if their measures include data they do not intend to capture. Ascertaining the validity of a measure requires a firm understanding of the data that go into each performance measure calculation, as well as the data excluded. Table III.7 shows selected state reports of performance measure validity concerns. We do not have detailed information on the concerns that states and localities have about the validity of each measure. Numerous respondents indicated validity concerns on their survey submissions without specifying the reason for their concern, and it would have been too burdensome to request this information from each respondent. Call Centers. States reported more concerns about their call centers' ability to capture what they intended to measure relative to other initiatives. This could reflect the high number of states with call center initiatives, the tendency to create more performance measures for this initiative, or could be a factor of how states perform calculations for these measures. It also may reflect some of the contextual limitations described earlier in this chapter. For example, South Carolina respondents said they were concerned about the validity of the Average Call Duration because the measure failed to consider the nature of the call, which tends to dictate its length. Overall, eight states reported validity concerns with call center measures, and some states had recurring concerns (for example, West Virginia had concerns about four measures, and Pennsylvania, Wisconsin, and Utah each had concerns about two). Online Systems. Perhaps because online system performance measures tend to be generated automatically by the systems themselves, there were fewer reports of validity concerns with these measures. The concerns reported clustered around two measures related to online applications: Average Number Started per Month and Average Number Submitted per Month. In addition, Michigan reported that the Average Number of Accounts Created per Month did not capture the number of individuals who view their own cases and thus likely underestimates the use of this function. Virginia staff were concerned with their measure of the number of online applications submitted because they thought people might be starting online applications but not submitting them that way. They also reported that, because applicants can submit applications with only a name, address, and signature, users may not get far enough into the application to indicate whether they received help to apply online. Document Imaging. States reported very few validity concerns for document imaging performance measures. Delaware expressed concern that local offices that retroactively scan in old documents and case files show better monthly performance on Percent of Documents Scanned, thus invalidating the accuracy of this measure. Partnering. Three states\u2014Massachusetts, Nebraska, and Rhode Island\u2014reported validity concerns with partnering performance measures on their survey. Although they did not report the nature of their concerns, several logical conclusions can be drawn based on what we learned about those states' circumstances. For example, the length of time the initiative has been in place and how much time the state had to create and test their measures may influence whether states reported validity concerns with partnering. Rhode Island, for instance, had implemented a new partnering arrangement that was still being reviewed when the data were reported to us. Additionally, the variety and quantity of partners in a state may introduce validity concerns for performance measures covering partnering. Massachusetts also voiced concern about the validity of partnering measures. It has both formal and informal partnership agreements, with different understandings of compensation and expected performance for each partner. Limitations Mathematica Policy Research 43 Interview Initiatives. The bulk of the validity concerns around interview initiative measures came from respondents in Ohio, although we do not know the specific nature of their concerns. It is not surprising that a state would be concerned about more than one measure for an initiative because if it doubts the data going into one calculation, that same data could impact other calculations as well. Additionally, we surveyed numerous local offices in Ohio, so this state had more opportunity to express validity concerns than other states. Also, the state initiatives are county administered and so may have atypically high within-state variation. Table III.7. Validity Concerns for Selected Performance Measures Across Initiatives Measure States with Validity Concernsa Call Center Average Call Duration PA, SC, UT, WV, WI Average Answer Speed PA, WI Average Calls Handled UT, WV Average Hold Time GA, PA, UT, WV Percent of Changes Processed DE Online Systems Average Number of Accounts Created per Month MI, UT Percent of Applicants Who Received Help to Apply Online VA Average Number (of Applications) Started Per Month FL, MA, VA Average Number (of Applications) Submitted Per Month SC, UT Document Imaging Percent of Documents Scanned DE Percent of Documents Received by Fax UT Partnering Percent of Partners Providing Access to Terminals\/Kiosks NE Percent of Partners Providing Application Assistance MA, NE Waiver of Face-to-Face Interview Error Rate for Cases with Face-to-Face Interviews and Telephone Interviews OH Percent of Interviews Conducted by Telephone OH Percent of Interviews Conducted Face-to-Face MA, OH, WI Application Tracking Percent Client Caused Errors LA Percent of Applicants that Appear Eligible for Expedited Benefits PA, UT Percent of Applications: Online MA, UT, TX Average Processing Time of Application\/Recertification Decision CA, MN, PA, UT, VA a When asked if they thought the measure truly captured the measure of performance or efficiency it was intended to measure, respondents in state or local offices indicated either no or that it had some limitations. Standards and Benchmarks. Across initiatives, most states with performance standards or benchmarks associated with their measures do think their measures reflect the aspects of program performance or efficiency they were intended to measure. However, some states think they have based performance standards or benchmarks on measures that do not truly reflect the aspects of program performance they were intended to capture. Table III.8 shows selected states and localities Limitations Mathematica Policy Research 44 that reported having benchmarks based on performance measures they viewed as having some16 validity limitations. When reviewing these data, it is important to consider that some standards and benchmarks may be informal or internal and used by local offices, not applied across the state. Some states also may set standards to motivate workers, such as those shown in Table III.8, thus tying standards to measures even though those measures are not perfect representations of what they are trying to gauge. Although we do not know why states and local offices chose to implement performance standards and benchmarks on data they felt might be invalid, the fact that they did so limits the comparability of these standards with those of states that based theirs on data they believed to be valid. Table III.8. Selected States and Localities Reporting Standards or Benchmarks on Measures Viewed as Possibly Being Invalid Measure State or Locality Standard or Benchmark Call Center Average After-Call Work Time by Staff UT 20 minutes (one local office) Average Calls Handled UT Within 5 percent of the team average for number of calls WV Minimum of 40 calls per day per worker Average Number of Calls per Case FL 1.57 calls Average Waiting Time (to Speak to Agent) WV Less than 5-7 minutes Interview Initiatives Percent of Interviews Conducted Face-to-Face WI At least 50 percent completed face-to-face Note: This illustrative sample of state or local offices reporting a benchmark on a performance measure they considered invalid does not represent every office reporting validity concerns and benchmarks. 3. Reliability Data reliability addresses the consistency of calculating performance measures over time. Reliable measures are those that have not been changed recently\u2014either in how they are defined or calculated. Changes to how performance measures are calculated or defined are not inherently bad but may indicate that a measure was not calculated in the best way prior to the change, so impairing comparability over time. Conducting cross-state comparisons when some measures are less reliable than others can over- or underestimate the difference between states in performance changes over time. Offices change their performance measure calculations for three main reasons: (1) computer system changes, (2) changing business practices, and (3) shifting programmatic requirements (Table III.9). Changes to software or computer systems can force changes in calculations. These types of system changes happened recently in part of Wisconsin, which obtained new call center software, and in Utah, when its online systems were combined across three programs. Other changes occurred because of changing business practices. Illinois's call center measures changed when a staff reduction forced the office to limit the queue capacity. Similarly, the conversion to a new office 16 States or local offices reported either some limitations or no [this does not reflect what it is intended to]. Limitations Mathematica Policy Research 45 Table III.9. Sample of Changes to Performance Measures Affecting Data Reliability State Measure(s) Change(s) in past year Call Center Illinois Average Waiting Time (to Speak to Agent) Average Hold Time Average Queue Time The measure has not changed, but a system change has affected the review of the data results. Due to a staff reduction, a business decision was made to decrease the number of calls allowed in the queue by approximately 40 percent, impacting all of these measures. Indiana Percent Calls Abandoned Some offices are converting to a new office model. For the new model, the standard will be 5 percent or less. For the old model, it was 7 percent or less. Wisconsina Number of Calls Answered\/Handled New phone system software. Wisconsina Average Time to Process Change New phone system software. Online System Utah Percent of Applications Received Average Number of Changes Submitted per Month The online system was consolidated from three applications into one. No information. Document Imaging Wisconsina Percent of Documents Scanned Developed a new scanning procedure. Partnering Massachusetts Percent of Partners Providing Application Assistance No information. Interview Initiatives Californiaa Percent of Interviews Conducted by Telephone Added a drop-down box to record how interview was conducted. Also including a place for clients to request a telephone interview. Massachusetts Percent of Interviews Conducted Face-to-Face\/by Telephone Error Rate for Cases with Telephone Interviews Now has a systematic way to collect these data and track them; used only quality control review data before. No information. Application Tracking New Mexico Average Benefit Amount Constantly changing eligibility system. Pennsylvania Percent of Community Partner Applications Approved Added a requirement that partners use online system. Before, partners also could use paper applications. An online application is easier for the state to manage. Rhode Island Average Processing Time of Application Decision Driven in part by a federal lawsuit, data are now entered up front using the application panel. aLocal office provided this information. model in some Indiana offices means an expectation of increased efficiency. In California, one office added a drop-down box to its system to capture data on rates of telephone versus face-to-face interviews. Pennsylvania began requiring partners to use their online systems to provide application assistance, thereby allowing the state to track the Percent of Community Partner Applications Limitations Mathematica Policy Research 46 Approved through the online system. Finally, shifting programmatic requirements led some offices to change their performance measure calculations. In Massachusetts, a move to reimburse partners for providing application assistance led to new calculations of the Average Cost of Partner Arrangements and the Percent of Partners Providing Application Assistance. Despite these examples, most states did not report recent changes in how they calculated their performance measures. Respondents reported two main reasons for not changing measures within the past year. The first and most common reason was that the measure or initiative was so new it had not been in place for a full year. The second was that the state or local office reported using the measure either rarely or never. C. Conclusion States implement and operate their modernization initiatives in complex circumstances unique to their areas and populations. They must carefully select performance measures and standards based on their service environment, the other initiatives in use, and their individual goals. Meaningful comparisons of performance outcomes across states require understanding the contextual and data limitations that each state faces. In this chapter, we focused on cross-state comparison limitations, but within-state variations in context and data quality also may be present. Not enough data are available from this study to systematically compare local office performance within states, but we can assume that they face similar limitations with their data. Local areas serve different populations with a special set of services, face unique system limitations, and have different reasons for measuring select aspects of performance. In addition, we uncovered instances in which state and local offices reported different methods of calculating performance measures and setting different standards. Except in states where all decisions about data entry\u2014from systems used to staff training on those systems\u2014are made at the state level and strictly followed at each local office, we can expect to see various data quality issues as well. Thus, within-state comparisons may encounter the same challenges described above. Future research could investigate within-state data variations, which might help states set common performance measures and standards using high-quality data. 47 REFERENCES Castner, Laura, Amy Wodarek O'Reilly, Kevin Conway, Maura Bardos, and Emily Sama-Miller. Performance Measures for Supplemental Nutrition Assistance Program Modernization Initiatives. Washington, DC: Mathematica Policy Research, 2012. Cody, Scott D., Ren\u00e9e Nogales, and Emily Sama Martin. Modernization of the Food Stamp Program in Florida. Washington, DC: Mathematica Policy Research, February 2008. Rowe, Gretchen, Sam Hall, Carolyn O'Brien, Nancy Pindus, Robin Koralek. Enhancing Supplemental Nutrition Assistance Program (SNAP) Certification: SNAP Modernization Efforts Interim Report Volume 1. The Urban Institute: Washington, DC, April 2010. 49 GLOSSARY Modernization Initiatives Call Center: A point of access through which individuals place a call to obtain information or some other service by listening to a recording, talking to a person, and\/or interacting with a computer. Within a state, call centers may operate statewide or regionally and place calls, receive calls, or both. Each call center may employ a different combination of functions, including answering basic questions about a program or a specific case, change reporting, screening for potential eligibility, and taking applications. Caseworkers at call centers also may process the information they receive, including faxes or reported changes. Online System: Internet sites with which individuals may access and use to submit or receive information about either their potential eligibility or existing SNAP case. (This does not include websites that provide only general information about a program.) Online systems may allow users to screen for potential eligibility, submit applications online, or check the status of\/report changes to an existing case. Some online applications integrate directly with state eligibility determination software, while others are separate tools. In some states, partners (see below) have special levels of access to the online system enabling them to serve the applicants they assist. Document Imaging: A system or a process through which states or local offices scan paper documents and create and store an electronic version for future access. Kiosks: A computer terminal located in the community or an office (but not in a home) through which an individual may access one or more functions of an online system. Partnering: Formal relationships or services that the SNAP agency solicits from nongovernmental groups. The solicitation may be in the form of a contract, memorandum of understanding, or another type of request. Waiver of Face-to-Face Interview: Formal approval received from FNS to perform interviews for application, recertification, or both by telephone rather than in person. Shortened Interviews: Use of an abbreviated interview protocol for certain types of households, based on their characteristics. Measurement Terminology Data Element: An instance of an activity or characteristic that forms the basis of aggregate data and performance measure calculations. Aggregate Data: Counts of data elements, such as the number of observations. Performance Measure: Calculation used to assess an activity, typically displayed as a percentage or average. Performance Standards: These include both standards and benchmarks. A standard is the desired outcome of an activity\u2014for example, answering a call within three minutes, while a benchmark is the desired rate of success\u2014for example, answering at least 80 percent of calls within three minutes. A.1 APPENDIX A STATE PROFILES This appendix contains, in alphabetical order, the profiles of SNAP modernization initiatives and their performance measures for the 45 states (including the District of Columbia) that participated in the study through surveys, interviews, and sharing of performance data. Each profile opens with key facts about the state: how SNAP is administered, the number and types of respondents contributing, and the initiatives active in the state. Next, we present a table that lists the performance measures and aggregate data each state or its localities or partners reported to us. Each table is arranged by modernization initiative, describing the area in which a given initiative is active, specifying whether a given measure is implemented statewide or locally, and listing the goals the state or locality is trying to attain through use of the measure. Finally, we provide any standards, benchmarks, or incentives the state or locality applies to the measure. Following the table, we supply a brief narrative, giving an overview of the initiatives active in each state to provide context for how the performance measures are implemented. In developing these appendices, we strove to report as exactly as possible what respondents had told us, within reason. For some measures, additional refinement was needed. For example, if a respondent reported on the survey that no standard was in place, but during their interview he or she described a standard, we made this update to our database and reported it in the appendix. We also renamed some measures for the sake of consistency across states. If one or more local offices in a state-administered state specified that they use a measure from reports the state office gave them, even if the state had not indicated doing so on its survey, we reported the measure as being implemented statewide. Across states, some acronyms appear frequently: \uf0b7 ARRA the American Recovery and Reinvestment Act, which provided some funds for development and implementation of new SNAP modernization initiatives in some states. \uf0b7 AVR and ARU Automated Voice Response and Automated Response Unit are two names typically given to the computer systems that run call centers; these work to answer calls and route and queue them according to an algorithm the call center establishes. \uf0b7 QC FNS Quality Control regulations and procedures that seek to maximize the accuracy and integrity of SNAP. Some responses also appear frequently, as a result of the structure of the survey we used to collect the data: \uf0b7 Not reported survey respondents did not enter any information about a question \uf0b7 No survey respondents answered no \uf0b7 Not sure survey respondent was unsure of the answer to a question \uf0b7 None no FNS goals were noted as a focus for a given measure \uf0b7 Yes, not specified survey respondent answered yes to say that a benchmark, standard, or incentive existed for a measure, but did not provide further detail A.3 Profile: Alabama State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 2 Number of Partners Interviewed: 0 Initiatives Active in State: Partnering, Waiver of Face\u2010to\u2010Face Interview, Shortened Interviews, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives PARTNERING (Statewide) Number of Applications Received per Partner Statewide Program Access, Efficiency, Customer Service No No WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Percent that are Missed and have Notice Mailed Locally Not reported Not reported Not reported Total Number of Interviews Locally Not reported Not reported Not reported Number of Redetermination Interviews Locally Not reported Not reported Not reported Number of Missed Interviews Locally Not reported Not reported Not reported SHORTENED INTERVIEWS (Statewide) Number of Shortened Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No APPLICATION TRACKING (Statewide) Percent of Community Partner Applications Approved Statewide Program Access No No Percent of Total Applications Approved Statewide Program Access, Accuracy & Integrity No No Percent of Total Recertifications Approved Statewide Program Access No No Average Benefit Amount (for those eligible) Statewide None No No Number of Applications Approved: Paper Submission to Local Office Statewide Program Access, Customer Service No No Number of Recertifications Approved: Paper Submission to Local Office Statewide Program Access, Customer Service No No Total Number of Recertifications Completed Statewide Program Access, Accuracy & Integrity No No Alabama (continued) A.4 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applications: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity No No Total Number of Applications Completed Locally Not reported No No Percent of Applicants that Appear Eligible for Expedited Benefits Statewide Program Access, Accuracy & Integrity, Customer Service No No Percent of Initial Applications Denied Statewide Program Access, Accuracy & Integrity, Customer Service No No Percent of Recertifications Denied Statewide Program Access, Accuracy & Integrity, Customer Service No No Number of Applications Denied: Paper Submissions to Local Office Statewide Program Access, Accuracy & Integrity, Customer Service No No Number of Recertifications Denied: Paper Submissions to Local Office Statewide Program Access, Accuracy & Integrity, Customer Service No No Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Description: General: Alabama runs a demonstration in three counties, called the Elderly Simplified Application Demonstration. For this demonstration, they have a waiver for face-to-face interviews. Recently, Alabama began working with two community partners that conduct outreach and assist individuals with applying for SNAP. Alabama's computer system is at least 30 years old and can limit their ability to collect information. Measurement Goals: Alabama primarily measures performance to ensure customer service, accuracy and integrity, and program access. Call Center: Alabama does not have a call center. Online System: Alabama does not have an online system. Document Imaging: Alabama does not have a document imaging initiative in place. Kiosks: Alabama does not have kiosks available for application access and submission. Waiver of Face-to-Face Interview: Alabama has a waiver of the face-to-face interview at both application and recertification. State staff reported that Alabama's mainframe system is old and is not able to distinguish between cases that had telephone versus face-to-face interviews. Local staff reported that Alabama (continued) A.5 they track the number of clients that miss the interview and the percentage that were mailed a notice after an interview were missed. Shortened Interviews: Alabama has a shortened interview demonstration, according to state staff, but did not provide additional detail. They track the number of shortened interviews and the result of each one (benefit amount and disposition). Online Expedited Applications: Alabama does not have online expedited applications. Application Tracking: Alabama tracks a wide range of measures and aggregate data related to applications and recertifications, including the percentage completed, approved, and denied, and the average benefit of those who are approved. Administrators in state regions examine the results of the percentage of applications approved every three to six months for their particular group of counties. Their reviews may be adjusted if the county is under corrective action. Alabama also tracks the accuracy measures required by FNS. Changes Over Time: Alabama has been collecting the application measures for the last 30 years without any changes. Desired or Planned Future Measures: Alabama is currently building the capacity to measure the percentage of applications approved and the percentage denied. Staff reported that they anticipate this will provide them with information on the quality of applications received. Partnering: Alabama works with two community partners and has formal arrangements with both. One partner is paid for each submitted application. Because their computer system is so old, Alabama is unable to automatically track the number of applications submitted through the partner and can only do so manually by matching names from the partner with names in the state data. Alabama pays their second partner a fee for outreach and limited application assistance. The fee is not contingent on the number of applications submitted, so the state does not track the number submitted from this partner. Measures Collected by Partners: We did not contact either of Alabama's partners as part of this study. A.6 Profile: Alaska State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 2 Number of Partners Interviewed: 0 Initiatives Active in State: Online System, Partnering, Waiver of Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives ONLINE SYSTEM (Statewide) None PARTNERING (Regionally) None WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Number of Telephone Interviews Statewide Program Access, Efficiency, Customer Service No No Total Number of Interviews Statewide Program Access, Efficiency, Customer Service No No Number of Face-to-Face Interviews Statewide Program Access, Efficiency, Customer Service No No Percent of Interviews Conducted by Telephone Statewide Program Access, Efficiency, Customer Service No No Percent of Interviews Conducted Face-to-Face Statewide Program Access, Efficiency, Customer Service No No Error Rate for Cases with Telephone Interviews Locally Not reported Not reported Not reported Error Rate for Cases with Face-to-Face Interviews Locally Not reported Not reported Not reported Number of Home Visit Interviews Locally Not reported Not reported Not reported Number of Missed Interviews Locally Not reported Not reported Not reported Number of Applicants Requesting a Phone Interview Locally Not reported Not reported Not reported Number of Applicants Requesting an In-Person Interview Locally Not reported Not reported Not reported Percent that are Missed and have Notice Mailed Locally Not reported Not reported Not reported Number of Applicants Requesting a Phone Interview Locally Not reported Not reported Not reported APPLICATION TRACKING (Statewide) Average Benefit Amount (for those eligible) Statewide Program Access Not reported Not reported Alaska (continued) A.7 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Average Processing Time of Application Decision Statewide Accuracy & Integrity, Efficiency, Customer Service Performance benchmark is an average processing time of 17 days. No Percent of Applications Approved After Paper Submission to Local Office Locally Not reported Not reported Not reported Percent of Recertifications Approved After Paper Submission to Local Office Locally Not reported Not reported Not reported Percent of Mailed Applications Approved Locally Not reported Not reported Not reported Percent of Mailed Recertifications Approved Locally Not reported Not reported Not reported Percent of Total Applications Approved Locally Not reported Not reported Not reported Percent of Total Recertifications Approved Locally Not reported Not reported Not reported Number of Applications Approved: Faxed Locally Not reported Not reported Not reported Number of Recertifications Approved: Faxed Locally Not reported Not reported Not reported Number of Applications Approved: Mailed Locally Not reported Not reported Not reported Number of Recertifications Approved: Mailed Locally Not reported Not reported Not reported Number of Applications Approved: Face-to-Face Interviewed Locally Not reported Not reported Not reported Number of Recertifications Approved: Face-to-Face Interviewed Locally Not reported Not reported Not reported Number of Applications Approved: Phone Interviewed Locally Not reported Not reported Not reported Number of Recertifications Approved: Phone Interviewed Locally Not reported Not reported Not reported Alaska (continued) A.8 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applications Approved: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Recertifications Approved: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Recertifications: Faxed Locally Not reported Not reported Not reported Number of Applications: Mailed Locally Not reported Not reported Not reported Number of Recertifications: Mailed Locally Not reported Not reported Not reported Percent of Recertifications: Paper Submission to Local Office Locally Not reported Not reported Not reported Percent of Applications: Paper Submission to Local Office Locally Not reported Not reported Not reported Percent of Applications: Faxed Locally Not reported Not reported Not reported Percent of Recertifications: Faxed Locally Not reported Not reported Not reported Percent of Applications: Mailed Locally Not reported Not reported Not reported Percent of Recertifications: Mailed Locally Not reported Not reported Not reported Total Number of Applications Completed Locally Not reported Not reported Not reported Total Number of Recertifications Completed Locally Not reported Not reported Not reported Number of Applications: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Recertifications: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Applications: Faxed Locally Not reported Not reported Not reported Number of Requests for Assistance\/One-page Applications Locally Not reported Not reported Not reported Alaska (continued) A.9 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Complete Applications Denied Locally Not reported Not reported Not reported Percent of Complete Recertifications Denied Locally Not reported Not reported Not reported Percent of One- page\/Requests for Assistance Denied Locally Not reported Not reported Not reported Percent of Initial Applications Denied Locally Not reported Not reported Not reported Percent of Recertifications Denied Locally Not reported Not reported Not reported Number of Applications Denied: Paper Submissions to Local Office Locally Not reported Not reported Not reported Number of Recertifications Denied: Paper Submissions to Local Office Locally Not reported Not reported Not reported Number of Applications Denied: Faxed Locally Not reported Not reported Not reported Number of Recertifications Denied: Faxed Locally Not reported Not reported Not reported Number of Applications Denied: Mailed Locally Not reported Not reported Not reported Number of Recertifications Denied: Mailed Locally Not reported Not reported Not reported Percent of Applicants that Appear Eligible for Expedited Benefits Locally Not reported Not reported Not reported Number of Cases with Complete Verification Submitted Locally Not reported Not reported Not reported Notes: State collects mandatory accuracy data through QC review. The table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data is listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Description: General: In Alaska, SNAP is state-administered, and the state's administration is divided into four regional offices. According to state staff, Alaska's population is not homogenous and this complicates introducing any statewide standards. Specifically, some areas are so rural that face-to-face interviews are Alaska (continued) A.10 difficult. The state reports they are less concerned with a set standard and more concerned with appropriately meeting the needs of their diverse population. Measurement Goals: Alaska primarily measures performance to ensure customer service, accuracy and integrity, and program access. Call Center: Alaska does not have a call center. Online System: Alaska has an online screening tool, but not an online application system. There are no measures in place for this initiative. Document Imaging: Alaska does not have document imaging. Kiosks: Alaska does not have kiosks. Waiver of Face-to-Face Interview: Measures of this initiative are produced automatically in a monthly state report. Alaska primarily uses this measure as a reference point when assessing approaches to program accessibility. The local offices we interviewed reported that they look at the state-level data and reports for tracking all measures. Shortened Interviews: Alaska does not have a shortened interview initiative. Online Expedited Applications: Alaska does not have online expedited applications. Application Tracking: For the case review system, Alaska intends to start issuing monthly reports (as they had done in the past). On a monthly basis, Alaska reports and reviews the average \"cycle time,\" or the average number of days between the application submission and the application decision. Alaska does not use the results of this measure on the state level, but reported that the four regional managers and district\/local supervisors assess their performance using these data and may make changes accordingly. The state can calculate approval rates and measures, but does not currently do so. Alaska samples the required number of cases for the FNS quality control sample. The state also uses a separate case review system to track information recorded, especially about agency-caused errors, during supervisory reviews. Reviews in that system are sampled by the supervisors at will, but are not a representative sample and are not related to the QC sample. For the first few months of employment, 100 percent of a new employee's cases are reviewed by their supervisor. The case review system also includes regional reviewers and some peer reviewers. Alaska has a monthly reporting system that identifies error trends from the case review system data. Changes Over Time: Alaska used to have a case numbering system that reported statistics by caseload (the cases a single employee was responsible to manage). However, with a new business process in place, work is pooled across staff under a specific supervisor. So, Alaska's caseload number no longer refers to a particular employee, but rather to a set of employees under a specific supervisor. Partnering: There is no statewide initiative for partnering, but one local office in the state has some partnerships. Their partners have paper applications available (for all benefit types, not only SNAP) and they offer application assistance. The local office also partners with a food bank to operate a Summer Feeding Program that distributes non-perishable meals to pregnant women and children. The local office staff trains the food bank staff on safe handling of the meals and asks them to count the number of meals and snacks distributed (for food inventory purposes), but does not track any measures related to SNAP. Local staff indicated that this may progress to statewide partnering arrangements and any performance measures would be established on the state-level at that time. Measures Collected by Partners: Not applicable. Alaska (continued) A.11 Desired or Planned Future Measures: Alaska is considering several modernization initiatives, including document imaging and an integrated online eligibility and application system. State staff said an integrated online system could improve the efficiency of data collection and data entry. But, they are cautious because they believe systems and procedures that would be appropriate for another state that had a primarily urban population or a population that was very well connected with the Internet and very technologically savvy would not necessarily work in Alaska. A.12 Profile: Arizona State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 6 Number of Partners Interviewed: 4 Initiatives Active in State: Call Center, Online System, Document Imaging, Kiosks, Partnering, Waiver of Face\u2010to\u2010 Face Interview, Shortened Interviews, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Statewide) Average Waiting Time (to Speak to Agent) Statewide Program Access, Customer Service No No Average Calls Handled Statewide Accuracy & Integrity, Efficiency Yes, not specified Yes, not specified Percent of Total Calls Handled Statewide Accuracy & Integrity There is a standard on only the agent level. The incentive is for agents to reach a certain number of calls per day. Agents are ranked based on the number of calls they have per day. Staff use this ranking for performance reviews. Average Staff on Phones Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Average Queue Time Statewide Efficiency No No Percent of Queued Calls Handled Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Calls (in flow) Statewide Program Access, Customer Service Not sure Not sure Number of Calls Abandoned Statewide Program Access, Efficiency, Customer Service No No Number of Calls Queued Statewide Efficiency Not sure Not sure Number of Calls Answered\/Handled Statewide Efficiency, Customer Service Not sure Not sure Number of Agents Assigned to Phones Statewide Efficiency, Customer Service Not sure Not sure Number of Calls Receiving Busy Signal\/Unable to Connect with Phone System Statewide Program Access, Customer Service No No Arizona (continued) A.13 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Average Time to Process Change Statewide Accuracy & Integrity, Efficiency Staff set the standards based on a time study. This time study was part of the initial review of call centers. Staff do not look at this measure now. High performance rankings. Number of Changes Received Statewide Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Number of Changes Processed Statewide Accuracy & Integrity, Efficiency Not sure Not sure Average Hold Time Statewide Program Access, Customer Service No No Percent Calls Abandoned Statewide Customer Service No No Average Call Duration Statewide Efficiency, Customer Service Yes, not specified Yes, not specified Number of Clients Accessing Computer Phone System Data Statewide Program Access, Efficiency, Customer Service Not sure Not sure Number of Calls Received Statewide Program Access, Efficiency, Customer Service Not sure Not sure Number of Hang-ups Statewide Accuracy & Integrity, Customer Service No No Number of Changes Received Statewide Accuracy & Integrity, Efficiency, Customer Service No No Number of Calls Not Completed in Computer Phone System Statewide Program Access, Customer Service No No Percent of Calls Completed in the Computer Phone System Statewide Program Access, Efficiency, Customer Service Yes, not specified Yes, not specified Percent of Changes Processed Statewide Accuracy & Integrity, Efficiency Yes, not specified Yes, not specified Number of Calls Completed in Computer Phone System Statewide Efficiency, Customer Service No No Number of Calls Transferred to Agent Statewide Program Access, Customer Service Not sure Not sure ONLINE SYSTEM (Statewide) Number of Online Screenings Started: Online Statewide Program Access No No Arizona (continued) A.14 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Online Screenings Completed: Online Statewide Program Access No No Number of Applications Started Statewide Program Access No No Number of Abandoned Applications Statewide Program Access No No DOCUMENT IMAGING (Statewide) Number of Documents Scanned Statewide Accuracy & Integrity, Efficiency, Customer Service No No Number of Tasks Completed Statewide Accuracy & Integrity, Efficiency, Customer Service No No Number of Alerts Processed Statewide Accuracy & Integrity, Efficiency, Customer Service No No KIOSKS (Statewide) None PARTNERING (Statewide) None WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) None SHORTENED INTERVIEWS (Statewide) None ONLINE EXPEDITED APPLICATIONS (Statewide) Percent of Applications Approved: Online Expedited Statewide Program Access, Efficiency, Customer Service No No Percent of Applications Approved: Online All Statewide Program Access, Efficiency, Customer Service No No Number of Expedited Applications Received Online Statewide Program Access, Efficiency, Customer Service Not sure Not sure Number of Applications Approved: Online Expedited Statewide Program Access, Efficiency, Customer Service No No Number of Applications Approved: Online All Statewide Program Access, Efficiency, Customer Service No No Arizona (continued) A.15 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives APPLICATION TRACKING (Statewide) Average Benefit Amount (for those eligible) Statewide None No No Percent of Complete Applications Denied Statewide Program Access No No Percent of Complete Recertifications Denied Statewide Program Access No No Percent of Initial Applications Denied Statewide Program Access No No Percent of Recertifications Denied Statewide Program Access No No Number of Applications Denied: Paper Submissions to Local Office Statewide Efficiency, Customer Service No No Number of Recertifications Denied: Paper Submissions to Local Office Statewide Efficiency, Customer Service No No Number of Applications Denied: Online Statewide Efficiency, Customer Service No No Number of Recertifications Denied: Online Statewide Efficiency, Customer Service No No Percent of Applications Approved After Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Customer Service No No Percent of Total Applications Approved Statewide Customer Service No No Percent of Total Recertifications Approved Statewide Customer Service No No Percent of Online Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Online Recertifications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Recertifications Approved After Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service N o No Average Processing Time of Application Decision Statewide Efficiency, Customer Service No No Arizona (continued) A.16 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Average Processing Time of Recertification Decision Statewide Efficiency, Customer Service No No Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Description: General: Arizona actively collects and reports performance measures on their modernization initiatives. All modernization efforts and performance measures are state-initiated and are standardized statewide. The state office distributes their data to local offices on a weekly and monthly basis and local offices can pull reports from the statewide mainframe system. Arizona reported that workforce management could be used to improve some data collection processes. Staff must currently compile many different reports to obtain certain performance measure data. Recently, SNAP caseloads in Arizona have increased dramatically and staffing has decreased. Based on this, some offices we interviewed reflected that they are less focused on performance measurement than handling a greater workload with fewer staff and a restricted budget. Measurement Goals: Arizona's performance measures touch on all four FNS goals. They have also designed and use performance measures to produce cost savings, increase timeliness and accuracy, track staffing needs, and for management purposes during performance reviews. Call Center: Arizona tracks numerous performance measures and sets of aggregate data through the state-run call center in Phoenix. This information is used to determine the appropriate level of call center staffing, analyze incoming call volumes, and support application activity such as change processing. Staff interviewed noted that the current system in place is inefficient and out-dated. Standards in place were developed from case studies conducted before developing the cal center. Online System: Arizona collects several sets of aggregate data that are distributed in reports for community partners and the general public. At the time of the interview, Arizona collected no performance measures on the online system. They were holding discussions to determine which performance measures to focus on, based on the measures required for the Arizona Medicaid program. One respondent interviewed noted recent problems with the transfer of information from the online system into the eligibility determination system, which may have skewed the approval\/denial rate and number of days for timeliness. They are working to resolve this issue. Partnering: Although Arizona participates in both formal (subscription based) and informal partnering arrangements, they do not track any measures under this initiative. Arizona partners help with outreach and application assistance. Some partners provide kiosks or a workspace for Department of Economic Security employees to help clients directly within the community. Measures Collected by Partners: Partners reported collecting data on the Arizona online system, the Arizona online system registration failures, percent of applications approved, percent of applications denied, and percent of applications pending. Arizona (continued) A.17 Document Imaging: Arizona collects the number of documents scanned, the number of tasks completed, and the number of alerts processed. They are working toward a more efficient document imaging system and currently have five pilot sites. Kiosks: Arizona reported the presence of two kiosks. One is part of a new initiative and is not yet widely used by the public. Rather, staff and customers have been using it together as a training start-up. A local office reported that they have a computer set up for online application submission, but it is not currently functioning. Waiver of Face-to-Face Interview: Arizona records the type of interview (face-to-face or telephone) conducted and would be able to produce ad-hoc reports, but they do not regularly track any measures under this initiative. Shortened Interviews: Arizona did not have a separate shortened interview statewide at the time of our data collection, but one local office reported they had implemented a shortened interview protocol in their area, and do not track any measures for this initiative. Online Expedited Applications: Arizona monitors approval rates for online applications and online expedited applications, as well as related aggregate data sets. Staff interviewed in one office indicated concern over the quality of these data, due to the system marking applications as expedites that are not truly expedites. Application Tracking: Arizona distinguishes between applications that were delivered online and applications that were not delivered online. Online applications can be further differentiated by their source. The general public uses a different model of the online system than partners (subscribers), who have assigned log-in information. This means the state can track partner versus non-partner applications as well as their outcomes. Arizona collects many measure related to a few areas of application tracking: approvals, accuracy, denials, and timely processing. Changes Over Time: The online application system became available to the public in 2009, and in late 2010 the state was in the process of improving the alignment between their document imaging system and the online application. Desired or Planned Future Measures: Most Arizona offices reported that they are satisfied with the performance measures they are calculating at this time. Some staff expressed an interest in seeing online application rates (by disposition) from other states in order to better understand the reasons clients are applying online and the characteristics of those applicants. Some staff are also interested in seeing performance measures and aggregate data from call centers in other states. One local office would like to see measures of case quality for individual workers (because they have no time to case read) and more detailed aggregate data on numbers of applications by their origin. A.18 Profile: California State Administered or County Administered: County Number of Local\/County Agencies Interviewed: 4 Number of Partners Interviewed: 5 Initiatives Active in State: Call Center, Online System, Document Imaging, Partnering, Waiver of Face\u2010to\u2010Face Interview, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Regionally) None ONLINE SYSTEM (Regionally) Number of Applications Downloaded Locally Not reported Not reported Not reported Number of Online Applications Submitted per Month Locally Not reported No No Number of Multiple Applications Locally Not reported No No Percent of Applicants with Hardship Reason for Requesting Phone Interviews Locally Not reported Not reported Not reported Average Level of Benefits Locally Not reported Not reported Not reported Number of First-Time Applications Denied Locally Not reported Not reported Not reported Number of Clients Requesting Help to Apply Online Locally Not reported Not reported Not reported Average Number of Changes Submitted per Month Locally Not reported Not reported Not reported Number of Changes Started Locally Not reported Not reported Not reported Number of Changes Submitted Locally Not reported Not reported Not reported Number of Applicants who did not Receive Help to Apply Online Locally Not reported Not reported Not reported Percent of Applications Received Locally Not reported No No Number of Applications Started Locally Not reported Not reported Not reported Number of First-Time Applications Approved Locally Not reported Not reported Not reported Average Number of Accounts Created per Month Locally Not reported Not reported Not reported California (continued) A.19 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Log-ins Locally Not reported Not reported Not reported Average Number of Screenings Started: Online Locally Not reported Not reported Not reported Average Number of Screenings Completed: Online Locally Not reported Not reported Not reported Percent of Users Selecting Spanish Language Locally Not reported Not reported Not reported Percent of Screenings Resulting in Application Submission: Online Locally Not reported Not reported Not reported Number of Online Screenings Started: Online Locally Not reported Not reported Not reported Number of Online Screenings Completed: Online Locally Not reported Not reported Not reported Percent of Applicants Who Received Help to Apply Online Locally Not reported Not reported Not reported Average Number Started per Month Locally Not reported No No Average Number Submitted per Month Locally Not reported No No Number of One- page\/Requests for Assistance Submitted Locally Not reported Not reported Not reported Number of Applications with Filing Date Only Locally Not reported Not reported Not reported Number of Applications Submitted With All Application Questions Answered Locally Not reported Not reported Not reported Number of Applicants who Received Help to Apply Online Locally Not reported Not reported Not reported DOCUMENT IMAGING (Regionally) Number of Documents Received Locally Not reported Not reported Not reported Number of Documents Processed Locally Not reported Not reported Not reported Number of Documents Scanned Locally Not reported No No California (continued) A.20 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Amount of Days in Queue to be Indexed Locally Not reported No No Number of Documents in Error Queue Locally Not reported No No Number of Documents Pending Indexing Locally Not reported Not reported Not reported Number of Barcode Errors Locally Not reported Not reported Not reported Number of Queues with Over 1,000 Documents Locally Not reported Not reported Not reported PARTNERING (Regionally) Percent of Partners Providing Application Assistance Locally Not reported Not reported Not reported Number of Partners Locally Not reported Not reported Not reported Number Accepting Applications Locally Not reported Not reported Not reported Number Providing Application Assistance Locally Not reported Not reported Not reported Number Clients Assisted per Partner Locally Not reported Not reported Not reported Number of Applications Received per Partner Locally Not reported Not reported Not reported WAIVER OF FACE-TO-FACE INTERVIEW (Regionally) Number of Redetermination Interviews Locally Not reported Not reported Not reported Number of Face-to-Face Interviews Locally Not reported Not reported Not reported Number of Telephone Interviews Locally Not reported Not reported Not reported Percent of Interviews Conducted by Telephone Locally Not reported No No Percent of Interviews Conducted Face-to-Face Locally Not reported No No Percent of Telephone Interview Requests Honored Locally Not reported Not reported Not reported Total Number of Interviews Locally Not reported Not reported Not reported Number of Home Visit Interviews Locally Not reported Not reported Not reported California (continued) A.21 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Missed Interviews Locally Not reported Not reported Not reported Notice of Missed Interview Mailed Locally Not reported Not reported Not reported Number of Applicants Requesting a Phone Interview Locally Not reported Not reported Not reported Number of Applicants Requesting an In-Person Interview Locally Not reported Not reported Not reported Number of Applicants who Did Not Answer to Type of Interview Requested Locally Not reported Not reported Not reported Percent that are Missed and have Notice Mailed Locally Not reported No No ONLINE EXPEDITED APPLICATIONS (Regionally) Percent of Applications Approved: Online Expedited Locally Not reported Not reported Not reported Percent of Applications Approved: Online All Locally Not reported Not reported Not reported Number of Applications Approved: Online All Locally Not reported Not reported Not reported Number of Expedited Applications Received Online Locally Not reported Not reported Not reported Number of Applications Approved: Online Expedited Locally Not reported Not reported Not reported APPLICATION TRACKING (Statewide) Percent of Total Recertifications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Average Benefit Amount (for those eligible) Statewide None No No Number of Recertifications Approved: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Applications Approved: Online Locally Not reported Not reported Not reported California (continued) A.22 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Recertifications Approved: Faxed Locally Not reported Not reported Not reported Number of Applications Approved: Mailed Locally Not reported Not reported Not reported Percent of Online Recertifications Approved Locally Not reported Not reported Not reported Number of Recertifications Approved: Online Locally Not reported Not reported Not reported Number of Applications Approved: Faxed Locally Not reported Not reported Not reported Number of Recertifications Approved: Face-to-Face Interviewed Locally Not reported Not reported Not reported Number of Applications Approved: Phone Interviewed Locally Not reported Not reported Not reported Number of Recertifications Approved: Phone Interviewed Locally Not reported Not reported Not reported Number of Applications Approved: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Applications Approved: Community Partner Locally Not reported Not reported Not reported Number of Recertifications Approved: Community Partner Locally Not reported Not reported Not reported Number of Recertifications Approved: Mailed Locally Not reported Not reported Not reported Number of Applications Approved: Face-to-Face Interviewed Locally Not reported Not reported Not reported Percent of Online Applications Approved Locally Not reported Not reported Not reported Percent of Applications Approved After Paper Submission to Local Office Locally Not reported Not reported Not reported California (continued) A.23 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Recertifications Approved After Paper Submission to Local Office Locally Not reported Not reported Not reported Percent of Community Partner Applications Approved Locally Not reported Not reported Not reported Percent of Mailed Applications Approved Locally Not reported Not reported Not reported Percent of Mailed Recertifications Approved Locally Not reported Not reported Not reported Percent of Total Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Complete Applications Denied Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Complete Recertifications Denied Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Initial Applications Denied Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Recertifications Denied Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Applications Denied: Paper Submissions to Local Office Locally Not reported Not reported Not reported Number of Recertifications Denied: Paper Submissions to Local Office Locally Not reported Not reported Not reported Number of Applications Denied: Online Locally Not reported Not reported Not reported Percent of Online Recertifications Denied for Failure to Submit Documentation Locally Not reported Not reported Not reported Number of Recertifications Denied: Online Locally Not reported Not reported Not reported Percent of Online Applications Denied for Failure to Submit Documentation Locally Not reported Not reported Not reported California (continued) A.24 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of One- page\/Requests for Assistance Denied Locally Not reported Not reported Not reported Number of Applications Denied: Community Partner Locally Not reported Not reported Not reported Number of Recertifications Denied: Community Partner Locally Not reported Not reported Not reported Number of Applications Denied: Faxed Locally Not reported Not reported Not reported Number of Recertifications Denied: Faxed Locally Not reported Not reported Not reported Number of Applications Denied: Mailed Locally Not reported Not reported Not reported Number of Recertifications Denied: Mailed Locally Not reported Not reported Not reported Percent of Applicants that Appear Eligible for Expedited Benefits Locally Not reported Not reported Not reported Number of Cases with Complete Verification Submitted Locally Not reported Not reported Not reported Total Number of Applications Completed Statewide Program Access, Efficiency, Customer Service Not reported No Total Number of Applications To Be Processed Statewide Program Access, Efficiency, Customer Service Not reported No Total Number of Recertifications Completed Statewide Not reported Not reported Not reported Percent of Recertifications: Paper Submission to Local Office Locally Not reported Not reported Not reported Percent of Recertifications: Online Locally Not reported Not reported Not reported Percent of Applications: Mailed Locally Not reported Not reported Not reported Percent of Recertifications: Mailed Locally Not reported Not reported Not reported Number of Applications: Faxed Locally Not reported Not reported Not reported Number of Recertifications: Faxed Locally Not reported Not reported Not reported California (continued) A.25 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Requests for Assistance\/One-page Applications Locally Not reported Not reported Not reported Percent of Applications: Online Locally Not reported No No Number of Applications: Online Locally Not reported Not reported Not reported Percent of Recertifications: Community Partner Locally Not reported Not reported Not reported Percent of Applications: Faxed Locally Not reported Not reported Not reported Percent of Recertifications: Faxed Locally Not reported Not reported Not reported Total Number of Applications Started Locally Not reported Not reported Not reported Total Number of Recertifications Started Locally Not reported Not reported Not reported Number of Applications: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Recertifications: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Recertifications: Online Locally Not reported Not reported Not reported Number of Applications: Community Partner Locally Not reported Not reported Not reported Number of Recertifications: Community Partner Locally Not reported Not reported Not reported Percent of Applications Not Completed Locally Not reported Not reported Not reported Percent of Applications: Paper Submission to Local Office Locally Not reported Not reported Not reported Percent of Applications: Community Partner Locally Not reported Not reported Not reported Number of Applications: Mailed Locally Not reported Not reported Not reported Number of Recertifications: Mailed Locally Not reported Not reported Not reported California (continued) A.26 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Applications Approved Timely Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service State: All applications should be approved timelyLocal: Internal performance measures track whether individual workers approve 90 to 95 percent of applications timely. No Percent of Recertifications Approved Timely Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service All recertifications should be approved timely No Average Processing Time of Application Decision Locally Not reported Within 15 days\u2014applies to all applications. No Average Processing Time of Recertification Decision Locally Not reported Not reported Not reported Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Description: General: California counties are responsible for administering SNAP with state oversight (typically, counties work in consortia to purchase and maintain eligibility systems). As a result, modernization and measurement are implemented variably across the state. The state office collects reports from the counties on the status of their application and recertification processes. It also allocates funding to the counties for its major initiatives. California local offices have faced a few challenges with capturing and reporting data accurately during the development and refinement of their systems. Offices facing these problems have addressed them through system reprogramming or manual data counts. Measurement Goals: A few of California's performance measures touch on all FNS goals for modernization. Counties also pursued performance measurement in order to: meet FNS and grant requirements, meet a legal services agreement, identify trends, maximize programs, measure the success of outreach efforts, determine needed staff and equipment, and track the outreach provided. California (continued) A.27 Call Center: The state office reported that, while there is no statewide call center, some county offices have call centers of their own. However, none of the county offices we interviewed for the study had a call center in place. Online System: The presence or absence of an online application system varies by county. Each consortium with an online application has its own vendor to operate it. As such, functionality of the system and the measures collected differ among offices. All California local offices surveyed that have an online application system track the percent of total applications received online. Document Imaging: A few California local offices we interviewed have established document imaging. These offices track aggregate data only. Kiosks: Neither the state nor local offices interviewed have implemented kiosks. Waiver of Face-to-Face Interview: Two California counties interviewed have a waiver of face-to- face interviews in place. They both track performance measures and data on the volume and type of interview provided (telephone interview or face-to-face interview) and on notices of missed interviews. Shortened Interviews: No counties participating in the study have implemented a shortened interview process. Online Expedited Applications: Some California county offices interviewed review the percentage and the number of online expedited applications approved. County offices report the number of expedited applications processed to the state office on a quarterly basis. Further reporting capabilities, such as summarizing demographic or other characteristics of online expedited applications, vary by county. Application Tracking: State staff reported that application tracking typically occurs at the local level, as no applications are delivered to the state office. Most local offices responding to the survey focus on application accuracy, application approvals\/denials, and application receipt (though other areas of application tracking are monitored by some local offices). Changes Over Time: Some counties reported they have redesigned or implemented new initiatives within the past ten years, such as a call center, document imaging, or an online application system. They are working on developing data and adding measures or reports. Desired or Planned Future Measures: State and local offices expressed a desire to collect more information. Specifically, they would like to track: percent of applications approved and denied (by online versus paper submission), accuracy measures on phone versus face-to-face recertifications, measures that compare the current approval rate of recertifications to the rate prior to the implementation of a waiver of face-to-face interviews, number of people completing their Partnering: California's partners work on outreach, screening, and application assistance. Some partners operate under an interagency agreement (M.O.U.) with the California Department of Public Health, and some partners are compensated for their services. One office we surveyed has specified standards on the number of applications and screenings their partner must meet per month. Two local offices we surveyed reported that they collect performance data on partnerships. They monitor data, such as the percent of partners providing application assistance and some aggregate data. Measures Collected by Partners: One county we surveyed stated that the purpose of tracking performance measure data for their partnering initiative is to provide the data to their partner. The county provides the partner the information required to submit grant applications and the partner provides the county with the number of applicants they assisted. Another county's partner also tracks the number of applications submitted to the county office. Partners also collect numerous measures on outreach, trainings, volume and efficiency of application assistance. California (continued) A.28 recertification, number of people submitting their quarterly report online, number of abandoned applications, number of people submitting their application from a particular screen of the online application, and additional data on the online application system. A.29 Profile: Connecticut State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 2 Number of Partners Interviewed: 0 Initiatives Active in State: Call Center, Partnering, Waiver of Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Regionally) None PARTNERING (Statewide) Number of Partners Statewide Program Access, Customer Service No No Number Accepting Applications Statewide Program Access, Customer Service No No Number Providing Application Assistance Statewide Program Access, Customer Service No No WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) None APPLICATION TRACKING (Statewide) Average Benefit Amount (for those eligible) Statewide Program Access, Accuracy & Integrity, Efficiency Customer Service No No Number of Overdue Applications Statewide Program Access, Accuracy & Integrity, Efficiency Customer Service As few untimely as possible Not sure Applications Process Time 0-7 days Statewide Program Access, Accuracy & Integrity, Efficiency Customer Service FNS standards for timeliness Not sure Applications Process Time 8-29 days Statewide Program Access, Accuracy & Integrity, Efficiency Customer Service FNS standards for timeliness Not sure Percent of Mailed Applications Approved Locally Not reported Not reported Not reported Percent of Mailed Recertifications Approved Locally Not reported Not reported Not reported Percent of Total Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency Customer Service No No Percent of Total Recertifications Approved Locally Not reported Not reported Not reported Number of Applications Approved: Paper Submission to Local Office Locally Not reported Not reported Not reported Connecticut (continued) A.30 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Recertifications Approved: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Recertifications Approved: Mailed Locally Not reported Not reported Not reported Percent of Applications Not Completed Locally Not reported Not reported Not reported Percent of Applications: Paper Submission to Local Office Locally Not reported Not reported Not reported Percent of Recertifications: Paper Submission to Local Office Locally Not reported Not reported Not reported Total Number of Applications Started Locally Not reported Not reported Not reported Total Number of Recertifications Started Locally Not reported Not reported Not reported Total Number of Applications Completed Locally Not reported Not reported Not reported Total Number of Recertifications Completed Locally Not reported Not reported Not reported Number of Applications: Paper Submission to Local Office Locally Not reported Not reported Not reported Percent of Complete Applications Denied Statewide Not reported No No Percent of Complete Recertifications Denied Locally Not reported Not reported Not reported Number of Applications Denied: Paper Submissions to Local Office Locally Not reported Not reported Not reported Percent of Unexcused Applications Overdue at End of Month Statewide Not reported No No Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Connecticut (continued) A.31 Description: General: In Connecticut, SNAP operates through 12 regional offices and 3 partners, with state-level contracts that work primarily on the local level. Their computer system is 37 years old, and staff noted that it has caused problems. One local respondent reported issues and concerns with accurate and timely reports. One main issue is that the system cannot function in real-time. For example, the computer system does not record actions taken the same day, such as a client who applies and is determined eligible on the same day. Such immediate actions are thus not reflected on the monthly reports. Local offices we spoke with look at many application tracking measures that are not calculated at the local level, but rather are provided in state data and reports. Measurement Goals: Connecticut measures performance with a focus on all four FNS goals. Call Center: One local office in Connecticut runs a call center, which does screening and conducts interviews over the phone as well as processing faxes and applications and returning client calls. No data or measures are being tracked for the call center, however, according to the local office that runs it. Online System: Connecticut does not have an online application system. Document Imaging: Connecticut does not have a document imaging initiative in place. Kiosks: Connecticut does not provide kiosks for application access and submission. Waiver of Face-to-Face Interview: Connecticut has a waiver for face-to-face interviews, but the offices we surveyed do not collect measures or track performance. Staff record the interview type in the case notes narrative, but not in a trackable data field. Shortened Interviews: Connecticut does not have a shortened interview initiative. Online Expedited Applications: Connecticut does not have online expedited applications. Application Tracking: Connecticut tracks a number of measures and aggregate data related to applications and recertifications, including the percentage of total applications approved and the average benefit of those who are approved. State staff reported that aggregate application data are tracked, but specific information that would allow the state to break that information down is not tracked. QC reviews are done in accordance with FNS regulations. Connecticut had weekly calls among staff to discuss QC problems. In late 2008, these meetings were discontinued due to a shortage of QC reviewers. They planned to reinstate these calls beginning in December 2010. Changes Over Time: The same state computer system has been used since 1984. Desired or Planned Future Measures: At the time of our study, Connecticut was negotiating with a vendor to purchase an Online Application System, Automated Voice Response Centers, Call Centers, and Document Imaging technology. Partnering: Connecticut has formal arrangements with each of its three community partners. The state does not collect any performance measures for their partners, nor do they require the partners to collect any data. The state respondent thought some partners may collect their own data, used for their own purposes. One local office said that partners are compensated, but the state did not provide further information on compensation. Partners are responsible for SNAP outreach. They accept applications and forward the applications on to the agency. They also do SNAP education and nutrition education. Outreach contractors also answer applicant question. Measures Collected by Partners: We did not contact any of Connecticut's partners as part of this study. A.32 Profile: Delaware State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 2 Number of Partners Interviewed: 1 Initiatives Active in State: Call Center, Online System, Document Imaging, Partnering, Waiver of Face\u2010to\u2010Face Interview, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Statewide) Average Answer Speed Statewide Program Access, Efficiency, Customer Service No If an individual's speed is consistently below peers, they form a performance improvement plan with corrective action steps. Average Hold Time Statewide Program Access, Efficiency, Customer Service It is compared to the unit and worker average. The rate is used in individual performance reviews. Excessive hold time is an indication of staff lack of knowledge or lack of attention to the caller. Percent Calls Abandoned Statewide Efficiency, Customer Service No Not sure Average Call Duration Statewide Efficiency, Customer Service Individual average call duration is compared to the average for staff in the unit. Staff who consistently spend a large amount of time on the majority of their calls may have this addressed in their performance review if improvement s are not made after supervisory intervention. Average Waiting Time (to Speak to Agent) Statewide Program Access, Efficiency, Customer Service No No Delaware (continued) A.33 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Average Calls Handled Statewide Efficiency No Staff who consistently have low or high averages will have this data discussed in their performance reviews and conferences. Individuals who consistently perform below their unit average will have corrective action measures imposed through a performance improvement plan. Percent of Total Calls Handled Statewide Efficiency No No Percent of Changes Processed Statewide Program Access, Accuracy & Integrity, Customer Service Not sure Not sure Average Staff on Phones Statewide Efficiency No No Average Queue Time Statewide Program Access No No Average After-Call Work Time by Staff Statewide Accuracy & Integrity, Efficiency, Customer Service No No Number of Calls (in flow) Statewide Program Access, Efficiency, Customer Service No Not sure Number of Calls Abandoned Statewide Program Access, Efficiency, Customer Service Not sure No Number of Calls Queued Statewide Program Access, Customer Service Not sure No Number of Calls Answered\/Handled Statewide Program Access, Efficiency, Customer Service Not sure Not sure Number of Agents Assigned to Phones Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure No Number of Calls Received Statewide Program Access, Customer Service No No Number of Changes Completed Statewide Not reported Not reported Not reported Delaware (continued) A.34 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Average Time to De- Queue Statewide Not reported Not reported Not reported Percent of Queued Calls Handled Statewide Not reported Not reported Not reported Number of Calls Receiving Busy Signal\/Unable to Connect with Phone System Statewide Not reported Not reported Not reported Percent of Calls Completed in the Computer Phone System Statewide Not reported Not reported Not reported Average Number of Calls per Case Statewide Not reported Not reported Not reported Average Time to Process Change Statewide Not reported Not reported Not reported Number of Clients Accessing Computer Phone System Data Statewide Not reported Not reported Not reported Number of Changes Received Statewide Not reported Not reported Not reported Number of Changes Processed Statewide Not reported Not reported Not reported Number of Calls Not Completed in Computer Phone System Statewide Not reported Not reported Not reported Number of Calls Completed in Computer Phone System Statewide Not reported Not reported Not reported Number of Client Calls Returned Statewide Not reported Not reported Not reported ONLINE SYSTEM (Statewide) Number of Online Screenings Completed: Online Statewide Program Access, Efficiency, Customer Service No No Number of Online Applications Submitted Statewide Not provided No No Number of One- page\/Requests for Assistance Submitted Statewide Program Access, Efficiency, Customer Service No No DOCUMENT IMAGING (Statewide) Number of Documents Tagged to a Case Statewide Not reported No No Number Indexed for a Document Type Statewide Not reported No No Delaware (continued) A.35 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Documents Scanned Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Average Processing Time Locally Not reported No No Time Between Scanning and Tagging Locally Not reported Not reported Not reported Time Between Scanning and Indexing Locally Not reported Not reported Not reported Number of Documents Received Statewide Not reported Not reported Not reported Number of Alerts Processed Statewide Not reported Not reported Not reported Percent of Documents Scanned Statewide Not reported No No Number of Documents Processed Statewide Not reported Not reported Not reported Number of Documents Received: Mail Statewide Not reported Not reported Not reported Number of Documents Received: Fax Statewide Not reported Not reported Not reported Number of Documents Received: Other Electronic Statewide Not reported Not reported Not reported Number of Tasks Completed Statewide Not reported Not reported Not reported PARTNERING (Statewide) Percent of Partners Providing Application Assistance Statewide Not reported Not reported Not reported Average Cost of Partner Arrangement Statewide Not reported Not reported Not reported Number of Partners Statewide Not reported Not reported Not reported Number of Applications Received per Partner Statewide Not reported Not reported Not reported Number Accepting Applications Locally Not reported Not reported Not reported Number Providing Application Assistance Locally Not reported Not reported Not reported Number of Clients Assisted Locally Not reported Not reported Not reported Number of Applications Completed Locally Not reported Not reported Not reported Delaware (continued) A.36 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Percent of Interviews Conducted by Telephone Statewide Program Access, Efficiency, Customer Service No No Percent of Interviews Conducted Face-to-Face Statewide Program Access, Efficiency, Customer Service No No Error Rate for Cases with Telephone Interviews Statewide Accuracy & Integrity, Efficiency, Customer Service No No Error Rate for Cases with Face-to-Face Interviews Statewide Accuracy & Integrity, Customer Service No No Total Number of Interviews Statewide Program Access, Efficiency, Customer Service Not sure Staff who have a lower than average number of interviews that result in an eligibility determinatio n are counseled and possibly put on an improvement plan. Number of Redetermination Interviews Statewide Program Access, Efficiency, Customer Service Not sure Staff with high performance are recognized at recognition events. Staff with low performance are counseled and may be put on a performance improvement plan. Number of Face-to-Face Interviews Statewide Not reported Not reported Not reported Number of Telephone Interviews Statewide Not reported Not reported Not reported Number of Missed Interviews Statewide Not reported Not reported Not reported ONLINE EXPEDITED APPLICATIONS (Statewide) Percent of Applications Approved: Online Expedited Statewide Not reported Not reported Not reported Delaware (continued) A.37 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Applications Approved: Online All Statewide Not reported Not reported Not reported Number of Applications Approved: Online Expedited Statewide Not reported Not reported Not reported Number of Expedited Applications Received Online Statewide Not reported Not reported Not reported APPLICATION TRACKING (Statewide) Number of Applications Approved Statewide Not reported Not reported Not reported Number of Applications Approved: Face-to-Face Interviewed Statewide Not reported Not reported Not reported Number of Recertifications Approved: Face-to-Face Interviewed Statewide Not reported Not reported Not reported Number of Applications Approved: Phone Interviewed Statewide Not reported Not reported Not reported Number of Recertifications Approved: Phone Interviewed Statewide Not reported Not reported Not reported Percent of Community Partner Applications Approved Locally Not reported Not reported Not reported Percent of Community Partner Recertifications Approved Locally Not reported Not reported Not reported Percent of Total Applications Approved Locally Not reported Not reported Not reported Percent of Total Recertifications Approved Locally Not reported Not reported Not reported Average Benefit Amount (for those eligible) Locally Not reported Not reported Not reported Number of Applications Approved: Community Partner Locally Not reported Not reported Not reported Number of Recertifications Approved: Community Partner Locally Not reported Not reported Not reported Number of Applications Approved: Mailed Locally Not reported Not reported Not reported Delaware (continued) A.38 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Recertifications Approved: Mailed Locally Not reported Not reported Not reported Total Number of Applications Started Statewide Not reported Not reported Not reported Total Number of Recertifications Started Statewide Not reported Not reported Not reported Total Number of Applications Completed Statewide Not reported Not reported Not reported Total Number of Recertifications Completed Statewide Not reported Not reported Not reported Number of Applications: Online Statewide Not reported Not reported Not reported Percent of Applications Not Completed Locally Not reported Not reported Not reported Number of Applications: Community Partner Locally Not reported Not reported Not reported Number of Recertifications: Community Partner Locally Not reported Not reported Not reported Number of Applications: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Recertifications: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Applications: Mailed Locally Not reported Not reported Not reported Number of Recertifications: Mailed Locally Not reported Not reported Not reported Number of Applications Denied Statewide Not reported Not reported Not reported Percent of Complete Applications Denied Locally Not reported Not reported Not reported Percent of Complete Recertifications Denied Locally Not reported Not reported Not reported Percent of Initial Applications Denied Locally Not reported Not reported Not reported Percent of Recertifications Denied Locally Not reported Not reported Not reported Number of Recertifications Denied: Community Partner Locally Not reported Not reported Not reported Delaware (continued) A.39 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applications Denied: Paper Submissions to Local Office Locally Not reported Not reported Not reported Number of Recertifications Denied: Paper Submissions to Local Office Locally Not reported Not reported Not reported Number of Applications Denied: Community Partner Locally Not reported Not reported Not reported Number of Applications Denied: Faxed Locally Not reported Not reported Not reported Number of Recertifications Denied: Faxed Locally Not reported Not reported Not reported Number of Applications Denied: Mailed Locally Not reported Not reported Not reported Number of Recertifications Denied: Mailed Locally Not reported Not reported Not reported Number of Cases Withdrawn Statewide Not reported Not reported Not reported Number of Cases Pending Statewide Not reported Not reported Not reported Percent of Applicants that Appear Eligible for Expedited Benefits Locally Not reported Not reported Not reported Number of Cases with Complete Verification Submitted Locally Not reported Not reported Not reported Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. General: The Delaware state office collects most of the performance data on its modernization efforts, which are all state-initiated. The state distributes reports to local offices. Local offices we interviewed also collect some measures in-house, particularly for application tracking. Respondents seemed satisfied overall with their data collection and calculation. One local office reported some software and equipment issues, but not all of these impact their performance measurement. Measurement Goals: Delaware's performance measures touch on all four FNS goals. They also use their data to assess worker performance, manage backlog, and for staffing purposes. Call Center: Delaware collects numerous measures and counts of data through their call center phone system. Unless otherwise noted in the table, the measures are calculated automatically by the Delaware (continued) A.40 phone system. There is one call center for the entire state. The data are stored and collected locally. When assessing performance, staff reported Delaware is less concerned with measures and more concerned with data, such as the number of calls and the number of calls abandoned. Delaware's call center handles changes for five different programs including SNAP. They are not able to break out call center data by program. Online System: Delaware has one statewide online application system. The state office records the number of online screenings completed, the number of online applications submitted, and the number of one-page\/requests for assistance submitted. Delaware watches the volume of people using the online system (in terms of the number of applications submitted) in particular, in order to ensure that the system is not backlogged with unprocessed cases. Document Imaging: In Delaware, documents are uploaded into the state run information system. The imaging system calculates the number of scanned documents. Data are entered directly into the system, which can be accessed statewide. Any item scanned is considered a document. Delaware tracks both performance measures and aggregate data, including the percent of documents scanned, number of documents tagged to a case, number indexed for a document type, and number of documents scanned. One local office reported that the computer reporting system associated with the scanner allows them also to track the average processing time, number of documents received, number of documents processed, and number of alerts processed. Kiosks: Delaware has not set up any kiosks. Waiver of Face-to-Face Interview: Delaware tracks measures on percent of interviews conducted by type (telephone versus face-to-face) and error rate by type (telephone versus face-to- face). They also collect aggregate data on the total number of interviews and redetermination interviews. Delaware can pull ad hoc reports on the application decision by type of interview. Currently, they only review that information at the end of the year. For every household, Delaware also captures whether there is an elderly member. Staff reported that theoretically, Delaware could examine case characteristics by interview type. However, the state does not pull these data regularly. Shortened Interviews: Delaware does not have a separate shortened interview. Online Expedited Applications: While the state did not report performance data on online expedited applications, both local offices interviewed reported that some measures and data are available to them, such as: percent of applications approved (online expedited and online all), number of online expedited applications approved, and number of expedited applications received online. Application Tracking: Delaware state and local offices collect a wide range of application tracking information, including data on application accuracy, approvals and denials, application processing and case characteristics, and application receipt. The eligibility system produces automated tallies for all application tracking aggregate data. While the state records the origin of Partnering: Delaware partners with multiple organizations at the state and county levels. The state does not collect any measures or data on their partners. One local office tracks aggregate data, such as the number of partners accepting applications, number of applications received per partner, and number of partners providing applications assistance. Measures Collected by Partners: One partner we interviewed collects aggregate counts of applications approved, applications denied, and applications submitted. Delaware (continued) A.41 applications that were delivered by partner or online, one local office interviewed also differentiates applications that were faxed in or delivered to the office in person. One local office noted issues with the date stamp on the computer system. If a client is already enrolled in one program, and wants to add another program, the system backdates all forms and verifications to the original program. When the local office runs any reports dealing with pending applications, the system will generate the number of days pending from the original application date, not the program add on date. This occurs rarely, and staff is aware of this system glitch. Changes Over Time: Delaware has not made any recent changes to the way they collect or calculate their performance data. One local office recently transitioned to a new process management model of business, wherein staff are divided into teams that handle specific tasks (answer phones, scan, intake, redeterminations, etc.). At the time of the interview, this office was in the process of measuring the efficiencies of the new system and working with other areas to create a standardized reporting system by February or March 2011. This system will include Excel spreadsheets that calculate performance measures for supervisors' use. Another local office discussed a new state initiative to synchronize reporting for all programs administered through the Department of Social Services, with the goal of limiting the need for clients to visit the office in person. Desired or Planned Future Measures: Call center workers handle five different programs including SNAP. Delaware would like to track how many calls relate to each program. When setting up the phone system, Delaware was unable to identify an easy and accurate way to automate this function. The state would also like to track client time in the office prior to the interview and time before benefits were issued. Right now, these can only be tracked manually. Staff interviewed reported that this calculation would allow Delaware to see which offices or staff are most efficient and which need coaching in terms of workload management. One local office expressed an interest reviewing a larger sample of cases for payment accuracy and case error rates. Currently, supervisors review 10 to 30 cases per worker each quarter. In addition, cases are reviewed for all targeted workers (either new workers or workers that have had issues in the past). A.42 Profile: District of Columbia State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 1 Number of Partners Interviewed: 0 Initiatives Active in State: Call Center, Shortened Interviews, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Statewide) Number of Calls (in flow) Statewide Not reported No No Number of Calls Abandoned Statewide Not reported No No Number of Calls Queued Statewide Not reported No No SHORTENED INTERVIEWS (Statewide) None APPLICATION TRACKING (Statewide) Percent of Total Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Total Recertifications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Average Benefit Amount (for those eligible) Statewide Accuracy & Integrity, Efficiency No No Percent of Applications Not Completed Statewide Accuracy & Integrity No No Percent of Applications: Paper Submission to Local Office Statewide Program Access No No Percent of Recertifications: Paper Submission to Local Office Statewide Program Access No No Total Number of Applications Started Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Total Number of Recertifications Started Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Total Number of Applications Completed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Complete Applications Denied Statewide Program Access, Accuracy & Integrity, Efficiency No No Percent of Complete Recertifications Denied Statewide Program Access, Accuracy & Integrity, Efficiency No No District of Columbia (continued) A.43 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Initial Applications Denied Statewide Program Access, Accuracy & Integrity, Efficiency No No Percent of Recertifications Denied Statewide Program Access, Accuracy & Integrity, Efficiency No No Number of Applications Denied: Paper Submissions to Local Office Statewide Program Access, Accuracy & Integrity No No Number of Recertifications Denied: Paper Submissions to Local Office Statewide Program Access, Accuracy & Integrity No No Percent of Applications with Incomplete Information Statewide Accuracy & Integrity No No Percent of Applicants that Appear Eligible for Expedited Benefits Statewide Program Access, Accuracy & Integrity, Customer Service No No Number of Cases with Complete Verification Submitted Statewide Program Access, Accuracy & Integrity No No Average Timeliness Rate: Approvals Statewide Program Access, Accuracy & Integrity No No Average Timeliness Rate: Recertifications Statewide Program Access, Accuracy & Integrity No No Notes: State collects mandatory accuracy data through QC reviews Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data is listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Description: General: The District of Columbia has two modernization initiatives and is not actively using performance measures for many purposes other than those required by FNS. The District has received a waiver of the face-to-face interview from FNS but has not yet implemented it. In order to implement the waiver, the District must first expand its call center. They do not yet know which performance measures will be tracked. However, they will track whether the interviewee is an English-language Learner in order to meet local reporting requirements. Measurement Goals: The District's performance measures touch on all four FNS goals. Call Center: The call center is run through the District. There is no performance measurement by the service center currently. When expanded, the call center will operate within the District's Enterprise Call Center. According to staff we interviewed, the District could make improvements in its modernization and performance measurement activities with additional employees. Although they did receive District of Columbia (continued) A.44 emergency funding for administrative purposes, that funding is only for a year. Ramping up the call center will require 6 months, although efficiencies may not be seen for 24 months. Online System: The District has not implemented an online system. Document Imaging: At the time of interview, the District had not yet implemented a document imaging system. Staff reported that the launch was planned for January 2011. Kiosks: The District does not have kiosks. Waiver of Face-to-Face Interview: The District has not yet implemented the waiver of the face-to-face interview. Shortened Interviews: The District reported that they use shortened interviews, but did not provide additional detail on this initiative (beyond noting that there are no measures associated with it). Online Expedited Applications: The District does not have online expedited applications. Application Tracking: The District can track which local office received the application, but not whether the application was received by mail, fax, or in person. Because they have not yet implemented the waiver of the face-to-face interview, nearly all applicants physically bring their applications to a service center. Changes Over Time: The launch of the District's document imaging system was planned for January 2011. They were testing the system's indexing function. They contracted with a vendor to complete the necessary back-scanning of documents. They do not anticipate recording the source of the documentation (mail, fax, etc.); however, they plan to track whether documents are indexed correctly and scanned timely. Indexing will track the document type, such as application, birth certificate, and so on. Document imaging will eventually be integrated with the case management system. Desired or Planned Future Measures: Although the District does not have additional measures in mind, staff reported that they would like to do more with the measures they have. Specifically, staff would like to be able to manipulate current measures more easily by breaking down by service center, worker, supervisor, and do on. The District currently has that data but cannot manipulate them in its current system. According to staff we interviewed, they have seen a significant spike in applications, and the necessary modernization to efficiently handle those caseloads requires significant upfront costs. Partnering: The District of Columbia does not have a formal initiative for partnering. Survey respondents speculated that some informal community partners might provide outreach or application assistance. Measures Collected by Partners: We did not contact any informal partners as part of this study. A.45 Profile: Florida State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 3 Number of Partners Interviewed: 1 Initiatives Active in State: Call Center, Online System, Document Imaging, Kiosks, Partnering, Waiver of Face\u2010to\u2010 Face Interview, Shortened Interviews, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Statewide) Percent of Agents Assigned to Phones Available to Take Calls Statewide Not reported 77 percent No Number of Calls (in flow) Statewide Program Access No No Number of Calls Abandoned Statewide Program Access No No Number of Calls Answered\/Handled Statewide Customer Service 65 calls per agent per day Meeting standards is part of call agents' performance appraisal Number of Agents Assigned to Phones Statewide Customer Service 276 agents No Number of Calls Receiving Busy Signal\/Unable to Connect with Phone System Statewide Customer Service 54 busy and incomplete calls No Average Answer Speed Statewide Efficiency, Customer Service 3 minutes No Percent Calls Abandoned Statewide Program Access, Customer Service 12 percent No Average Call Duration Statewide Efficiency 6 minutes 40 seconds No Average Calls Handled Statewide Program Access, Efficiency, Customer Service 60 times average staff on the phones No Percent of Total Calls Handled Statewide Program Access, Efficiency, Customer Service 88 percent No Average Staff on Phones Statewide Program Access, Efficiency, Customer Service 276 No Average Time Prior to Abandonment Statewide Program Access, Efficiency, Customer Service No No Number of Changes Received Statewide None No Not reported Number of Changes Processed Statewide Efficiency 26 changes completed per day per worker Yes, not specified Average Number of Calls per Case Statewide Program Access 1.57 (July 2010) Not reported Florida (continued) A.46 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Changes Processed Statewide None No No Average Time to Process Change Statewide Program Access 10 days No Average Waiting Time (to Speak to Agent) Statewide Not reported No No Number of Clients Accessing Computer Phone System Data Statewide Program Access No No Number of Calls Received Statewide Program Access No No Number of Hang-ups Statewide None No No Number of Changes Processed Statewide Efficiency Not reported Not reported Percent of Calls Requesting Agent Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service 66 percent No Number of Calls Completed in Computer Phone System Statewide Accuracy & Integrity No Not reported Number of Calls Transferred to Agent Statewide Accuracy & Integrity Not reported Not reported ONLINE SYSTEM (Statewide) Average Number of Screenings Completed: Online Statewide Program Access No No Number of Online Screenings Completed: Online Statewide Program Access No No Percent of Applicants Who Received Help to Apply Online Statewide Program Access No No Percent Reporting Easy Experience to Complete Application Statewide Program Access, Customer Service No No Percent Who Would Use Again Statewide Program Access, Customer Service No No Percent of Applications Received Statewide Not reported Not reported Not reported Average Number Started per Month Statewide Program Access No No Average Number Submitted per Month Statewide Program Access No No Florida (continued) A.47 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applications Started Statewide Program Access No No Number of Clients Requesting Help to Apply Online Statewide Program Access, Customer Service No No Number of Applicants who Received Help to Apply Online Statewide Program Access, Customer Service No No Average Number of Log- ins per Month Statewide Program Access, Efficiency No No Average Number of Accounts Created per Month Statewide Program Access, Customer Service Not reported No Average Number of Changes Submitted per Month Statewide Efficiency, Customer Service No No Number of Online Accounts Created Statewide Program Access, Efficiency No No Number of Log-ins Statewide Program Access, Efficiency No No Number of Changes Submitted Statewide Efficiency No No DOCUMENT IMAGING (Statewide) Number of Documents Scanned Statewide None No No KIOSKS (Statewide) None PARTNERING (Statewide) Percent of Partners Providing Access to Terminals\/Kiosks Statewide Not reported All level two and three partners provide assistance No Percent of Partners Providing Application Assistance Statewide Not reported Level two and three partners are expected to meet certain benchmarks Not reported Number of Partners Statewide Program Access, Customer Service Not reported Not reported Number Accepting Applications Statewide Program Access, Customer Service Not reported Not reported Number Providing Application Assistance Statewide Program Access, Customer Service Not reported Not reported Florida (continued) A.48 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number with Terminals\/Kiosks to Submit Application Statewide Program Access, Customer Service Not reported Not reported Number of Applications Received per Partner Statewide Program Access, Customer Service No No Average Cost of Partner Arrangement Statewide Not reported Not reported Not reported Number Clients Assisted per Partner Statewide Program Access, Customer Service Not reported Not reported WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) None SHORTENED INTERVIEWS (Statewide) None ONLINE EXPEDITED APPLICATIONS (Statewide) Percent of Applications Approved: Online All Statewide Efficiency 96 percent No Number of Expedited Applications Received Online Statewide None Not reported Not reported Number of Applications Approved: Online Expedited Statewide Efficiency No No Number of Applications Approved: Online All Statewide Efficiency No No APPLICATION TRACKING (Statewide) Percent of Applications: Online Statewide Not reported Not reported Not reported Number of Requests for Assistance\/One-page Applications Statewide Program Access No No Average Processing Time of Approvals (Application and recertifications combined) Statewide Efficiency FNS 7- or 30- day standard No Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. ARU = Automated Response Unit Florida (continued) A.49 Description: General: Florida was one of the first states to launch many of the initiatives examined here. From 2004 to 2006, they implemented changes through an effort known as Automated Community Connection to Economic Self-Sufficiency or ACCESS Florida. They restructured how applications were processed by having staff specialize in certain tasks, so that many individuals were involved in the processing of a case, and an applicant no longer worked with a single case worker. They screened applicants to identify those at low-risk for errors and conducted shortened interviews with those individuals. They began using online applications, developed internally, and provided kiosks, copiers, and fax machines in the local offices for applicants to access and submit information electronically. They also developed four call centers through which participants could report changes and ask questions about their case\u2014some questions can be answered through the automated response unit; others require transferring to a live agent. They also implemented document imaging and maintain all cases electronically so any worker can access information about any client. Florida continues to upgrade their systems\u2014they are continually moving toward a more automated system of transferring data from the online application to the eligibility system. Through customer surveys of individuals calling the call center, they identify questions that are commonly asked and try to provide better information about that issue to clients. They also identify phone numbers of individuals that commonly call the call center (screening out partners and other agencies that would be likely to have a high call volume) and call the individual to identify the issue that leads them to call often. If possible, they try to find a more automated way to address issues identified by the high-volume callers. Measurement Goals: Having had the initiatives in place for such a long period of time, Florida's measurement focus is on processing applications timely and without errors. They have the ability to track a very large number of measures, but do not regularly track many of them. Call Center: The call center has an automated response unit that answers the phone and can answer many of the callers' questions. If the question is not answered, the caller may choose to speak to an agent, in which case the call is transferred to one of four call centers (one of these call centers, which was opened with funds from the American Recovery and Reinvestment Act, was not intended to be permanent is and closing in 2011). Clients can apply or recertify for SNAP through the call center as well as report changes. Florida has fielded customer satisfaction surveys with callers entering the system. They have a benchmark to answer all calls within three minutes; however; due to high call volume and a focus on providing benefits in a timely manner, they have struggled to meet it. Online System: Applications submitted through the online system are automatically streamed to the access management system for processing by staff members. Once the application has been Partnering: Partnering arrangements are established by region of the state, through the community partner liaison. They are divided into levels based on the types of assistance they provide. Level two and three partners are those that have kiosks on site, self service, and assisted service sites. Some partners are compensated for their work and others are not, depending on the assistance provided and the need in the region. At the time of our interview, 87 partners were compensated, and they tend to be social service agencies, workforce boards, faith- based partners, and community centers. The compensation can be based on the volume of applications, but is not always tied to it. Partners may have formal arrangements with the state even though they are not compensated. Partners have unique addresses for ACCESS (URL codes) that identify the site. They also have their own view for application tracking. If a partner logs into the system without using their unique address, the state does not know that the application came from the partner. Measures Collected by Partners: Partners track the number of applicants who use their services, sometimes through sign-in sheets. Florida (continued) A.50 acknowledged by the staff member and checked for completion, the data are manually passed into the eligibility system. Florida would like to see 70 percent of its customers with ongoing cases using the online system. Document Imaging: The document imaging system started out as a regional initiative, and then became statewide. Faxes are automatically converted into electronic documents. Some scanning is done in a centralized document imaging center; some documents are scanned by staff in local offices. Once scanned, staff categorize the documents into approximately 15 categories, though with subcategories being added, they expect to have about 40 categories. Currently, the documents are attached to the full case, but Florida intends to change this to attach files to individuals (for example, attaching a birth certificate to a person, rather than a household\u2014allowing the birth certificate to be accessed for that individual for other purposes). Once attached to a case, the case worker is notified automatically that a document has been received. Florida stores information about the scanner that imaged the document and whether a document was received by fax or in paper form. Certain documents require access privileges. Kiosks: Kiosks are available in lobbies of local offices and partner agencies. Clients can use the kiosks to apply through ACCESS, check the status of their case, and submit changes. Applicants completing an online application are asked to identify their location when they complete the application, but the information is not required or verified. Waiver of Face-to-Face Interview: Florida stores information about whether the interview is conducted by phone or in person, however, they calculate no measures for this initiative. For the purposes of their waiver, they are required to track error rates for cases with each type of interview. Shortened Interviews: Florida screens applicants to identify those at low risk of having errors. These individuals participate in a shortened interview. Others participate in the full-length interview. Online Expedited Applications: Clients can submit expedited applications through the state's online system, and the state tracks application statistics (volume and approval rate) for these applications. Application Tracking: Florida built a quality management system that allows them to identify cases that should be reviewed for errors. Tracking of applications is largely to redistribute the workload and keep processing applications within the required window. Changes Over Time: Florida now views modernization as the new way of doing business\u2014looking at the combination of initiatives rather than as individual initiatives. Desired or Planned Future Measures: Staff interviewed reported that Florida would like to be better able to track the source of online applications. A.51 Profile: Georgia State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 7 Number of Partners Interviewed: 1 Initiatives Active in State: Call Center, Online System, Partnering, Waiver of Face\u2010to\u2010Face Interview, Shortened Interviews, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Statewide) Average After-Call Work Time by Staff Statewide Not reported Not reported Not reported Number of Calls (in flow) Statewide Customer Service Not reported Not reported Number of Calls Abandoned Statewide Customer Service Not reported Not reported Number of Calls Queued Statewide Customer Service Not reported Not reported Number of Calls Answered\/Handled Statewide Customer Service Not reported Not reported Number of Agents Assigned to Phones Statewide Customer Service Not reported Not reported Percent of Changes Processed Statewide Not reported Not reported Not reported Average Time to Process Change Statewide Not reported Not reported Not reported Number of Changes Received Statewide Customer Service Not reported Not reported Number of Changes Processed Statewide Customer Service Not reported Not reported Average Answer Speed Statewide Customer Service Less than one minute A governor's award that acknowledges good performance. Average Hold Time Statewide Not reported Not Sure Not sure Percent Calls Abandoned Statewide Customer Service Less than 9 percent abandoned rate was the governor's office standard. A governor's award that acknowledges good performance. Average Call Duration Statewide Customer Service 4-6 minutes total time per call A governor's award that acknowledges good performance. Average Waiting Time (to Speak to Agent) Statewide Not reported Not reported Not reported Average Calls Handled Statewide Not reported Not reported Not reported Georgia (continued) A.52 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Total Calls Handled Statewide Not reported Not reported Not reported Average Staff on Phones Statewide Not reported Not reported Not reported Average Queue Time Statewide Not reported Not reported Not reported Number of Calls Received Statewide Customer Service Not reported Not reported ONLINE SYSTEM (Statewide) Average Number of Screenings Started: Online Statewide Not reported Not reported Not reported Average Number of Screenings Completed: Online Statewide Not reported Not reported Not reported Percent of Users Selecting Spanish Language Statewide Not reported Not reported Not reported Percent of Screened Individuals Potentially Eligible: Online Statewide Not reported Not reported Not reported Percent of Screenings Resulting in Application Submission: Online Statewide Not reported Not reported Not reported Number of Online Screenings Started: Online Statewide Not reported Not reported Not reported Number of Online Screenings Completed: Online Statewide Not reported Not reported Not reported Percent Reporting Easy Experience to Complete Application Statewide Not reported Not reported Not reported Percent Who Would Use Again Statewide Not reported Not reported Not reported Percent of Applications Received Statewide Program Access, Customer Service No No Average Number Started per Month Statewide Not reported Not reported Not reported Average Number Submitted per Month Statewide Not reported Not reported Not reported Average Number of Changes Submitted per Month Statewide Program Access, Customer Service No No Number of Changes Submitted Statewide Not reported No No Number of Applications Started Statewide Not reported Not reported Not reported Georgia (continued) A.53 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applications Received Locally Not reported No No PARTNERING (Statewide) Percent of Partners Providing Application Assistance Statewide Program Access No No Average Cost of Partner Arrangement Statewide Efficiency Not reported Not reported Number of Partners Statewide Not reported Not reported Not reported Number Accepting Applications Statewide Not reported Not reported Not reported Number Providing Application Assistance Statewide Not reported All partners provide assistance Not reported Number with Terminals\/Kiosks to Submit Application Statewide Not reported Not reported Not reported Number Clients Assisted per Partner Statewide Program Access, Customer Service No Number of Applications Received per Partner Statewide Not reported Not reported Not reported Number of Outreach Events Statewide Program Access, Customer Service No Not reported Number of Clients served Statewide Program Access, Customer Service Not reported Not reported WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Error Rate for Cases with Telephone Interviews Locally Not reported Not reported Not reported Error Rate for Cases with Face-to-Face Interviews Locally Not reported Not reported Not reported Total Number of Interviews Locally Not reported Not reported Not reported Number of Redetermination Interviews Locally Not reported Not reported Not reported Number of Home Visit Interviews Locally Not reported Not reported Not reported Number of Applicants Requesting a Phone Interview Locally Not reported Not reported Not reported Number of Face-to-Face Interviews Locally Not reported Not reported Not reported Georgia (continued) A.54 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Telephone Interviews Locally Not reported Not reported Not reported Notice of Missed Interview Mailed Locally Not reported Not reported Not reported SHORTENED INTERVIEWS (Regionally) None ONLINE EXPEDITED APPLICATIONS (Statewide) Number of Expedited Applications Received Online Statewide Not reported Not reported Not reported Number of Applications Approved: Online Expedited Statewide Not reported Not reported Not reported APPLICATION TRACKING (Statewide) Percent of Total Applications Approved Statewide Not reported No No Percent of Total Recertifications Approved Statewide Not reported Not reported Not reported Average Benefit Amount (for those eligible) Statewide Not reported Not reported Not reported Number of Applications Approved: Paper Submission to Local Office Statewide Not reported Not reported Not reported Number of Recertifications Approved: Paper Submission to Local Office Statewide Not reported Not reported Not reported Percent of Complete Applications Denied Statewide Not reported No No Percent of Complete Recertifications Denied Statewide Not reported Not reported Not reported Percent of Initial Applications Denied Statewide Not reported Not reported Not reported Percent of Recertifications Denied Statewide Not reported Not reported Not reported Number of Applications that Appear Eligible for Expedited Benefits Statewide Not reported No No Number of Cases with Complete Verification Submitted Statewide Not reported Not reported Not reported Georgia (continued) A.55 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Applicants that Appear Eligible for Expedited Benefits Statewide Accuracy & Integrity, Customer Service No No Total Number of Applications Completed Statewide Not reported Not reported Not reported Total Number of Recertifications Completed Statewide Not reported Not reported Not reported Total Number of Applications Started Locally Not reported Not reported Not reported Number of Applications: Paper Submission to Local Office Statewide Not reported Not reported Not reported Number of Recertifications: Paper Submission to Local Office Statewide Not reported Not reported Not reported Number of Applications: Online Statewide Not reported Not reported Not reported Number of Recertifications: Online Statewide Not reported Not reported Not reported Pending Applications by Days in Process Statewide Efficiency, Customer Service FNS timeliness standards Not reported Unexpedited Standard of Promptness Rate (County\/Worker Only) Locally Not reported Not reported Not reported Expedited Standard of Promptness Rate (County\/Worker Only) Locally Not reported Not reported Not reported Notes: State collects mandatory accuracy data through QC reviews Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data is listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Description: General: Georgia relies on a regional structure for SNAP administration; the state office communicates with the region and the regional offices communicate with the counties. Offices at all levels actively collect performance measures and data on Georgia's initiatives, which are primarily implemented statewide. The state provides monthly reports that contain application tracking measures and data, which are used by offices in conjunction with county data collection. A regional office may monitor accuracy reports and perform second level readings of case accuracy reviews. Although Georgia is generally satisfied with their data collection and reporting, staff interviewed reported issues, such as: keeping track of data, workforce and staffing challenges, some data quality problems (initially the logic for identifying Georgia (continued) A.56 expedited cases was wrong and needed to be fixed), occasional system glitches, and difficulty accessing data that are older than 6 to 8 months. Georgia used ARRA funding to hire temporary workers. Staff were added to the call center to assist with registration. Other workers were located in county offices and assisted with interviewing clients, processing benefits, making and finding case records, and delivering case records to case managers. No data collection or performance measurement was used to evaluate the efficiencies gained by hiring temporary staff or to look at how these workers impacted measures such as application processing time. In addition to modernization, Georgia has implemented a statewide work redistribution plan in fall 2010 to address an increase in caseloads and a decrease in available staff. Online COMPASS applications are typically shifted from high volume, urban areas like Atlanta to other parts of the state to better manage incoming work and to spread it more evenly across the state. While still in the early phases, staff interviewed reported increased efficiencies, more timely case processing, and increased customer service. Georgia does not currently calculate performance measures to compare the efficiencies gained by this project or see how work redistribution affects application time. One regional office also reported a recent change in their business model. The region closed local offices and consolidated staff into area hubs and workers are assigned a specific task within the application process. There are associated benchmarks for each role, and the regional office reviews worker production daily to evaluate whether they are meeting daily quotas. Measurement Goals: Georgia's performance measures touch on all four FNS goals. Georgia also uses measurement to ensure fair workload distribution among workers and for staffing purposes. Call Center: Georgia has a statewide call center operated by state staff. The state reviews the percent of changes processed, percent calls abandoned, and percent of total calls handled. They also collect many averages and counts of aggregate data. Online System: Georgia worked with a contractor to set up a statewide online application system, which has been in place since 2008. Document Imaging: Georgia has not established a document imaging system. Kiosks: Georgia has not set up any kiosks in state or local offices. Waiver of Face-to-Face Interview: One county office calculates the error rate for cases by type of interview (telephone versus face-to-face), total number of interviews, number of redetermination interviews, and number of home visit interviews. Another county office reported collecting the number of interviews by type (telephone versus face-to-face), number of applicants requesting a phone interview, and number of notices of missed interview mailed. Partnering: At the state level, Georgia has established outreach contracts with all partners that are based on application assistance. Partners are reimbursed for half of their outreach costs, so the more outreach that is provided the higher the partner's budget. The state mainly collects aggregate data counts on their partnering arrangements, in addition to the percent of partners providing application assistance and the average cost of partner arrangements. Two county offices also have informal partnerships in place. These partners have computers available for clients to access the online application system and provide staff to help clients navigate the system. These offices look at the number of partners and the number of partners providing application assistance. Measures Collected by Partners: The partner office we spoke with does not collect any performance data. Georgia (continued) A.57 Shortened Interviews: Information not provided. Online Expedited Applications: While the state office does not collect any measures related to online expedited applications, regional and local offices noted two measures that are available to them: number of expedited applications received online and number of expedited applications received online that were approved. Application Tracking: Georgia state and local offices collect a variety of application tracking measures and data, in the areas of application accuracy, approvals and denials, application processing and case characteristics, application receipt, and application processing time. Georgia is able to discern the origin of applications by whether they were filed online versus not online. Changes Over Time: Georgia noted no significant changes over the past year to the way measures are constructed or data are collected. One regional office used to be able to track applications submitted via partner agencies, but they stopped tracking these two years ago. Desired or Planned Future Measures: The Georgia state office is interested in computing the percent of applications received that are approved and examining these rates separately for online and paper submissions. Regional offices would like to track how applications were received (via mail, fax, online, etc.), percentage of people the office is not able to contact to whom they have to mail appointments, and number of interviews by type (phone versus face-to-face). One region wants to see improvements in the technology and computer system, giving them access to both higher quality and more usable data. For example, even though data are recorded on the waiver of the face-to-face interview, reports cannot be run on this initiative. Another office interviewed would like the computer system's monthly reports on the number of applications received by county to separate out the number of COMPASS applications. One local office would like data on the reasons for denials, and another local office would like more frequent reporting on expedite cases, since these applications are time sensitive. Seeing data from other states on the percent of applications by source (online versus paper) would be useful to Georgia. They would also like to see the number of expedited applications filed online and the average SNAP benefit allotment, broken down by case characteristics and demographics. A.58 Profile: Hawaii State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 0 Number of Partners Interviewed: 0 Initiatives Active in State: Waiver of Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Number of Redetermination Interviews Statewide Program Access, Efficiency, Customer Service No No Total Number of Interviews Statewide Program Access, Efficiency, Customer Service No No APPLICATION TRACKING (Statewide) Percent of Total Applications Approved Statewide Program Access, Efficiency, Customer Service No No Percent of Total Recertifications Approved Statewide Program Access, Efficiency, Customer Service No No Average Benefit Amount (for those eligible) Statewide Accuracy & Integrity, Customer Service No No Number of Applications Approved: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Recertifications Approved: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Applications: Paper Submission to Local Office Statewide Program Access, Efficiency, Customer Service No No Percent of Recertifications: Paper Submission to Local Office Statewide Program Access, Efficiency, Customer Service No No Total Number of Applications Started Statewide Program Access, Efficiency, Customer Service No No Total Number of Recertifications Started Statewide Program Access, Efficiency, Customer Service No No Total Number of Applications Completed Statewide Program Access, Efficiency, Customer Service No No Total Number of Recertifications Completed Statewide Program Access, Efficiency, Customer Service No No Hawaii (continued) A.59 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applications: Paper Submission to Local Office Statewide Program Access, Efficiency, Customer Service No No Number of Recertifications: Paper Submission to Local Office Statewide Efficiency, Customer Service No No Percent of Complete Applications Denied Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Complete Recertifications Denied Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Applications Denied: Paper Submissions to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Recertifications Denied: Paper Submissions to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Applicants that Appear Eligible for Expedited Benefits Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Average Processing Time of Application Decision Statewide Efficiency, Customer Service No No Notes: State collects mandatory accuracy data through QC reviews Description: General: Hawaii has a state-administered SNAP program. The state reported that they have two modernization initiatives and rarely use performance measures for purposes other than those required by FNS. Measurement Goals: Hawaii primarily measures performance to ensure accuracy and integrity, efficiency, and customer service. Call Center: Hawaii does not have a call center. Online System: Hawaii does not have an online system. Document Imaging: Hawaii does not have document imaging. Kiosks: Hawaii does not have kiosks. Waiver of Face-to-Face Interview: Hawaii tracks the number of initial eligibility interviews and redetermination interviews under this initiative. Partnering: Hawaii does not have a partnering initiative. Measures Collected by Partners: Not applicable. Hawaii (continued) A.60 Shortened Interviews: Hawaii does not have a shortened interview initiative. Online Expedited Applications: Hawaii does not have online expedited applications. Application Tracking: Hawaii's reported measures for application tracking were designed along with the system in 1988. Hawaii tracks a wide range of measures and aggregate data related to applications and recertifications, including the number and percent of applications and recertifications processed and the average benefit amount. In addition to FNS QC reviews, there is a supervisory review of one case per worker per month (with cases selected at random). Changes Over Time: Hawaii faces corrective action for timeliness problems, and there is a pending lawsuit that may force the state to make changes to the administration of the program. Staff reported that they were not meeting federal timeliness rates at the time of data collection. Desired or Planned Future Measures: State staff reported that Hawaii would like to track phone vs. in-person interviews in the future. A.61 Profile: Idaho State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 0 Number of Partners Interviewed: 0 Initiatives Active in State: Call Center, Document Imaging, Waiver of Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Statewide) Average Answer Speed Statewide Efficiency, Customer Service Not reported Not reported Average Hold Time Statewide Efficiency, Customer Service Not reported Not reported Percent Calls Abandoned Statewide Efficiency, Customer Service Not reported Not reported Average Call Duration Statewide Efficiency, Customer Service Not reported Not reported Average Waiting Time (to Speak to Agent) Statewide Efficiency, Customer Service Not reported Not reported Average Calls Handled Statewide Efficiency, Customer Service Not reported Not reported Number of Calls (in flow) Statewide Not reported Not reported Not reported Number of Calls Answered\/Handled Statewide Not reported Not reported Not reported Number of Agents Assigned to Phones Statewide Not reported Not reported Not reported Number of Changes Received Statewide Not reported Not reported Not reported Number of Changes Processed Statewide Not reported Not reported Not reported Number of Cases Pending Benefit Decision: Call Center Statewide Not reported Not reported Not reported Number of Applications Processed: Call Center Statewide Not reported Not reported Not reported WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Percent of Interviews Conducted by Telephone Statewide Not reported Not reported Not reported Percent that are Missed and have Notice Mailed Statewide Not reported Not reported Not reported Total Number of Interviews Statewide Not reported Not reported Not reported Number of Redetermination Interviews Statewide Not reported Not reported Not reported Number of Face-to-Face Interviews Statewide Not reported Not reported Not reported Idaho (continued) A.62 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Telephone Interviews Statewide Not reported Not reported Not reported Number of Missed Interviews Statewide Not reported Not reported Not reported Notice of Missed Interview Mailed Statewide Not reported Not reported Not reported DOCUMENT IMAGING (Statewide) Number of Documents Received Statewide Not reported Not reported Not reported Number of Documents Received: Mail Statewide Not reported Not reported Not reported Number of Documents Scanned Statewide Not reported Not reported Not reported Number of Tasks Completed Statewide Not reported Not reported Not reported APPLICATION TRACKING (Statewide) Percent of Total Applications Approved Statewide Not reported Not reported Not reported Average Processing Time of Application Decision Statewide Not reported Not reported Not reported Percent of Applicants that Appear Eligible for Expedited Benefits Statewide Not reported Not reported Not reported Average Processing Time of Recertification Decision Statewide Not reported Not reported Not reported Notes: State collects mandatory accuracy data through QC reviews. Description: General: In 2006, Idaho began a redesign of their eligibility process to address high error rates, timeliness problems, high staff workloads, and low staff morale. They developed a process-driven business design, through which applicants are not required to visit the office and eligibility workers attempt to touch the case just once before making an eligibility decision. Idaho states that the redesign decreased their case processing from 21 days to 1 hour and substantially improved their error rates. Measurement Goals: When designing performance measures, Idaho considers: 1) FNS expectations, 2) customer demand, and 3) operational effectiveness and efficiency. In addition to performance measurements, Idaho has compliance measurements to ensure that the state meets federal requirements. Formerly, Idaho's performance measurements were compliance-based, but did not enable the state to determine whether it was truly effective in terms of its own performance. The drive for new Idaho (continued) A.63 performance measurements originated out of a realization of the shortcomings of the compliance measurements. Call Center: Idaho focuses on performance-based or outcome measures. Although they collect measures such as average call and wait times, they do not use these to make management decisions. Online System: Idaho does not have an online application system. Document Imaging: Idaho has counts of the number of documents being processed, but does not use it as a measure of the initiative. They use it to project data storage requirements. Kiosks: Idaho does not have kiosks for application access and submission. Waiver of Face-to-Face Interview: Idaho tracks the percentage of interviews that are conducted by telephone and the percentage of interviews that are missed and have a notice mailed to them. Shortened Interviews: Idaho does not have a shortened interview initiative. Online Expedited Applications: Idaho does not have an online expedited application. Application Tracking: Idaho tracks the percentage of applications that are approved and denied. They measure accuracy for FNS through the QC system and supervisors also conduct internal reviews. Idaho recently changed their QA process to permit reviewers to correct and coach workers in real-time during the interview process. The state established a benchmark that 70 percent of applications be processed the day of receipt. Changes Over Time: Idaho has changed its measures substantially in order to support its process- driven business redesign. Their measurement focus is now on short-term outcomes for application processing. Desired or Planned Future Measures: Idaho currently calculates if the case was processed in one day based on when an application was created in the system and when it was processed. In addition, they would like to track the time between the interview date and the date an application was processed. Such a measure would enable Idaho to account for those applications where they cannot conduct the interview at the point of creation (for example, for mailed-in applications or for applicants who do not wish to wait for a same-day interview). Idaho also would like to be able to identify, in real-time, the workload across the state in order to distribute the work and improve efficiency. Partnering: Idaho prefers to have state employees conduct all work, including running the call center. However, they recently contracted with a partner to create a state-wide virtual contact center to help them achieve their goal of having a universal workforce. Measures Collected by Partners: We did not interview any partners in Idaho as part of this study. A.64 Profile: Illinois State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 0 Number of Partners Interviewed: 4 Initiatives Active in State: Call Center, Online System, Partnering, Waiver of Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Statewide) Average Answer Speed Statewide Customer Service No No Average Hold Time Statewide Customer Service No No Percent Calls Abandoned Statewide Not reported No No Average Call Duration Statewide Program Access, Efficiency No No Average Waiting Time (to Speak to Agent) Statewide Efficiency, Customer Service No No Average Calls Handled Statewide Efficiency, Customer Service No No Percent of Total Calls Handled Statewide Accuracy & Integrity, Efficiency, Customer Service No No Average Queue Time Statewide Not reported No No Average After-Call Work Time by Staff Statewide Accuracy & Integrity, Customer Service No No Percent of Queued Calls Handled Statewide Accuracy & Integrity, Efficiency, Customer Service No No Percent of Calls Completed in the Computer Phone System Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Calls Abandoned Statewide Customer Service No No Number of Calls Received Statewide Program Access, Efficiency, Customer Service No No Number of Calls Queued Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Calls Answered\/Handled Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Agents Assigned to Phones Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Illinois (continued) A.65 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Calls Receiving Busy Signal\/Unable to Connect with Phone System Statewide Customer Service No No ONLINE SYSTEM (Statewide) Number of applications submitted Statewide Program Access, Efficiency, Customer Service No No Percent Completed After Hours Statewide Program Access, Customer Service No No Number Providing Application Assistance Regional Program Access No No Number of Applications Received per Partner Regional Program Access No No Error Rate for Cases with Telephone Interviews Statewide Accuracy & Integrity No No Number of Telephone Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of interviews that are automated Statewide Customer Service No No APPLICATION TRACKING (Statewide) Number of Applications Approved: Community Partner Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Total Number of Applications Completed Statewide Program Access, Efficiency, Customer Service No No Total Number of Recertifications Completed Statewide Program Access, Efficiency, Customer Service No No Number of Applications: Online Statewide Program Access, Efficiency, Customer Service No No Number of Applications: Community Partner Statewide Program Access, Efficiency, Customer Service No No Number of Applications Denied: Community Partner Statewide Program Access, Efficiency, Customer Service No No Notes: State collects mandatory accuracy data through QC reviews. PSI = Phone System Interview Description: General: Illinois has a state-wide call center that SNAP applicants and participants can use to get some questions answered and submit limited types of changes\u2014more complicated questions must go Illinois (continued) A.66 through an eligibility worker. At recertification, many individuals are eligible for automated interviews. They work with a variety of partners that are funded through FNS. They participated in a demonstration called Express Stamps that allowed some food banks and their partners to complete shortened applications for their clientele. The demonstration ended in March 2011\u2014it is not being continued because of concerns identified in an evaluation related to accuracy and fraud. They continue to work with a variety of other partners that provide application assistance and outreach. Measurement Goals: They collect a variety of measures related to their call center, and also conduct mandatory reviews of cases with phone interviews to ensure their accuracy. Partners collect a variety of measures that are not required by the state. Call Center: Computer Telephony Assistance System (CTAS) is the software behind the call center and is completely automated. The call center number is centralized and has been in place since the 1980s for the all of the agency's programs including SNAP; TANF; Medicaid; drug, alcohol, and health prevention services; and programs for people with physical or developmental disabilities. CTAS is an automated system that can answer questions about benefit amount and case status. Although it resolves most of the calls that come in, small number of agents (caseworkers) are available to answer calls to field the remaining questions. Call center staff can make simple case changes, such as a change of address, but more complex changes, such as to reported income or household composition that affect the benefit amount, must be reported to the local office. CTAS also conducts Phone System Interviews (PSI) for recertifications only, although the local office determines eligibility. CTAS determines who is eligible for a PSI and sends the person a PIN to gain entry into the system and complete the PSI telephonically. Illinois has made numerous upgrades to the system architecture. The initial transition from the legacy system to CTAS was in November 2001, and the most upgrade was in September 2010. The initial version of CTAS did not allow for tracking call reason codes, but that capability was included in the most recent upgrade. The absence of these reason codes means that analysts have not had the ability to attribute call volume to specific agency programs. However, it is possible to track incoming calls to 1 of 15 toll-free numbers published by program. During these periods of upgrades, the performance measures calculations did not change. Online System: The online system was built by a contractor, and is maintained and modified by the state. Data entered into the online system are reviewed by a caseworker and then automatically transferred to the legacy system. Document Imaging: Illinois does not have document imaging. Kiosks: Illinois has not implemented kiosks. Waiver of Face-to-Face Interview: Illinois has an automated system for conducting the recertification by phone. Elderly and disabled individuals are not eligible for this automated interview, nor are individuals with complicated income calculations. These cases are too complex for the automated system to handle. However, these individuals may request a phone interview with their case worker. Shortened Interviews: Illinois does not have a separate shortened interview. Online Expedited Applications: Illinois does not have online expedited applications. Application Tracking: Statewide software that lets anyone check on the processing work of applications that have been submitted, allowing them to examine on-demand reports to look at trends. Illinois (continued) A.67 Changes Over Time: Illinois staff did not report major changes to measures over time. However, they reported that a recent system system change affects values observed for call center measures. Due to a staff reduction, a business decision was made to decrease the number of calls allowed in the queue by approximately 40 percent. Desired or Planned Future Measures: Illinois recognizes that the measures for the call center do not allow them to gauge how well the call system is meeting the needs of the callers or why callers abandoned calls. According to staff interviewed, they would like to be able to better identify when clients are struggling, such as how long clients spend on a given page of the application so that they could troubleshoot wording or functionality. These kinds of diagnostic tools are hard to add later when attention and resources have shifted elsewhere. They would also like to be able to compare processing time across states\u2014not just an indicator for timeliness or percent timely, but the average number of days to process applications in each state. Partnering: Illinois conducted a demonstration study, which ended in March 2011, called Express Stamps. The state partnered with several agencies, primarily foodbanks, who partnered with local agencies, such as food pantries, to assist individuals with applying for one to two months of benefits. Under the demonstration, applicants do not need to submit as much information, provide documentation, or participate in an interview. Eligibility is determined onsite. Illinois also has formal contracts with several other organizations for outreach and application assistance. Some outreach is targeted to individuals who are found to be participating in other programs, such as parents of students receiving free- and reduced-price school lunches, but not SNAP; other outreach efforts include attendance at fairs, school events, senior centers, et cetera. The partners provide information about SNAP and may also assist the applicants in completing the application process and submit the application for them by fax. The partners are paid for their services. They provide a report to the state discussing their activities. The state is collecting a variety of information for internal use, and would like the ability to determine the outcome of the applications that they have helped to submit. Currently, this is achieved by asking the applicant. Other partners providing SNAP information have informal agreements and are not paid for their services. Measures Collected by Partners: Partners we interviewed in Illinois commonly keep track of application outcomes, measuring either the number of applications approved or their approval rate. Other measures tracked by partners include: average level of benefits, number of applications submitted, number of clients contacted, number of screenings, and application origin (by ZIP code). A.68 Profile: Indiana State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 2 Number of Partners Interviewed: 3 Initiatives Active in State: Call Center, Online System, Document Imaging, Kiosks, Partnering, Waiver of Face\u2010to\u2010 Face Interview, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Regionally) Average Hold Time Locally Customer Service Not to exceed 3 minutes When metric not achieved for three consecutive months, a full-time employee will be added to this workgroup at full cost to the vendor Percent Calls Abandoned Locally Customer Service Not to exceed 7 percent for hybrid; not to exceed 5 percent for modern When metric not achieved for three consecutive months, a full-time employee will be added to this workgroup at full cost to the vendor Average Call Duration Locally Customer Service No No Average Waiting Time (to Speak to Agent) Locally Program Access, Customer Service 300 seconds (excludes internal call centers) When metric is not achieved for three consecutive months, a full-time employee will be added to this workgroup at full cost to the vendor Number of Calls (in flow) Locally None No No Number of Calls Abandoned Locally Efficiency, Customer Service No No Number of Calls Answered\/Handled Locally Efficiency, Customer Service No No Number of Calls Received Statewide Not reported No No Percent of Voicemail Calls Locally Not reported No No Indiana (continued) A.69 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives ONLINE SYSTEM (Regionally) Percent of Applications Received Locally Program Access, Efficiency, Customer Service No No Total Number of Applications Submitted per Month Locally Program Access, Efficiency, Customer Service No No DOCUMENT IMAGING (Statewide) Percent of Documents Received by Mail Statewide Program Access, Efficiency No No Percent of Documents Received by Fax Statewide Program Access, Efficiency No No Average Processing Time Statewide Efficiency 90 percent of documents without a barcode are processed within two business days of receipt. All documents are processed within three business days of receipt. When metric is not achieved for three consecutive months, a full-time employee will be added to this workgroup at full cost to the vendor Number of Documents Received Statewide Program Access, Efficiency No No Number of Documents Received: Mail Statewide Program Access, Efficiency No No Number of Documents Received: Fax Statewide Program Access, Efficiency No No Number of Documents Received: Other Electronic Statewide Program Access, Efficiency No No Number of Documents Processed Statewide Efficiency No No Number of Documents Scanned Statewide Efficiency No No KIOSKS (Regionally) None PARTNERING (Statewide) Number of Partners Statewide Efficiency, Customer Service No No Number Accepting Applications Regional Efficiency, Customer Service No No Indiana (continued) A.70 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number Providing Application Assistance Regional Efficiency, Customer Service No No WAIVER OF FACE-TO-FACE INTERVIEW (Regionally) None ONLINE EXPEDITED APPLICATIONS (Regionally) None APPLICATION TRACKING (Statewide) Percent of Total Applications Approved Statewide Efficiency, Customer Service No No Percent of Total Recertifications Approved Statewide Efficiency, Customer Service No No Percent of Applications: Online Statewide Program Access No No Percent of Applications: In-Office Statewide Program Access No No Percent of Applications: Mailed Statewide Program Access No No Number of Applications: Online Statewide Program Access No No Number of Applications: Mailed Statewide Program Access No No Notes: State collects mandatory accuracy data through QC reviews. Data are collected for all programs combined and cannot be separated into the separate programs. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Description: General: Indiana is divided into eight administrative regions and operates with three service delivery models. Characteristics of the three types include: \uf0b7 As-is: Staff process all applications on paper, and each worker has a specific caseload of cases for which she or he is responsible. Clients report changes directly to their caseworker. In January 2011, this model was in place in three of the eight regions, covering 52 percent of the caseload. \uf0b7 Modern: Both document imaging and online applications are in place. All interviews are by phone unless the client requests otherwise, using a two-step process where contractor staff gather data and then caseworkers conduct the interactive and eligibility determination portion of the interview. Indiana (continued) A.71 Contractors manage staff assigned to the phones. Cases are managed at the regional level, so work on a customer's case is not necessarily occurring in their county. In January 2011, this model was in place in two regions. \uf0b7 Hybrid. Both document imaging and online applications are in place, but client interviews occur in-person, using a one-step process with only a state merit employee. The interview takes place at the local county office unless the client requests a phone interview. Cases are pooled, and managers assign tasks to caseworkers as they need to be worked (no worker-specific caseloads are in place). Clients report changes to a regional center, and submit documents to a centralized location for scanning and indexing. State staff manage contractors assigned to answer phones in their local office. Work is managed at the county level, so most casework is done in the county where a customer lives, though it can be shifted as needed to balance the workload. In January 2011, this model was in place in three regions. The state planned to transition the modern regions to the hybrid model in February 2011 and hoped to convert all regions to the hybrid model by the end of 2011. Contractors run the call center, document imaging, outgoing mail, and systems development and maintenance. State merit staff continue to make all SNAP eligibility decisions, but almost everything else, including answering questions not related to eligibility and receiving client documents, has been outsourced. The state's system of service delivery has evolved over time and has shifted hands among contractors. This results in a range of measures and systems, some overlapping, that state staff and contractors use to manage their work. It also means that the history of measure development is somewhat fractured, and that some measures are being collected because former contractors programmed systems to collect them at an earlier time. Indiana has two systems, run by two contractors and used in different parts of the state, that assign tasks to workers based on an algorithm. One has more detail available for tracking status of cases, while the other allows workers to create form letters that are mailed centrally. Measurement Goals: At the time of our visit, Indiana's driving motivation for measurement was timeliness, for several reasons. First, requirements stemming from a lawsuit mandate that the state have a 90 percent timeliness rate by March 2011. Second, in order to get FNS approval for the waiver that would allow them to convert as-is regions to the hybrid model, they need to demonstrate their ability to satisfy timeliness goals. To meet the timeliness goals, they are able to look at several measures in real-time, and reassign staff if any of them are being overloaded or if there is a risk of not meeting timeliness standards. Call Center: The call center is run by a contractor and handles more than one program. It operates only in the modern and hybrid models. Calls are routed to the region and then, if a hybrid area, to the county within the region. Clients can report changes or request information through the call center. If no one answers the call within one minute during business hours, the caller can opt to leave a voicemail. After hours, all calls are routed directly to voicemail. Online System: Online applications are available in the hybrid and modern models. When an application comes in, a worker reviews it to verify it is complete, and then sends the application to the eligibility system to be registered and assigned a case number. Not all information is transferred over Partnering: Indiana has formal arrangements with a variety of contractors, providing application services; software development and support; mail, data entry, and document imaging services; and outbound mail and electronic mail services. Measures Collected by Partners: Because these are formal contracts, Indiana collects, or has the contractor collect, several measures to demonstrate their performance. In addition to the items mentioned in each initiative above, Indiana requires that contractors provide training and maintain the levels of accuracy and timeliness required by FNS. Indiana (continued) A.72 because client-entered information is subject to being both incomplete and incorrect. The system guides caseworkers through interview questions (skipping those that are irrelevant based on prior responses) while they enter additional client information. A contractor built and maintains the system, but state staff oversee the day-to-day operations. Document Imaging: The document imaging center operates out of a county service center and services the entire state. Once imaged, documents can be viewed by workers in any location. Documents may be scanned at local offices as well. The imaging software stores information about the origin of the document. All physical documents are shredded. Kiosks: Kiosks are available in regions operating with the modern model. Individuals can use these kiosks to access the application as they would through any other computer. The state intends to add kiosks to other regions in 2011. Waiver of Face-to-Face Interview: This waiver is used only in the modern model. Supervisors have access to discreetly monitor staff interview calls to monitor performance. Workers are prompted to record the type of interview. Shortened Interviews: Indiana does not have a shortened interview. Online Expedited Applications: In regions that allow online applications, Indiana also accepts expedited applications online. Application Tracking: The state uses the findings from the required Quality Control reviews to identify training opportunities for staff. As the result of a lawsuit, Indiana must process 90 percent of its applications and recertifications timely (higher than its internal goals of 80 percent and 85 percent, respectively). However, the state has a waiver that enables it to extend the processing time to 60 days in cases where the client has not, during the first 30 days, submitted sufficient documentation to make an eligibility decision. While most states would close out the application as ineligible after the first 30 days due to incomplete information, Indiana can keep it open for an additional 30 days to collect the missing information, without requiring the client to resubmit the entire application. A respondent from Indiana noted that the state's timeliness has improved since it increased efforts to track application processing. Changes Over Time: With the transition into the hybrid model across the state, Indiana will likely be relying on a different set of measures than they have been in the past. Desired or Planned Future Measures: None reported. A.73 Profile: Iowa State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 0 Number of Partners Interviewed: 0 Initiatives Active in State: Call Center, Online System, Document Imaging, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Statewide) Average Answer Speed Statewide Efficiency, Customer Service Not sure No Average Hold Time Statewide Efficiency, Customer Service Not sure No Percent Calls Abandoned Statewide Efficiency, Customer Service Not sure No Average Call Duration Statewide Efficiency, Customer Service Not sure No Average Waiting Time (to Speak to Agent) Statewide Efficiency, Customer Service Not sure No Average Calls Handled Statewide Efficiency, Customer Service Not sure No Percent of Total Calls Handled Statewide Efficiency, Customer Service Not sure No Average Staff on Phones Statewide Efficiency, Customer Service Not sure No Average Queue Time Statewide Efficiency, Customer Service Not sure No Average Time Prior to Abandonment Statewide Efficiency, Customer Service Not sure No Average After-Call Work Time by Staff Statewide Efficiency, Customer Service Not sure No Percent of Queued Calls Handled Statewide Efficiency, Customer Service Not sure No Number of Calls (in flow) Statewide Efficiency, Customer Service Not sure No Number of Calls Abandoned Statewide Efficiency, Customer Service Not sure No Number of Calls Queued Statewide Efficiency, Customer Service Not sure No Number of Calls Answered\/Handled Statewide Efficiency Not sure No Number of Agents Assigned to Phones Statewide Customer Service Not sure No Average Time to Process Change Statewide Efficiency, Customer Service Not sure No Number of Changes Received Statewide Efficiency Not sure No Iowa (continued) A.74 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Calls Received Statewide Efficiency, Customer Service Not sure No Number of Changes Processed Statewide Efficiency, Customer Service Not sure No Number of Calls Transferred to Agent Statewide Efficiency, Customer Service No No ONLINE SYSTEM (Statewide) None DOCUMENT IMAGING (Statewide) Number of Documents Received: Other Electronic Statewide Efficiency No No APPLICATION TRACKING (Statewide) Percent of Applications Denied Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure No Average Benefit Amount (for those eligible) Statewide Program Access, Accuracy & Integrity No No Percent of Total Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure No Total Number of Recertifications Approved Statewide Program Access Not sure No Total Number of Applications Approved Statewide Program Access Not sure No Total Number of Applications Statewide Program Access Not sure No Total number of Recertifications Statewide Program Access Not sure No Total Number of Applications Completed Statewide Program Access No No Total Number of Recertifications Completed Statewide Program Access No No Percent of Complete Applications Denied Statewide Program Access No No Percent of Complete Recertifications Denied Statewide Program Access No No Percent of Applications with Incomplete Information Statewide Program Access, Customer Service No No Iowa (continued) A.75 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Average Processing Time of Application Decision Statewide Program Access, Efficiency, Customer Service Federal regulations\/ state law No Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. Description: General: In Iowa, SNAP is state administered. Other than measures linked to the call center and application tracking, performance measurement in Iowa is limited. Measurement Goals: Iowa measures performance with a focus on customer service, accuracy and integrity, and program access. Accuracy measures conform to, but do not exceed, FNS quality control requirements. Call Center: All Iowa call center measures are automatically calculated by their Cisco call center software. The staff we interviewed were not familiar with the process used to design these measures. Online System: Iowa has an online application system maintained by a contractor. At the time of the interview, Iowa did not report any performance measures or aggregate data tied to this initiative. Document Imaging: At the time of the interview, Iowa's policy and documentation for the document imaging initiative was in the early revision stages. They had not yet established any measures (but one measure was planned). Kiosks: Iowa does not have kiosks for application access and submission. Waiver of Face-to-Face Interview: Iowa does not have a waiver of face-to-face interviews in place. Shortened Interviews: Iowa does not have a shortened interview. Expedited Online Applications: Iowa does not have expedited online applications. Application Tracking: Iowa tracks a number of measures for the application process, using data directly extracted from the state data system, CISCO. In addition to the measures noted, the respondent stated that cases are manually tagged with an expedited tag, to note those requiring expedited processing. Changes Over Time: Not reported. Desired or Planned Future Measures: Not reported. Partnering: Iowa does not have a partnering initiative. Measures Collected by Partners: Not applicable. A.76 Profile: Kansas State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 2 Number of Partners Interviewed: 0 Initiatives Active in State: Online System, Partnering, Waiver of Face\u2010to\u2010Face Interview, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives ONLINE SYSTEM (Statewide) None PARTNERING (Statewide) Number of Partners Statewide Program Access, Efficiency, Customer Service No No Number Accepting Applications Statewide Program Access, Efficiency, Customer Service No No Number Providing Application Assistance Statewide Program Access, Efficiency, Customer Service No No Number Clients Assisted per Partner Statewide Program Access, Efficiency, Customer Service The agencies set goals in their grant application No Number of Applications Received per Partner Statewide Program Access, Efficiency, Customer Service Meet agency goals set in their grant application No WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Total Number of Interviews Statewide Not reported Not reported Not reported Number of Redetermination Interviews Statewide Not reported Not reported Not reported ONLINE EXPEDITED APPLICATIONS (Statewide) None APPLICATION TRACKING (Statewide) Average Timeliness Rate: Approvals Statewide Efficiency, Customer Service 95 percent timely No Average Timeliness Rate: Recertifications Statewide Efficiency, Customer Service 95 percent timely No Percent of Applications: Online Locally Program Access Not reported Not reported Number of Applications: Paper Submission to Local Office Locally Program Access Not reported Not reported Kansas (continued) A.77 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Applications: Paper Submission to Local Office Locally Program Access Not reported Not reported Percent of Applications: Mailed Locally Program Access Not reported Not reported Number of Applications: Mailed Locally Program Access Not reported Not reported Total Number of Applications Completed Locally Not reported Not reported Not reported Total Number of Recertifications Completed Locally Not reported Not reported Not reported Percent of Complete Applications Denied Locally Not reported Not reported Not reported Percent of Complete Recertifications Denied Locally Not reported Not reported Not reported Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Description: General: Kansas has an online system and works with partners. They do not widely use measures outside of the partnering arrangements and QC reviews. Measurement Goals: Kansas primarily measures performance to ensure accuracy and integrity, efficiency and customer service. Call Center: Kansas does not have a call center. Online System: Data from the online system is not directly transferred into the eligibility system\u2014 staff print the application and enter the information manually. There is no state level performance tracking related to the online system. One region tracks the origin of their applications, including whether they are submitted online, but does so independently of the state. Document Imaging: Kansas has not implemented a document imaging system. Kiosks: Kansas does not have kiosks. Waiver of Face-to-Face Interview: The state did not report the collection of any measures related to the waiver, but one local office reported using state reports of the number of total and redetermination interviews to track its progress on interviews for this initiative. Kansas (continued) A.78 Shortened Interviews: Kansas does not use a shortened interview. Online Expedited Applications: Kansas has online expedited applications, but did not report tracking any performance data. Application Tracking: Kansas has little application tracking aside from required measures that are part of the quality control process. One exception is for timeliness the state tracks the number of days an application has been pending, and when it reaches 20 days, the application goes on a hotlist to alert a worker to take action on the case before it become out of compliance with the FNS 30-day standard. There is no parallel list for expedited applications. Staff in one local office reported that they look at a limited number of performance measures using state level data and reports. One region tracks the time it takes the staff in their seven offices to enter an application into the system once it is received. This is an independent effort by the region and does not occur statewide. Changes Over Time: None. Desired or Planned Future Measures: None. Partnering: Most partnering arrangements are informal and uncompensated in Kansas because of limited staff time. However, the state does track the performance of five formal outreach partners (those on their state SNAP outreach plan). These partners work closely with the state's local offices, but all contracts are maintained and negotiated on the state level. The state distributes applications pre-printed with partner ID numbers to each partner agency to use for tracking purposes. Tracking numbers only track the level of application supplies at each partner, but staff report they could be used for state level tracking in the future. However, because the computer system cannot track the origin of applications, all partner tracking in Kansas is manual. Measures Collected by Partners: We did not contact any of Kansas' partners as part of this study. The state does not require the partner to collect any data. A.79 Profile: Kentucky State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 2 Number of Partners Interviewed: 0 Initiatives Active in State: Online System, Partnering, Waiver of Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives ONLINE SYSTEM (Statewide) None PARTNERING (Statewide) None WAIVER OF FACE-TO-FACE INTERVIEW (Regionally) Total Number of Interviews Locally Not reported No No Error Rate for Cases with Telephone Interviews Locally Not reported Not reported Not reported Error Rate for Cases with Face-to-Face Interviews Locally Not reported Not reported Not reported Number of Applicants Requesting a Phone Interview Locally Not reported Not reported Not reported Number of Redetermination Interviews Locally Not reported Not reported Not reported Number of Missed Interviews Locally Not reported Not reported Not reported APPLICATION TRACKING (Statewide) Total Number of Applications Completed Statewide Not reported No No Total Number of Recertifications Completed Statewide Not reported No No Percent of Total Applications Approved Locally Not reported Not reported Not reported Percent of Total Recertifications Approved Locally Not reported Not reported Not reported Number of Applications Approved: Face-to-Face Interviewed Locally Not reported Not reported Not reported Number of Recertifications Approved: Face-to-Face Interviewed Locally Not reported Not reported Not reported Kentucky (continued) A.80 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applications Approved: Phone Interviewed Locally Not reported Not reported Not reported Number of Recertifications Approved: Phone Interviewed Locally Not reported Not reported Not reported Total number of Applications Approved Locally Not reported Not reported Not reported Total number of Recertifications Approved Locally Not reported Not reported Not reported Total Number of Applications Completed Locally Not reported No No Total Number of Recertifications Completed Locally Not reported No No Total Number of Applications Started Locally Not reported Not reported Not reported Total Number of Recertifications Started Locally Not reported Not reported Not reported Total Number of Applications Completed Locally Not reported Not reported Not reported Total Number of Recertifications Completed Locally Not reported Not reported Not reported Number of Total Applications Denied Locally Not reported Not reported Not reported Number of Total Recertifications Denied Locally Not reported Not reported Not reported Number of Cases with Complete Verification Submitted Locally Not reported Not reported Not reported Percent of Applications Approved Timely Locally Not reported Not reported Not reported Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Kentucky (continued) A.81 Description: General: In Kentucky, SNAP is state-administered. The two county offices interviewed reported that most performance measures are reviewed on a regional level. Measurement Goals: Kentucky did not report any goals for state-level measures. Call Center: Kentucky does not have a call center. Online System: Kentucky does not have an online system for application submission, but the state reported that an application in PDF form is available online and can be downloaded and printed for submission. Another web-based system can be used by clients to track their EBT balance. The state reported no measures for either of these online tools. Document Imaging: Kentucky does not have a document imaging initiative in place. Kiosks: Kentucky does not have kiosks for application access and submission. Waiver of Face-to-Face Interview: Kentucky has a waiver of face-to-face interviews in place. Although the state office did not report collecting any measures for this waiver, local offices reported tracking measures related to the volume of such interviews and the accuracy of cases in which the face-to-face interview had been waived. Shortened Interviews: Kentucky does not have shortened interviews. Online Expedited Applications: Kentucky does not have online expedited applications. Application Tracking: Kentucky tracks measures and aggregate data related to applications and recertifications, including the total number of completed, denials, and processing time. Kentucky previously won a timeliness prize, and they strive for 100 percent timeliness. Changes Over Time: Kentucky staff reported that they had made no changes in the last year to performance measurement, and one respondent noted that the application tracking system has not changed in several years . Desired or Planned Future Measures: At the time of the interview, Kentucky was working towards implementing a document imaging system. Partnering: While the state respondent indicated that there is no state-level initiative, a local respondent indicated two partners. One has a formal contract with the state office to provide SNAP nutrition assistance and outreach. Another works with the county office on an informal agreement, relying on a grant through the National Council on Aging to do Medicare and SNAP outreach. Measures Collected by Partners: We did not interview any Kentucky partners as part of this study. A.82 Profile: Louisiana State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 2 Number of Partners Interviewed: 0 Initiatives Active in State: Call Center, Online System, Kiosks, Waiver of Face\u2010to\u2010Face Interview, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Statewide) Average Answer Speed Statewide Not reported Not reported Not reported Average Hold Time Statewide Not reported Not reported Not reported Percent Calls Abandoned Statewide Not reported Not reported Not reported Average Call Duration Statewide Not reported Not reported Not reported Average Calls Handled Statewide Not reported Not reported Not reported Percent of Total Calls Handled Statewide Not reported Not reported Not reported Average Queue Time Statewide Not reported Not reported Not reported Average Time to De- Queue Statewide Not reported Not reported Not reported Average Time Prior to Abandonment Statewide Not reported Not reported Not reported Percent of Queued Calls Handled Statewide Not reported Not reported Not reported Number of Calls (in flow) Statewide Not reported Not reported Not reported Number of Calls Abandoned Statewide Not reported Not reported Not reported Number of Calls Queued Statewide Not reported Not reported Not reported Number of Calls Answered\/Handled Statewide Not reported Not reported Not reported Number of Calls Receiving Busy Signal\/Unable to Connect with Phone System Statewide Not reported Not reported Not reported Percent of Calls Completed in the Computer Phone System Statewide Not reported Not reported Not reported Number of Clients Accessing Computer Phone System Data Statewide Not reported Not reported Not reported Number of Calls Received Statewide Not reported Not reported Not reported Number of Hang-ups Statewide Not reported Not reported Not reported Louisiana (continued) A.83 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives ONLINE SYSTEM (Statewide) Average Number of Online Applications Statewide Efficiency No No KIOSKS (Statewide) Number of Users to Access Online Account Locally Not reported Not reported Not reported WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Notice of Missed Interview Statewide Program Access, Efficiency, Customer Service Not reported Not reported ONLINE EXPEDITED APPLICATIONS (Statewide) None APPLICATION TRACKING (Statewide) Percent of Total Applications Approved Statewide Program Access, Accuracy & Integrity, Customer Service No No Percent of Total Recertifications Approved Statewide Program Access, Accuracy & Integrity, Customer Service No No Percent of Complete Applications Denied Statewide Accuracy & Integrity, Customer Service No No Percent of Complete Recertifications Denied Statewide Accuracy & Integrity, Customer Service No No Average Processing Time of Application Decision Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Federal 30 day processing requirement. Cases meeting expedited criteria must be processed to enable household to access benefits within 4 days of application date. The state may qualify for incentive awards for timeliness of application processing. Average Processing Time of Recertification Decision Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Applications Approved After Paper Submission to Local Office Locally Not reported Not reported Not reported Louisiana (continued) A.84 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Recertifications Approved After Paper Submission to Local Office Locally Not reported Not reported Not reported Percent of Mailed Applications Approved Locally Not reported Not reported Not reported Percent of Mailed Recertifications Approved Locally Not reported Not reported Not reported Average Benefit Amount (for those eligible) Locally Not reported Not reported Not reported Number of Applications Approved: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Recertifications Approved: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Applications Approved: Mailed Locally Not reported Not reported Not reported Number of Recertifications Approved: Mailed Locally Not reported Not reported Not reported Number of Applications Approved: Face-to-Face Interviewed Locally Not reported Not reported Not reported Number of Recertifications Approved: Face-to-Face Interviewed Locally Not reported Not reported Not reported Number of Applications Approved: Phone Interviewed Locally Not reported Not reported Not reported Number of Recertifications Approved: Phone Interviewed Locally Not reported Not reported Not reported Percent of Applications Not Completed Locally Not reported Not reported Not reported Percent of Applications: Paper Submission to Local Office Locally Not reported Not reported Not reported Percent of Recertifications: Paper Submission to Local Office Locally Not reported Not reported Not reported Louisiana (continued) A.85 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Applications: Community Partner Locally Not reported Not reported Not reported Percent of Applications: Mailed Locally Not reported Not reported Not reported Percent of Recertifications: Mailed Locally Not reported Not reported Not reported Total Number of Applications Started Locally Not reported Not reported Not reported Total Number of Applications Completed Locally Not reported Not reported Not reported Number of Applications: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Recertifications: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Applications: Community Partner Locally Not reported Not reported Not reported Number of Applications: Kiosk Locally Not reported Not reported Not reported Number of Applications: Mailed Locally Not reported Not reported Not reported Number of Recertifications: Mailed Locally Not reported Not reported Not reported Number of Applications Denied: Mailed Locally Not reported Not reported Not reported Number of Recertifications Denied: Mailed Locally Not reported Not reported Not reported Percent of Initial Applications Denied Locally Not reported Not reported Not reported Percent of Recertifications Denied Locally Not reported Not reported Not reported Number of Applications Denied: Paper Submissions to Local Office Locally Not reported Not reported Not reported Number of Recertifications Denied: Paper Submissions to Local Office Locally Not reported Not reported Not reported Percent of Applications with Incomplete Information Locally Not reported Not reported Not reported Louisiana (continued) A.86 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Recertifications with Incomplete Information Locally Not reported Not reported Not reported Percent of Applicants that Appear Eligible for Expedited Benefits Locally Not reported Not reported Not reported Number of Cases with Complete Verification Submitted Locally Not reported Not reported Not reported Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Description: General: In Louisiana, SNAP is state-administered. The two local offices interviewed reported that most performance measures are reviewed on a regional level. Measurement Goals: Louisiana primarily measures performance to ensure customer service, accuracy and integrity, and efficiency. Call Center: Louisiana has an interactive voice response system that permits clients to access basic information (benefit level, application status, office locations, etc.) A contractor operates the system. Louisiana residents also have access to Disaster SNAP through a call center that transfers callers to live agents when requested. Online System: At the time of the interview, Louisiana was in the first month of the official rollout of their online system. They track the daily number of online applications submitted. A contractor developed the online system with state staff. That contract has expired and the state is trying to acquire a maintenance contract to assist their programmers with system changes. Staff reported that the information uploaded to the online system does not currently transition to the eligibility system; a state worker manually enters all information from the online applications system into the eligibility system. Document Imaging: Louisiana does not have a document imaging initiative in place. Kiosks: Kiosks located in the lobby of SNAP offices provide access to the online application for download and submission. Applications that have already been submitted may not be changed at the kiosk. If a client has not yet submitted an online application, an assigned pin number allows them to make changes to their information for up to 30 days. Waiver of Face-to-Face Interview: The state does not collect measures for their waiver of face-to- face interviews (though they do record when a notice of missed interview has been mailed), but Louisiana Partnering: Louisiana does not have a partnering initiative. Measures Collected by Partners: Not applicable. Louisiana (continued) A.87 can request aggregate data on the number of missed interviews. The state SNAP office does not request or use that data on a regular basis. Shortened Interviews: Louisiana does not have a shortened interview initiative. Online Expedited Applications: Louisiana has an online expedited application initiative, but the state does not collect any measures about this initiative. Application Tracking: Louisiana must report measures for application tracking in accordance with FNS national regulations. According to a Louisiana SNAP staff member, \"I've been involved for 32 years, and we've always had those performance measures. This is nothing new.\" Louisiana has automatic monthly programming that produces reports based on these data. In addition to FNS quality control, Louisiana has a case review system utilized by regional program coordinators (or \"regional specialists\") and first-line supervisors. The case review system is another type of corrective action. Regional specialists will use those case review cases to conduct re-reviews and identify error trends by region, parish, worker, or unit. Louisiana recently began incorporating these error trends into the management evaluation review, another corrective action tool. State staff reported that this case review system uses a more comprehensive sample than FNS quality control, but the sample quality depends upon the quality of first-line supervisors' reviews. Louisiana's case review system does not calculate error rates and does not use or review the information beyond FNS QC guidelines. Changes Over Time: Not reported. Desired or Planned Future Measures: Louisiana is in the process of developing more community partnerships and assigning numbers to each of the community partners and their sites. State staff reported that they plan to track which applications came from which site. Louisiana anticipates having many more modernization initiatives underway in 2011. A.88 Profile: Maine State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 2 Number of Partners Interviewed: 0 Initiatives Active in State: Online System (active January 2011), Partnering, Waiver of Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives ONLINE SYSTEMa (Statewide) Average Number of Screenings Started: Online Statewide Accuracy & Integrity Not reported Not reported Average Number of Screenings Completed: Online Statewide Accuracy & Integrity Not reported Not reported Percent of Screened Individuals Potentially Eligible: Online Statewide Accuracy & Integrity Not reported Not reported Percent of Applications Received Statewide Accuracy & Integrity Not reported Not reported Average Number Submitted per Month Statewide Accuracy & Integrity Not reported Not reported Average Number Started per Month Statewide Accuracy & Integrity Not reported Not reported Percent of Multiple Applications: Online Statewide Accuracy & Integrity Not reported Not reported Average Time to Complete Online Application Statewide Accuracy & Integrity Not reported Not reported Average Time to Complete Online Screening Statewide Accuracy & Integrity Not reported Not reported PARTNERINGb (Statewide) None WAIVER OF FACE-TO-FACE INTERVIEWb (Statewide) None APPLICATION TRACKING (Statewide) Total Number of Applications Completed Statewide Accuracy & Integrity No No Number of Applications Denied: Total Statewide Accuracy & Integrity No No Number of Applications Approved: Total Statewide Accuracy & Integrity No No Maine (continued) A.89 Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. aThe call center was not in place at the time of our interview and data collection. The measures provided here are the measures the state planned to collect. bThe initiative was under development at the time of our interview and measures were not yet in place. Description: General: Maine has an integrated benefit delivery system, the Automated Client Eligibility System (ACES). It integrates Medicaid, TANF, ASPIRE (Maine's employment and training program), Food Supplement (Maine's SNAP), childcare, and child support. At the time of our interview, Maine was preparing their online system, which was to be launched in January 2011. They system was planned to tie in directly into ACES. They were also working with community partners to plan how partners would be able to access the system to provide application assistance. Conversations were underway with FNS about a waiver for the face-to-face interview. Measurement Goals: Maine plans to track a variety of measures related to their online system. They currently track the counts of applications approved and denied. The goals for tracking these measures are to ensure continued accuracy and integrity of the program. Call Center: Maine does not have a call center. Online System: The online system will have a screening tool and application\u2014unsigned applications can be submitted online. The online system was planned to tie directly into ACES. Document Imaging: Maine has not implemented a document imaging system. Kiosks: Maine does not have kiosks. Waiver of Face-to-Face Interview: The waiver was under discussion at the time of our data collection. Shortened Interviews: Maine does not have a shortened interview process. Online Expedited Applications: Maine does not have online expedited applications. Application Tracking: Maine tracks the counts of applications submitted, approved, and denied. Changes Over Time: None reported. Desired or Planned Future Measures: Maine plans to track a variety of measures related to their online system, and will track the use of the system by partners as well. Partnering: Several community-based organizations have paper copies of the application available. Maine plans to give community partners access to the online system. Measures (to be) Collected by Partners: When partners have access to the online system, Maine anticipates collecting aggregate data with the number of people accessing the screening tool, number of people applying, and number of people granted benefits. The state will also track if an individual applied on their own or received assistance from the community partner. A.90 Profile: Maryland State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 0 Number of Partners Interviewed: 3 Initiatives Active in State: Online System, Partnering, Waiver of Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives ONLINE SYSTEM (Statewide) Number of Online Screenings Started: Online Statewide Customer Service No No Number of Online Screenings Completed: Online Statewide Program Access, Customer Service No No Number of Applications Started Statewide Customer Service No No Number of Abandoned Applications Statewide Customer Service No No Number of Applicants who Received Help to Apply Online Statewide Program Access, Customer Service No No Number of Applicants who Did Not Receive Help to Apply Online Statewide Customer Service No No PARTNERING (Statewide) Number of Partners Statewide Program Access, Customer Service No No Number Accepting Applications Statewide Program Access, Customer Service No No Number Providing Application Assistance Statewide Program Access, Customer Service No No WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Total Number of Interviews Statewide Program Access, Efficiency, Customer Service No No APPLICATION TRACKING (Statewide) Number of Applications Approved: Phone Interviewed Statewide Efficiency, Customer Service No No Number of Recertifications Approved: Phone Interviewed Statewide Efficiency, Customer Service No No Total Number of Applications Completed Statewide Program Access No No Maryland (continued) A.91 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Total Number of Recertifications Completed Statewide Program Access No No Number of Applications: Online Statewide Program Access, Efficiency, Customer Service No No Number of Recertifications: Online Statewide Program Access, Efficiency, Customer Service No No Percent of Complete Applications Denied Statewide Program Access No No Average Processing Time of Application Decision Statewide Efficiency, Customer Service No No Timeliness Rate Statewide Efficiency, Customer Service 90 percent to avoid corrective compliance with the USDA; 96 percent or better to comply with the Thompson v. Donald court order The local department directors are evaluated on this measure. Notes: State collects mandatory accuracy data through QC reviews. Description: General: Maryland has an online system, supported by a contractor, for application submission and change reporting. Maryland also has numerous community partners providing application assistance\u2014 some formal and some informal. Measurement Goals: Many of Maryland's measurement goals are related to program access and customer service. State staff reported that they would like to be able to track more information about the approvals and denials for their community partners. Call Center: Maryland does not have a call center. Online System: Maryland operates an online system, Service Access and Information Link (SAIL), which they own, but contract with another company to maintain. It can be used for screening, application submission (with electronic signature), and to report changes. When partners log in using their specific login IDs, the system is able to track the applications originating with the partners. Applicants can create an account and return to it at a later time via use of a control number issued when the account is created. These accounts remain active indefinitely. Once an application is filed online, information from the online application (with the exception of a narrative section) transfer to the eligibility system through a subsystem that brings up screens resembling the legacy system. The screens in the legacy system are pre-populated with information from the online application (alongside any pre-existing information about the client from the legacy system). Information from the online application appears in a different color. The worker compares any contradictory information and determines which to keep before submitting the information to the legacy system. The legacy system maintains an indicator noting which applications Maryland (continued) A.92 were filed online. The narrative section of the application may be printed from SAIL; soon, caseworkers will be able to do so from the eligibility system. With regard to measures, Maryland staff noted that the counts of the total number of screenings and total number of applications started include counts from those who may not be serious about applying\u2014 without a way to track why people abandon the screener or the application, they are limited in what they can learn from the measures. Document Imaging: Maryland does not have document imaging. Kiosks: Maryland does not have kiosks. Waiver of Face-to-Face Interview: Maryland tracks whether an interview occurred via telephone, mail-in, or group intake, so with a special query, they can report the percentage of interviews from each source but this is not regularly reported. They could also examine case characteristics by interview type using a special query. They track when an applicant has a language barrier. Shortened Interviews: Maryland does not have a shortened interview initiative. Online Expedited Applications: No information reported. Application Tracking: When examining application accuracy, Maryland tracks error codes in addition to those required by FNS when examining application accuracy. They use the quality control reviews to identify training opportunities. Changes Over Time: None reported. Desired or Planned Future Measures: Maryland would like to systematically track the approvals and denials of applications submitted by partners, which they can currently only do on an ad hoc basis. They also indicate that with online applications, having a national database of participants would assist them in preventing individuals from receiving benefits in multiple states, without waiting for the quarterly review currently conducted. Partnering: Maryland has seven formal outreach partners receiving compensation and several informal partners not receiving compensation. Among formal partners, two provide assistance with paper submission of applications (delivering them to a local office) and five provide assistance with submitting applications through SAIL. They are compensated for their expenditures toward assisting SNAP clients. Informal partners provide assistance with submitting applications through SAIL. They must sign a confidentiality agreement. Partners do not have special access to SAIL, but they do enter identification codes so that the application can be tracked back to their organization. Maryland also partners with a contractor to maintain and support SAIL. Measures Collected by Partners: The contractual arrangements that Maryland makes with partners do not require a specific level of application submissions or approvals. The company maintaining the on- line system must ensure the system is operational 99.3 percent of the time. They report additional system measures, such the system response time. For their own purposes, they collect the measures at a finer level of detail than required by the state. Some of the outreach partners also set their own goals for the number of applications submitted, even though the state does not set a minimum level of assistance. A.93 Profile: Massachusetts State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 5 Number of Partners Interviewed: 3 Initiatives Active in State: Call Center, Online System, Partnering, Waiver of Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Regionally) Number of Applications Processed: Call Center Locally None Not reported Not reported ONLINE SYSTEM (Statewide) Average Number of Screenings Completed: Online Statewide Program Access, Efficiency, Customer Service No No Average Number of Screenings Started: Online Statewide Program Access, Customer Service No No Average Number Started per Month Statewide Program Access, Efficiency, Customer Service Yes, not specified No Average Time to Complete Screening Statewide Program Access, Efficiency, Customer Service No Not sure Number of Abandoned Applications Statewide Efficiency, Customer Service No Not sure Number of Applications Started Statewide Program Access, Efficiency, Customer Service Not sure Not sure Percent of First-Time Applications Approved Statewide Program Access, Efficiency, Customer Service Minimum of 50 percent No Percent of First-Time Applications Denied Statewide Program Access, Efficiency, Customer Service 40 percent No Number of Online Accounts Created Statewide Program Access, Efficiency, Customer Service No Not sure Percent of Applications Received Statewide Program Access, Efficiency, Customer Service 40 percent No Percent of Multiple Applications Statewide Efficiency No No Massachusetts (continued) A.94 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives PARTNERING (Statewide) Percent of Partners Providing Application Assistance Statewide Program Access, Efficiency This measure should always be at 100 percent Partners receive set amount per application they submit on behalf of their clients if the application got approved for SNAP benefits Number of Partners Statewide Program Access, Efficiency, Customer Service No No Number Accepting Applications Statewide Program Access, Efficiency, Customer Service Reimburse- ments based solely on number of applications submitted\/ approved Cost reimburseme nt Number Providing Application Assistance Statewide Program Access, Efficiency, Customer Service No Not sure Number Clients Assisted per Partner Statewide Program Access, Efficiency, Customer Service Yes, agencies must submit a certain number of applications per month\/ quarter (varies based on agency size and resources) Yes, reimburseme nt to these agencies based solely on the number of applications they submitted and were approved Number of Applications Received per Partner Statewide Program Access, Efficiency, Customer Service Agencies must submit a certain number of applications per month\/ quarter (varies based on agency size and resources) Financial incentives (not specified) WAIVER OF FACE-TO-FACE INTERVIEWa (Statewide) Percent of Interviews Conducted by Telephone Statewide Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Percent of Interviews Conducted Face-to-Face Statewide Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Massachusetts (continued) A.95 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Error Rate for Cases with Telephone Interviews Statewide Accuracy & Integrity Yes, Annual performance goals set with USDA Yes, Annual performance goals set with USDA Error Rate for Cases with Face-to-Face Interviews Statewide Accuracy & Integrity Yes, Annual performance goals set with USDA Yes, Annual performance goals set with USDA Total Number of Interviews Statewide Accuracy & Integrity, Efficiency, Customer Service No No Number of Redetermination Interviews Statewide Accuracy & Integrity, Efficiency, Customer Service State has a waiver for certain elderly\/ disabled cases at recertifica- tion Annual performance goals set with USDA Number of Face-to-Face Interviews Statewide Accuracy & Integrity, Efficiency, Customer Service No No Number of Telephone Interviews Statewide Accuracy & Integrity, Efficiency, Customer Service No No APPLICATION TRACKING (Statewide) Number of Applications Approved: Mailed Statewide Accuracy & Integrity, Efficiency, Customer Service Generally expected at 70 percent Not sure Number of Recertifications Approved: Mailed Statewide Accuracy & Integrity, Efficiency, Customer Service No Not sure Number of Applications Approved: Face-to-Face Interviewed Statewide Accuracy & Integrity, Efficiency Not sure Not sure Percent of Community Partner Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Partners are paid for approved applications Partners are paid for approved applications so there is financial incentive to improve this percentage Percent of Applications Approved After Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Generally expected at 70 percent Not sure Percent of Recertifications Approved After Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Massachusetts (continued) A.96 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Online Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Expected minimum of 50 percent Participating agencies receive financial incentive for submitting approved applications Percent of Mailed Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Generally expected at 70 percent No Percent of Mailed Recertifications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service 70 percent is generally expected No Percent of Total Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Yes, 70 percent is generally expected No Percent of Total Recertifications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Average Benefit Amount (for those eligible) Statewide Accuracy & Integrity No No Number of Applications Approved: Paper Submission to Local Office Statewide Accuracy & Integrity, Efficiency, Customer Service No Not sure Number of Recertifications Approved: Paper Submission to Local Office Statewide Accuracy & Integrity, Efficiency, Customer Service No No Number of Applications Approved: Online Statewide Accuracy & Integrity, Efficiency, Customer Service 50 percent minimum (for participating agencies) Financial incentives for outreach participating agencies Number of Applications Approved: Community Partner Statewide Accuracy & Integrity, Efficiency 50 percent minimum (for participating agencies) Financial incentive for participating agencies Number of Applications Approved: Faxed Statewide Accuracy & Integrity, Efficiency, Customer Service Not reported Not sure Number of Recertifications Approved: Face-to-Face Interviewed Statewide Accuracy & Integrity, Efficiency No Not sure Number of Applications Approved: Phone Interviewed Statewide Accuracy & Integrity, Efficiency, Customer Service No Not sure Massachusetts (continued) A.97 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Recertifications Approved: Phone Interviewed Statewide Accuracy & Integrity, Efficiency, Customer Service No Not sure Percent of Applications: Online Statewide Program Access, Efficiency, Customer Service Goal set at 40 percent No Percent of Applications: Community Partner Statewide Program Access, Efficiency, Customer Service No Some community partners will be reimbursed for costs associated with application assistance Total Number of Applications Started Statewide Program Access, Efficiency, Customer Service No No Total Number of Recertifications Completed Statewide Accuracy & Integrity, Efficiency, Customer Service No No Number of Applications: Online Statewide Program Access, Efficiency, Customer Service Goal is set for 40 percent (of total applications received) No Number of Applications: Community Partner Statewide Program Access, Efficiency, Customer Service No No Percent of Online Applications Denied for Failure to Submit Documentation Statewide Accuracy & Integrity, Customer Service No No Percent of Online Recertifications Denied for Failure to Submit Documentation Statewide Accuracy & Integrity, Customer Service No No Percent of Online Applications Denied for Failure to Complete Interview Statewide Accuracy & Integrity, Customer Service No No Percent of Initial Applications Denied Statewide Accuracy & Integrity, Customer Service No No Number of Applications Denied: Online Statewide Program Access, Accuracy & Integrity, Customer Service Generally expected to be less than 50 percent No Number of Applications Denied: Community Partner Statewide Program Access, Accuracy & Integrity, Customer Service Generally expected to be less than 50 percent No Massachusetts (continued) A.98 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Average Processing Time of Application Decision Statewide Efficiency, Customer Service Goal set for 17 days average No Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. BEACON is the Massachusetts state eligibility system. aThe Massachusetts waiver of face-to-face interviews cannot have a detrimental effect on timeliness and accuracy. This is reflected in their standards. Description: General: The Massachusetts Department of Transitional Assistance (DTA) reported that they lead all initiatives and performance measurement, so there is much standardization across all offices. Two exceptions are the call center and waiver of face-to-face interviews. The call center has been set up in two local offices only, one of which tracks performance data on it. The waiver of face-to-face interviews has not yet been established in one local office. The state office generates reports and distributes them to the local offices, and offices can also run ad-hoc queries as needed. Staff reported that Massachusetts is generally satisfied with the quality and accuracy of their data. They reported working carefully on their data warehousing system so that most data they need can be meaningfully retrieved or compiled through it. They noted some concerns on the accuracy of the waiver of face-to-face interview measures, due to staff error and the difficulty of finding someone with the business knowledge and technical skills to run the required reports, and timely processing measures, because a case cannot be denied until day 30 even if it is clearly ineligible. It can also be difficult to code the type of application (brand new, recertified, expedited benefits, reinstated), which makes approval rates and recertification measures harder to accurately capture. Measurement Goals: Massachusetts' performance measures touch on all FNS goals for modernization. Their offices also use data for management purposes, staffing and workflow, addressing operational issues, reporting for researchers and legislature, and FNS or Executive Office of Health and Human Services (EOHHS) requirements. Call Center: The state worked with a contractor to establish two local call centers. One office collects the number of applications processed at their call center, but Massachusetts does not track any performance data on call centers beyond this. Because the offices received money to establish the call centers, the state had been requiring a greater level of reporting. They have not used call center data in the past two years. Online System: The online application was designed and built by a contractor to incorporate multiple benefit programs in the state. By default, the system can produce various reports that show the online system measures. The state office collects measures on the percent of applications received, percent of first-time applications approved\/denied, and percent of multiple applications. They also look at averages and counts of aggregate data, such as average number of screenings started\/completed, average time to complete screening, and average number of applications started\/submitted per month. Document Imaging: Massachusetts has not implemented document imaging. Massachusetts (continued) A.99 Kiosks: Neither Massachusetts state nor local offices participating in the study have set up kiosks for application access and submission. Waiver of Face-to-Face Interview: The Massachusetts state office reports the percent of interviews conducted by type (face-to-face and telephone) and the error rate for cases by type (face-to-face and telephone), in addition to collecting some aggregate data. Shortened Interviews: There is no shortened interview initiative in Massachusetts. Online Expedited Applications: Massachusetts reported that expedited applications can be submitted online, but did not provide us with further information. Application Tracking: The Massachusetts state office has many measures and aggregate data on application accuracy, approvals\/denials, application receipt, and processing time. The state records the origin of applications by their source on hard copy applications and within their eligibility system. They break down origin into the following categories: walk-in, online (via partner or consumer), fax, mail or other (such as a home visit). One local office we interviewed tracks applications by origin but they do not have any performance measures or data. Massachusetts staff noted that it is difficult to clearly distinguish new applications from recertifications from the system perspective, and that they are currently working on making this easier. In the past there were three indicators: was the case previously active, is the case currently active, is the application a reinstated case. A combination of these indictors is used to determine if the application is new. These questions are not asked in the application process, so the information must be obtained through the system instead. Changes Over Time: Massachusetts began recording the type and number of interviews provided to applicants about one and a half years ago. Staff reported that no other measures have changed in the past year. Partnering: All partnering arrangements are coordinated out of the state office. Massachusetts staff reported that both formal partnership (that involves federal FFP funding channeled back to partnering agencies) and informal partnership (that does not involve funding) initiatives are in place. Massachusetts tracked the actual expenditure data for eight partnering agencies participating in their outreach initiative during the first quarter of 2010. The state reported that they look at the percent of partners providing applications assistance. They also review the number of partners, number of partners accepting applications, number of partners providing application assistance, number of clients assisted per partner, and number of applications received per partner. Measures Collected by Partners: At the time of the interview, the state office was designing partnering contracts. They planned to track percent of community partner applications approved and set a benchmark of 50 percent approval rate for submissions within the first year. This will be tracked by the state, rather than the partner. Partners will not be required to collect any performance data beyond what can be accessed automatically from the Virtual Gateway. Massachusetts partners reported collecting data on: percent eligible and not eligible for prescreens; percent eligible and not eligible for application assistance; percent of screenings eligible and not eligible; percent of applications approved and denied; percent of applications submitted by origin (online versus paper); number of household approved for SNAP benefits; and number of applications submitted by site. Massachusetts (continued) A.100 Desired or Planned Future Measures: The state is in the process of updating their eligibility system and the data warehouse. This will facilitate presentation of the data and streamline calculations. They are also working on a partnering outreach initiative this year. Under this program they will use federal money to reimburse non-profit organizations for approximately half of their application assistance costs. This program will also set up benchmarks and incentives related to approvals for applications submitted via partners. The state plans to revise their method for tracking application source, in order to better track which agencies are submitting applications. One local office would be interested in seeing performance measures (percentages) and aggregate data (numbers) on the approval rate and reason for incomplete and denied applications, in an effort to decrease their denial rate. Another local office would like to see all states report on the length of time to process applications and the percent of recertifications processed timely, in order to learn more about states' ability to process work. A.101 Profile: Michigan State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 1 Number of Partners Interviewed: 1 Initiatives Active in State: Online System, Kiosks, Partnering, Waiver of Face\u2010to\u2010Face Interview, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives ONLINE SYSTEM (Statewide) Number Submitted per Day Statewide Program Access, Efficiency, Customer Service No No Average Number of Accounts Created per Month Statewide Efficiency, Customer Service No No Average Number of Changes Submitted per Month Statewide Efficiency, Customer Service No No Percent of Screened Individuals Potentially Eligible: Online Statewide Not reported Not reported Not reported Percent of Screenings Resulting in Application Submission: Online Statewide Not reported Not reported Not reported Percent of Screenings Resulting in Applicant Receiving Benefits: Online Statewide Not reported Not reported Not reported Number of Online Screenings Started: Online Statewide Not reported Not reported Not reported Average Number of Screenings Started: Online Statewide Not reported Not reported Not reported Average Number of Screenings Completed: Online Statewide Not reported Not reported Not reported Percent of Users Selecting Spanish Language Statewide Not reported Not reported Not reported Number of Online Screenings Completed: Online Statewide Not reported Not reported Not reported Number of Applications Downloaded Statewide Not reported Not reported Not reported Number of Applications with Filing Date Only Statewide Not reported Not reported Not reported Number of Applications Submitted With All Application Questions Answered Statewide Not reported Not reported Not reported Michigan (continued) A.102 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Applications Received Statewide Not reported Not reported Not reported Average Number Started per Month Statewide Not reported Not reported Not reported Average Number Submitted per Month Statewide Not reported Not reported Not reported Percent of Applicants with Hardship Reason for Requesting Phone Interviews Statewide Not reported Not reported Not reported Average Level of Benefits Statewide Not reported Not reported Not reported Average Number of Changes Submitted per Month Statewide Not reported Not reported Not reported Number of Online Accounts Created Statewide Not reported Not reported Not reported Number of Changes Submitted Statewide Not reported Not reported Not reported KIOSKS (Statewide) None PARTNERING (Statewide) Number with Terminals\/Kiosks to Submit Application Statewide Program Access, Efficiency, Customer Service No No Number of Clients Receiving Application Assistance Statewide Program Access, Efficiency, Customer Service No No Number of Partners Statewide Program Access, Efficiency, Customer Service No No Number Accepting Applications Statewide Program Access, Efficiency, Customer Service No No Number Providing Application Assistance Statewide Program Access, Efficiency, Customer Service No No Number of Applications Received per Partner Statewide Not reported Not reported Not reported WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Percent of Interviews Conducted by Telephone Statewide Program Access, Efficiency, Customer Service No No Percent of Interviews Conducted Face-to-Face Statewide Accuracy & Integrity No No Percent of Telephone Interview Requests Honored Statewide None No No Michigan (continued) A.103 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Total Number of Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Redetermination Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Face-to-Face Interviews Statewide Program Access, Accuracy & Integrity, Customer Service No No Number of Telephone Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Home Visit Interviews Statewide Program Access, Accuracy & Integrity, Customer Service No No Number of Missed Interviews Statewide Program Access, Accuracy & Integrity, Customer Service No No Notice of Missed Interview Mailed Statewide Program Access, Accuracy & Integrity, Customer Service No Yes Percent that are Missed and have Notice Mailed Locally Not reported Not reported Not reported Error Rate for Cases with Telephone Interviews Locally Not reported Not reported Not reported Error Rate for Cases with Face-to-Face Interviews Locally Not reported Not reported Not reported Number of Applicants Requesting a Phone Interview Locally Not reported Not reported Not reported Number of Applicants Requesting an In-Person Interview Locally Not reported Not reported Not reported ONLINE EXPEDITED APPLICATIONS (Statewide) Percent of Applications Approved: Online Expedited Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Applications Approved: Online All Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Expedited Applications Received Online Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Applications Approved: Online Expedited Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Michigan (continued) A.104 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applications Approved: Online All Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure APPLICATION TRACKING (Statewide) Percent of Online Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Total Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Average Benefit Amount (for those eligible) Statewide Accuracy & Integrity Not sure Not sure Number of Recertifications Approved: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Customer Service Not sure Not sure Number of Applications Approved: Online Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Percent of Applications Approved After Paper Submission to Local Office Statewide Not reported Not reported Not reported Percent of Recertifications Approved After Paper Submission to Local Office Statewide Not reported Not reported Not reported Percent of Mailed Applications Approved Statewide Not reported Not reported Not reported Percent of Mailed Recertifications Approved Statewide Not reported Not reported Not reported Percent of Total Recertifications Approved Statewide Not reported Not reported Not reported Number of Applications Approved: Paper Submission to Local Office Statewide Not reported Not reported Not reported Number of Applications Approved: Mailed Statewide Not reported Not reported Not reported Number of Recertifications Approved: Mailed Statewide Not reported Not reported Not reported Number of Applications Approved: Face-to-Face Interviewed Statewide Not reported Not reported Not reported Michigan (continued) A.105 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Recertifications Approved: Face-to-Face Interviewed Statewide Not reported Not reported Not reported Number of Applications Approved: Phone Interviewed Statewide Not reported Not reported Not reported Number of Recertifications Approved: Phone Interviewed Statewide Not reported Not reported Not reported Percent of Applications: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Percent of Recertifications: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Applications: Online Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Applications: Community Partner Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Applications: Kiosk Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Applications: Faxed Statewide Not reported Not reported Not reported Percent of Applications: Mailed Statewide Not reported Not reported Not reported Total Number of Applications Completed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Total Number of Recertifications Completed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Recertifications: Paper Submission to Local Office Statewide Program Access, Customer Service No No Number of Applications: Online Statewide Program Access, Efficiency, Customer Service No No Number of Applications: Community Partner Statewide Efficiency, Customer Service No No Number of Recertifications: Community Partner Statewide Efficiency, Customer Service No No Michigan (continued) A.106 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applications: Kiosk Statewide Efficiency, Customer Service No No Percent of Recertifications: Online Statewide Not reported Not reported Not reported Total Number of Applications Started Statewide Not reported Not reported Not reported Total Number of Recertifications Started Statewide Not reported Not reported Not reported Number of Recertifications: Online Statewide Not reported Not reported Not reported Number of Requests for Assistance\/One-page Applications Statewide Not reported Not reported Not reported Percent of Complete Applications Denied Statewide Not reported Not reported Not reported Number of Applications Denied: Online Statewide Program Access, Customer Service No No Number of Applications Denied: Community Partner Statewide Program Access, Customer Service No No Percent of Online Applications Denied for Failure to Submit Documentation Statewide Not reported Not reported Not reported Percent of Online Recertifications Denied for Failure to Submit Documentation Statewide Not reported Not reported Not reported Percent of Complete Recertifications Denied Statewide Not reported Not reported Not reported Percent of One- page\/Requests for Assistance Denied Statewide Not reported Not reported Not reported Percent of Initial Applications Denied Statewide Not reported Not reported Not reported Percent of Recertifications Denied Statewide Not reported Not reported Not reported Number of Applications Denied: Paper Submissions to Local Office Statewide Not reported Not reported Not reported Number of Recertifications Denied: Paper Submissions to Local Office Statewide Not reported Not reported Not reported Michigan (continued) A.107 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Recertifications Denied: Online Statewide Not reported Not reported Not reported Percent of Applicants that Appear Eligible for Expedited Benefits Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Percent of Applications with Incomplete Information Statewide Not reported Not reported Not reported Percent of Recertifications with Incomplete Information Statewide Not reported Not reported Not reported Number of Cases with Complete Verification Submitted Statewide Not reported Not reported Not reported Average Processing Time of Application Decision Statewide Not reported A determination must be made for online applications within the standard of promptness. FNS looks at timeliness for all applications that are processed Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. SOP = Standard of Promptness (all online expedited applications must be processed within six days) The FS-100 report is a Michigan document with data on waivers. It became available annually effective January 2011. Description: General: The Michigan state office directs all modernization efforts and leads performance measurement and reporting. Local offices receive monthly accuracy reports from the state, and they can pull state-compiled reports from the BRIDGES (computerized eligibility) system. Michigan noted that they are working to resolve issues with reporting accuracy, access to the online system, collecting data elements, and worker input errors. Three of Michigan's four contracted partners received ARRA funding. Of those, one partnership was newly created using only ARRA funds. The other two partners used the funds to grow outreach activities, expanding senior center locations and creating a food assistance hotline. Staff interviewed reported that these initiatives would not have been implemented or would have been delayed without ARRA funds. Measurement Goals: Michigan's performance measures touch on all four FNS goals. Michigan emphasized that their initiatives, standards, and incentives are all driven by FNS guidelines and requirements. Call Center: Michigan has not implemented a call center. Michigan (continued) A.108 Online System: Michigan developed a state-wide online application system that was implemented in 2010. They track three counts of aggregate data: number of applications submitted online per day, number of accounts created per month, and number of changes submitted per month. Document Imaging: Michigan does not have a document imaging initiative in place. Kiosks: Michigan has set up kiosks in some local and partner offices for online screening and application, change reporting, and checking and printing benefit summaries. They do not track any data on their kiosks. Waiver of Face-to-Face Interviews: Michigan collects performance measures and data on their waiver, including: percent of interviews by type (telephone versus face-to-face), the percent of telephone interview requests honored, and the percent of interviews that are missed and have a notice mailed. In addition, the state office records certain case characteristics, such as whether a household has an elderly person or English language learner. Shortened Interview: The state office and the local office interviewed do not offer a shortened interview process. Online Expedited Applications: The Michigan state office records the percent of applications approved (all online and expedited online), the number of expedited applications received online, and the number of applications approved (all online and expedited). If needed, the state office could produce a special query reporting case characteristics of expedited and regular applications according to demographics and\/or approval status. Application Tracking: Michigan reports numerous application tracking measures from the state office, especially in the areas of application accuracy, receipt, and approvals. While the state office differentiates applications by source, the local office interviewed does not. Changes Over Time: Michigan recently instated its BRIDGES eligibility system and a second party review system. Desired or Planned Future Measures: One local Michigan office would like to see a faster turn- around time at the local level for error rate reporting. Currently local offices must wait for these data to come from the state, but they would prefer having the capability to pull their own weekly reports. This does not seem feasible to them, as error rates are not reported until the state goes through the QC process. Michigan also reported a desire to track telephone and face-to-face interviews at finer levels of detail. Partnering: Michigan has four formalized, contract-based partnering arrangements and 397 registered community partners (any group that signs up to act as a partner). Many more partners may exist who choose not to register. Michigan monitors the cost of each of their partner contracts. Partners provide application access and assistance. Each of the four, formalized partners has a kiosk. Measures Collected by Partners: Michigan monitors service provision and funding for the four formalized partnerships. One component of this is an activity report provided on a monthly and yearly basis by each partner. Partners must track the number of clients served (via which services) and the number of applications submitted. Depending on the contract, partners may also collect information on the number of applicants deemed eligible, benefit amounts, and case characteristics. One partner interviewed looks at the discrepancies between screening estimates and the actual amount of benefits paid out, as well as differences between their eligibility system and the state's. They report three aggregate counts to the state: screening outcomes, summaries of issues, and the primary sources of referrals to their helpline. A.109 Profile: Minnesota State Administered or County Administered: County Number of Local\/County Agencies Interviewed: 4 Number of Partners Interviewed: 0 Initiatives Active in State: Call Center, Document Imaging, Partnering, Waiver of Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Regionally) Average Answer Speed Locally Efficiency, Customer Service 5 minutes (or less) No Average Hold Time Locally Efficiency, Customer Service No No Percent Calls Abandoned Locally Efficiency, Customer Service The office established a benchmark input from the workers No Average Call Duration Locally Efficiency, Customer Service No No Average Waiting Time (to Speak to Agent) Locally Efficiency, Customer Service No No Average Calls Handled Locally Efficiency, Customer Service No No Percent of Total Calls Handled Locally Efficiency, Customer Service No No Average Queue Time Locally Efficiency, Customer Service No No Average Time to De- Queue Locally Efficiency, Customer Service No No Average Time Prior to Abandonment Locally Efficiency, Customer Service No No Average After-Call Work Time by Staff Locally Efficiency, Customer Service No No Percent of Queued Calls Handled Locally Efficiency, Customer Service No No Number of Calls (in flow) Locally Efficiency, Customer Service No No Number of Calls Abandoned Locally Efficiency, Customer Service Not reported Not reported Number of Calls Queued Locally Efficiency, Customer Service Not reported Not reported Number of Calls Answered\/Handled Locally Efficiency, Customer Service Not reported Not reported Percent of Calls Completed in the Computer Phone System Locally Efficiency, Customer Service No No Number of Clients Accessing Computer Phone System Data Locally Efficiency, Customer Service Not reported Not reported Number of Calls Received Locally Efficiency, Customer Service Not reported Not reported Minnesota (continued) A.110 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Hang-ups Locally Efficiency, Customer Service Not reported Not reported Number of Calls Not Completed in Computer Phone System Locally Efficiency, Customer Service Not reported Not reported Number of Calls Completed in Computer Phone System Locally Efficiency, Customer Service Not reported Not reported Number of Client Calls Returned Locally Efficiency, Customer Service Not reported Not reported DOCUMENT IMAGING (Regionally) Percent of Documents Scanned Locally Efficiency No No Percent of Documents Received by Mail Locally Efficiency No No Percent of Documents Received by Fax Locally Efficiency No No Average Processing Time Locally Efficiency No No Number of Documents Received Locally Not reported Not reported Not reported Number of Documents Received: Mail Locally Not reported Not reported Not reported Number of Documents Received: Fax Locally Not reported Not reported Not reported Number of Documents Received: Other Electronic Locally Not reported Not reported Not reported Number of Documents Processed Locally Not reported Not reported Not reported Number of Documents Scanned Locally Not reported Not reported Not reported Number of Tasks Completed Locally Not reported Not reported Not reported PARTNERING (Regionally) None WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) None APPLICATION TRACKING (Statewide) Average Benefit Amount (for those eligible) Statewide Program Access, Customer Service No No Minnesota (continued) A.111 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Total Applications Approved Locally Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Total Recertifications Approved Locally Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Total Number of Applications Approved Locally Not reported Not reported Not reported Total Number of Recertifications Approved Locally Not reported Not reported Not reported Total Number of Applications Completed Locally Not reported Not reported Not reported Total Number of Recertifications Completed Locally Not reported Not reported Not reported Percent of Complete Applications Denied Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Applicants that Appear Eligible for Expedited Benefits Locally Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Applications: Expedited Locally Not reported No No Number of Cases with Complete Verification Submitted Locally Not reported Not reported Not reported Application Processing Time for Approvals for Initial Applications Statewide Efficiency, Customer Service FNS standards FNS incentives Average Processing Time of Application Decision Locally Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. The MAXIS system requires a lag of a few months on some data pulled from it. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Description: General: SNAP in Minnesota is state supervised and county administered. The extent of their modernization initiatives and performance measurement varies by county. Of counties we interviewed, only one office has implemented a call center, and it is one of two offices interviewed with a document imaging system. This office collects call center, document imaging measures, and application tracking measures. Minnesota (continued) A.112 Minnesota county offices receive some data through monthly state reports. The reports show the number of applications, the number of expedites, and programs for new application or recertification. Staff in one county office noted that these reports are of limited use. The MAXIS system is 20 years old and presents a barrier to collecting more information. Information is manually entered into the MAXIS system and transferred to a data warehouse. Not all of the information in the MAXIS system can be transferred to the data warehouse because the state does not have the resources to add fields to the data warehouse. The system can produce point-in-time reports of all pending cases by worker and the number of days pending. It cannot distinguish the type of application (new, repeat, or recertification) or the reason for denial, and it cannot differentiate by program (SNAP, Medicaid, etc.) or calculate any performance measures. In Minnesota, application determinations can take longer than 30 days, because workers give applicants more time to supply verification documents. Thus the data used for performance measures are always lagged by at least two months. The state has no official start and stop time when determining processing time. Measurement Goals: Minnesota's performance measures touch on all FNS goals for modernization, with a focus on efficiency and customer service for call center measures. Minnesota offices also track measures for state-conducted management evaluations and to identify potential lags in information availability. Call Center: One county in Minnesota operates a call for all social services programs (not specific to SNAP). They collect numerous performance measures and counts of aggregate data. Online System: Minnesota has no online application system. Document Imaging: Document imaging is handled at the local level in Minnesota, and half of the local offices interviewed reported having document imaging capabilities. One office reported collecting performance measures on percent of documents scanned, percent of documents received by mail and fax, and average processing time, as well as several sets of aggregate data. Kiosks: Minnesota has not implemented kiosks in the state or local offices. Waiver of Face-to-Face Interview: While Minnesota has a waiver of face-to-face interviews in place, the state and local offices do not collect any performance measures on this initiative. Every office records the type of interview (face-to-face versus telephone) in the MAXIS system or case narrative, but there is no corresponding field in their data warehouse system. Therefore, Minnesota cannot calculate any performance measures from this information. According to staff interviewed, creating measures tied to this initiative is not a priority for the state office because data warehouse systems are extremely expensive, they are facing staff layoffs, and FNS did not require any performance measures with the waivers. Partnering: Any Minnesota partnerships would be overseen at the state level because SNAP is county- administered. The state office did not report any partnerships and no local offices reported official partnership arrangements. One office has contracts with food distribution and public application assistance organizations (food shelves, case management groups), but these relationships are not specific to SNAP. Other offices reported informal associations, wherein community agencies provide SNAP information or refer clients to the county office to complete applications. Measures Collected by Partners: Not applicable. Minnesota (continued) A.113 Shortened Interviews: Minnesota does not use shortened interviews. Online Expedited Applications: Minnesota has no online application system. Application Tracking: All Minnesota applications are delivered to county offices. While the state tracks three measures (average benefit amount, percent of complete applications denied, application processing time for approvals for initial applications), the majority of performance measurement for this initiative is done at the local level. Only one local office that we spoke to tracks its applications by origin. This office separates applications received through a partner from all other application sources. Changes Over Time: The Minnesota state office has been calculating application tracking data in the same way for over ten years. Minnesota local offices noted no changes in data collection or reporting in the past year. Desired or Planned Future Measures: Minnesota offices expressed few plans for future expansions in their measurement or reporting. According to staff we interviewed, the data warehouse systems currently in use are expensive and only programmed to calculate certain measures. Due to staff layoffs and budget constraints, additional measures have to be high priority to add fields to the data warehouse system. One local office was interested in knowing the number of face-to-face interviews conducted and plans to perform a quarterly analysis of trends in monthly performance data. Additionally, this office has recently deployed surveys on customer service in an effort to identify areas for improvement. A.114 Profile: Mississippi State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 2 Number of Partners Interviewed: 0 Initiatives Active in State: Document Imaging, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives DOCUMENT IMAGING (Statewide) Number of Documents Received Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Yes, not specified The staff Performance Appraisal Rating (PAR) is based on data from the system reviews Number of Documents Scanned Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Review of all documents with 100 percent accuracy rate. The staff PAR is based on data from the system reviews. Number of Documents Per Case Statewide Customer Service No No Average Processing Time Locally Not reported Not reported Not reported Number of Documents Processed Locally Not reported Not reported Not reported Number of Tasks Completed Locally Not reported Not reported Not reported Percent of Documents Scanned Locally Not reported Not reported Not reported Number of Alerts Processed Locally Not reported Not reported Not reported APPLICATION TRACKING (Statewide) Percent of Applications Approved After Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Recertifications Approved After Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Total Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Total Recertifications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Average Benefit Amount (for those eligible) Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Mississippi (continued) A.115 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applications Approved: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Recertifications Approved: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Applications Approved: Faxed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Recertifications Approved: Faxed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Applications Approved: Mailed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Recertifications Approved: Mailed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Applications Approved: Face-to-Face Interviewed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Recertifications Approved: Face-to-Face Interviewed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Applications Not Completed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Applications: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Recertifications: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Applications: Faxed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Recertifications: Faxed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Applications: Mailed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Recertifications: Mailed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Mississippi (continued) A.116 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Total Number of Applications Started Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Total Number of Recertifications Started Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Total Number of Applications Completed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Total Number of Recertifications Completed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Applications: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Recertifications: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Applications: Faxed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Recertifications: Faxed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Applications: Mailed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Recertifications: Mailed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Requests for Assistance\/One-page Applications Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Initial Applications Denied Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Recertifications Denied Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Applications Denied: Paper Submissions to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Recertifications Denied: Paper Submissions to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Mississippi (continued) A.117 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Complete Applications Denied Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Complete Recertifications Denied Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of One- page\/Requests for Assistance Denied Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Average Processing Time of Recertification Decision Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Average Processing Time of Application Decision Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Applications with Incomplete Information Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Recertifications with Incomplete Information Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Applicants that Appear Eligible for Expedited Benefits Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Cases with Complete Verification Submitted Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Applications Approved: Phone Interviewed Locally Not reported Not reported Not reported Number of Recertifications Approved: Phone Interviewed Locally Not reported Not reported Not reported Number of Applications Denied: Mailed Locally Not reported Not reported Not reported Number of Recertifications Denied: Mailed Locally Not reported Not reported Not reported Number of Applications Denied: Faxed Locally Not reported Not reported Not reported Number of Recertifications Denied: Faxed Locally Not reported Not reported Not reported Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data is listed as Statewide . If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Mississippi (continued) A.118 Description: General: Mississippi conducts telephone interviews using the hardship waiver. Within the last year, they began using document imaging. Local offices primarily use state data and reports to measure performance. State staff noted that Mississippi has reduced the length of their SNAP application from 24 down to 2 pages, in order to ease the burden on workers with high caseloads. Mississippi reports that they use an expanded definition of the hardship waiver, allowing staff to conduct many interviews via telephone. (The state did not report an official waiver for face-to-face interviews). Local county offices track a wide variety of performance data for interviews conducted face- to-face and over the telephone for both initial applications and redeterminations, including the total number of interviews, percent of interviews by type, number of times unable to schedule an interview with an applicant, error rates for cases with telephone and face-to-face interviews, number of applicants requesting phone interviews, and percent of telephone interview requests honored. These measures are not tied to any modernization initiative, and are therefore not included in the table above. Measurement Goals: Mississippi measures performance to ensure program access, accuracy and integrity, efficiency, and customer service. Call Center: Mississippi does not have a call center initiative. Online System: Mississippi does not have an online system for screening or application. Document Imaging: The document imaging system has been in place since mid-2010. Mississippi records the number of documents received and scanned and the counties they came from, however they do not record the source of documents to be imaged (mail, fax, etc.). Local offices we interviewed reported that they track the number of documents processed, tasks completed, and alerts processed, in addition to average processing time. One local office noted that all quality control for imaging is manual and that there is no automated report to check at this time. Kiosks: Mississippi does not have kiosks for screening and application submission. Waiver of Face-to-Face Interview: Mississippi did not report having a waiver of face-to-face interviews. Shortened Interviews: Mississippi did not report having a shortened interview initiative. Online Expedited Applications: Mississippi does not have online expedited applications. Application Tracking: Mississippi tracks measures related to applications and recertifications, including the percentage received, approved, and denied, and the average benefit of those who are eligible. They also calculate the average processing time and percent of applicants that appear eligible for expedited benefits. Mississippi also calculates accuracy measures based on federal guidelines. Accuracy measures are used for QC review, as well as internally to review cases for training and management purposes. With all application tracking measures, state staff noted their motivation is to ensure accuracy, and that the information from application tracking measures is part of the performance review process for individual staff. However, there are no measure-specific standards or incentives. Staff reported that they measure timeliness in a way that differs from the QC process. Specifically, they said that data from the QC sample is too old for caseworkers to use to track timeliness of their own work. Partnering: Mississippi does not have partnering arrangements. Measures Collected by Partners: Not applicable. Mississippi (continued) A.119 Changes Over Time: Mississippi began imaging documents in mid-2010. The application and interview process overall were both shortened within a year of data collection for this study. In addition, the state added functionality to the eligibility system. Each screen in the eligibility system is now linked to a comment screen, and if a document is scanned, hot key links exist between the eligibility page and the scanned document and they can both be viewed at the same time. Desired or Planned Future Measures: None reported. A.120 Profile: Missouri State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 2 Number of Partners Interviewed: 0 Initiatives Active in State: Call Center, Partnering, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Statewide) Number of Calls Received Statewide None No No PARTNERING (Regionally) None APPLICATION TRACKING (Statewide) Percent of Total Applications Approved Statewide Program Access, Accuracy & Integrity No No Number of Applications Approved Statewide Program Access, Accuracy & Integrity No No Average Benefit Amount (for those eligible) Statewide Program Access No No Number of Applications: Mailed Statewide Program Access; Customer Service No No Number of Recertifications: Mailed Statewide Program Access; Customer Service No No Number of Applications Denied Statewide Program Access, Accuracy & Integrity No No Percent of Complete Applications Denied Statewide Program Access, Accuracy & Integrity No No Percent of Applicants that Appear Eligible for Expedited Benefits Statewide Program Access No No Average Processing Time of Application Decision Statewide Program Access, Efficiency No No Average Processing Time of Recertification Decision Statewide Program Access, Efficiency No No Application Currency Statewide Program Access, Efficiency Benchmark of 95 percent Corrective action plans are required if application currency falls below 95 percent Expedited Currency Statewide Program Access, Efficiency Benchmark of 95 percent Corrective action plans are required if expedited currency falls below 95 percent Missouri (continued) A.121 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Claims Processing Currency Statewide Program Access, Efficiency Benchmark of 90 percent Corrective action plans are required if claims processing currency falls below 90 percent. Also, FNS could take away state's option\/thres hold if benchmark is not met. Number of Applications: Online Locally Customer Service No No Number of Applications: Faxed Locally Customer Service No No Number of Applications: Mailed Locally Customer Service No No Number of Applications: Paper Submission to Local Office Locally Customer Service No No Percent of Applications: Paper Submission to Local Office Locally Customer Service No No Percent of Applications: Online Locally Customer Service No No Percent of Applications: Faxed Locally Customer Service No No Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Description: General: Missouri has a limited state-run call center. Partnerships are maintained on an informal basis. Applications are available for download on the Department of Children and Family Services (DCFS) website. The site also links to the FNS prescreening tool, but this does not pre-populate the application. Missouri does not have a waiver of the face-to-face interview, though they make use of the hardship waiver to conduct telephone interviews. The type of interview (telephone versus face-to-face) is recorded in each case's narrative section, in addition to the reason for using the waiver. If this information is needed, it must be manually retrieved from each case; there is no easy or quick way to compile statistics on this. Measurement Goals: Missouri tracks error rates with the goal of program access as well as accuracy and integrity. They track the percentage of applications denied and approved for program Missouri (continued) A.122 access and integrity. Timeliness measures (also known as currency measures) relate to efficiency and program access. Overall, program access is an overarching goal for all these measures. At the local level, monthly totals of applications received through all methods are tracked to enhance customer service. Call Center: Missouri has a call center that only consists of an Automated Voice Response System. It operates around the clock, and clients can call to find automated information about: whether their benefits are available and on what date they are available; what the benefit amount is; and what month their certification is set to expire. Prior month's information is also available. The system has 96 phone lines, 12 of which are for SNAP callers (the rest service other DCFS programs). State staff reported that they only track the number of calls received. Reports are produced monthly but no actions are taken based on the reports. Online System: Missouri does not have an online system for screening and application submission for SNAP (but does have a system for the Children's Health Insurance Program and Medicaid). Applications for SNAP are available for download on the DCFS website. The site also links to the FNS prescreening tool, but this does not pre-populate the application. Document Imaging: Missouri does not have document imaging. Kiosks: Missouri does not have kiosks. Waiver of Face-to-Face Interview: Missouri does not have a waiver in place. Shortened Interviews: Missouri does not have a shortened interview initiative. Online Expedited Applications: Missouri does not accept expedited applications online for SNAP. Application Tracking: At the local level, clerical staff manually capture the number of applications received via mail, in-person drop-off, fax, or online. Monthly counts are compiled and tracked. One local office we spoke with also tracks the percentage of applications received via mail, online, paper, and fax. However, the state eligibility system only has two codes for tracking the method of application delivery: mail-in and in-person. As a result, mail-ins, drop-offs, and faxed applications are coded as mail-in in the eligibility system. All applications are screened for expedited service. In addition to the required quality control measures, Missouri has its own case review system that operates at the county level. For the FNS QC process, they read about 120 cases per month, whereas about 9,000 to 10,000 cases are read each month for their internal case review system. Most of these cases are for SNAP (4,000-5,000 cases). The state's QC unit looks at all Case Review System reports, QC reports, and other data. They also work with the counties to identify trends that need to be addressed and develop Corrective Action Plans. The calculation for application timeliness (or currency) differs from the FNS calculation. Missouri focuses on whether eligibility is determined within 30 days, not on whether applicant receives benefits within 30 days. Because of this difference, Missouri does not know their official application timeliness until FNS sends them six-month updates. Changes Over Time: None reported. Partnering: One county has a long-standing relationship with both the two partners. These partners keep paper applications available to improve access to the program. Measures Collected by Partners: We did not interview partners in Missouri for this study. Missouri (continued) A.123 Desired or Planned Future Measures: Missouri staff reported that they would like to have a document imaging system, but they are currently satisfied with the collection of performance measures from their current system. At the local level, offices would like to have data on reasons for denial. The data are currently captured by the system, but reports are not available. Because clerical staff manually record the method by which applications are received, one local office interviewed is currently looking into an electronic means of tracking. A.124 Profile: Montana State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 2 Number of Partners Interviewed: 0 Initiatives Active in State: Online System, Partnering, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives ONLINE SYSTEMa (Statewide) Number of Online Applications Submitted Locally Program Access, Efficiency, Customer Service No No PARTNERING (Statewide) None ONLINE EXPEDITED APPLICATIONS (Statewide) None APPLICATION TRACKING (Statewide) Average Benefit Amount (for those eligible) Statewide No No Applications Not Timely Statewide Efficiency Federal benchmarks are used No Number of Applications: Paper Submission to Local Office Locally Program Access, Customer Service No No Denials for Expedited Applications by Reason Statewide Not reported Not reported Processing Time Report Statewide Federal timeliness standards are used Federal benchmarks are used Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. a Online system was launched in December 2010, after our interview with the state office staff. Description: General: Montana has a new online application system that was launched following our interview with the state agency staff but prior to our interviews with local staff. In general, state and local offices collectively record a limited number of measures, primarily in relation to application tracking. Local offices in Montana do not receive reports from the state, although the state generates a monthly report with county break-downs of application tracking data that is posted on their public website. At the time of the interview, Montana was working toward putting a document imaging initiative in place. Measurement Goals: Montana's measures pursue the FNS goals of program access, efficiency, and customer service. Montana (continued) A.125 Online System: The online application system in Montana is new as of December 2010. One local office reported that they collect the number of online applications submitted. The state may collect additional measures that are not shown here because of the timing of our data collection. Call Center: Montana does not operate a call center. Document Imaging: Montana has not established a document imaging system. Kiosks: Montana has not set up kiosks. Waiver of Face-to-Face Interview: Montana does not have a blanket waiver. They only have a waiver for instances of hardship. Shortened Interviews: Montana did not report having shortened interviews. Online Expedited Applications: The online application system in Montana is new as of December 2010. Montana did not report any measures related to online expedited applications. Application Tracking: State and local offices collect some measures and data on application approvals, receipt, and processing time. At the time of the interview, local offices were recording or planning to record whether applications were delivered online. Changes Over Time: Montana has not changed the way their data are collected or reported recently. At the state's request, one local office did a monthly cross-check of application status to compile a tally of applications approved within the reporting month, but they no longer do this. Another local office tracked applications received through partners, but they discontinued this practice in 2008. Desired or Planned Future Measures: Staff interviewed noted that it would be helpful to have ad hoc reports available without having to request them. One local office is interested in knowing the number of applications received through the online system and plans to ask front desk managers to track these data in their spreadsheets. They are also interested in seeing timeliness data, a breakdown of interviews by type (face-to-face versus telephone), and data on the online system and screening tool usage. At the time of the interview, contractors were working on a new computer application processing system for TANF and SNAP that would provide some of these data as well as more workload management tools. Partnering: Montana has one formal partnership with a food bank. The partner received funds from a participation grant. They do not receive applications. Measures Collected by Partners: We did not interview any of Montana's partners as a part of this study. A.126 Profile: Nebraska State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 4 Number of Partners Interviewed: 6 Initiatives Active in State: Call Center, Online System, Document Imaging, Kiosks, Partnering, Waiver of Face\u2010to\u2010 Face Interview, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Regionallya) Average Waiting Time (to Speak to Agent) Regional Efficiency, Customer Service No No Average Calls Handled Regional Efficiency, Customer Service Not reported Not reported Percent of Total Calls Handled Regional Efficiency, Customer Service Not reported Not reported Average Staff on Phones Regional Efficiency, Customer Service Not reported Not reported Average Hold Time Regional Efficiency, Customer Service Not reported Not reported Percent Calls Abandoned Regional Efficiency, Customer Service Nine in ten calls are answered Not reported Average Call Duration Regional Efficiency, Customer Service Not reported Not reported Average Queue Time Regional Efficiency, Customer Service Not reported Not reported Average Time Prior to Abandonment Regional Efficiency, Customer Service Not reported Not reported Average After-Call Work Time by Staff Regional Efficiency, Customer Service Not reported Not reported Percent of Queued Calls Handled Regional Efficiency, Customer Service Not reported Not reported Percent of Changes Processed Regional Efficiency, Customer Service Not reported Not reported Percent of Calls Completed in the Computer Phone System Regional Efficiency, Customer Service No No Percentage Answered in Less than Three Minutes Regional Efficiency, Customer Service No No Average Time to Process Change Locally Not reported Not reported Not reported Number of Changes Received Locally Not reported Not reported Not reported Number of Changes Processed Locally Not reported Not reported Not reported Nebraska (continued) A.127 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Average Time to De- Queue Locally Not reported Not reported Not reported Number of Calls (in flow) Locally Not reported Not reported Not reported Number of Calls Abandoned Locally Not reported Not reported Not reported Number of Calls Queued Locally Not reported Not reported Not reported Number of Calls Answered\/Handled Locally Not reported Not reported Not reported Number of Agents Assigned to Phones Locally Not reported Not reported Not reported Number of Calls Receiving Busy Signal\/Unable to Connect with Phone System Locally Not reported Not reported Not reported Average Time to Process Change Locally Not reported Not reported Not reported Number of Clients Accessing Computer Phone System Data Locally Not reported Not reported Not reported Number of Calls Received Locally Not reported Not reported Not reported Number of Hang-ups Locally Not reported Not reported Not reported Number of Changes Received Locally Not reported Not reported Not reported Number of Changes Processed Locally Not reported Not reported Not reported Number of Calls Not Completed in Computer Phone System Locally Not reported Not reported Not reported Number of Calls Completed in Computer Phone System Locally Not reported Not reported Not reported Number of Calls Transferred to Agent Locally Not reported Not reported Not reported Number of Cases Pending Benefit Decision: Call Center Locally Not reported Not reported Not reported Number of Applications Processed: Call Center Locally Not reported Not reported Not reported ONLINE SYSTEM (Statewide) Percent of Applications Received Statewide Program Access, Efficiency, Customer Service No No Number of Changes Submitted Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Nebraska (continued) A.128 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives DOCUMENT IMAGING (Regionallya) Percent of Documents Scanned Regional Efficiency Not reported Not reported Percent of Documents Received by Mail Regional Efficiency Not reported Not reported Percent of Documents Received by Fax Regional Efficiency Not reported Not reported Average Processing Time Regional Efficiency Not reported Not reported KIOSKS (Statewide) None PARTNERING (Statewide) Percent of Partners Providing Access to Terminals\/Kiosks Statewide Program Access Not reported Not reported Percent of Partners Providing Application Assistance Statewide Program Access Not reported Not reported WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Percent of Interviews Conducted by Telephone Statewide Program Access Not reported Not reported Percent of Interviews Conducted Face-to-Face Statewide Program Access Not reported Not reported Error Rate for Cases with Telephone Interviews Statewide Program Access, Accuracy & Integrity Not reported Not reported Error Rate for Cases with Face-to-Face Interviews Statewide Program Access, Accuracy & Integrity Not reported Not reported ONLINE EXPEDITED APPLICATIONS (Statewide) Percent of Applications Approved: Online Expedited Statewide Not reported Not reported Not reported Percent of Applications Approved: Online All Statewide Not reported Not reported Not reported APPLICATION TRACKING (Statewide) Percent of Total Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Total Recertifications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Nebraska (continued) A.129 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Average Processing Time of Application Decision Statewide Accuracy & Integrity, Efficiency, Customer Service No No Average Processing Time of Recertification Decision Statewide Accuracy & Integrity, Efficiency, Customer Service No No Total Number of Applications Completed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Total Number of Recertifications Completed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Applications: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Recertifications: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Applications: Online Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. aThe call center and document imaging initiatives are being rolled out across the state in phases. They were not in all areas of the state at the time of our interview. Description: General: Nebraska is divided into five regions. At the time of our interviews, Nebraska was in a state of transition. Across the regions, they were introducing document imaging, a universal caseload system, and call centers. The document imaging and call centers will each cover specific regions of the state, rather than the entire state. Measurement Goals: Nebraska seeks to become a highly modernized state, and thus aspires to compare performance with other highly modernized states. They primarily use performance measurement and reporting to ensure client access and ease of use and reduce the number of times clients need to contact them to receive their benefits. Reports are broken down by service area, supervisor, and staff level. All levels of staff have access to the reports. Call Center: The state is rolling out its call centers, called Customer Service Centers. The first one is located in Lincoln and serves the southeast region of the state. The second one will serve the northern and eastern regions of the state. There will be a total of four. The Customer Service Centers have both automated responses and the ability to transfer calls to agents. They can also be used for accepting and processing applications and recertifications, phone interviews, and reporting changes. To ensure continued access for clients, measures on which they will focus include the call wait time, the number of dropped calls, and how often the system hangs up on callers. They have not yet discussed what standards could be put into place. Nebraska (continued) A.130 Online System: The state accepts applications online. The data do not automatically transfer to the eligibility system, but the system creates a visual link between the fields in the application and the field in the eligibility system using a graphical user interface. The worker can then enter information from the application into the eligibility system. According to staff interviewed, they hope to begin transferring the data automatically, but incomplete data entry by clients makes this difficult. Nebraska tracks whether applications are received online or by other means, and receipt time. Clients also report where they completed the online application, for example, whether they completed it at home, at a friend's house, or at a community partner's office. The computer system sorts the applications by the service area of the client and forwards the application to that work area queue. Document Imaging: Nebraska has two document imaging centers. For several offices, all mail is forwarded directly to the centers\u2014eventually all mail will be sent directly to the imaging centers. Documents are scanned in large batch jobs then transferred to workers for manual indexing and linkage to cases. At this point, the files can be accessed by all staff. They are able to track the number of documents scanned, the worker, and which scanner was used. Kiosks: Kiosks are available inside local offices and at the offices of numerous community-based partners. Primarily, the kiosks are used to access Nebraska's website to apply for benefits. Applicants indicate the place in which they are accessing the system. Waiver of Face-to-Face Interview: Nebraska is working toward having most of its interviews by phone. The eligibility system keeps track of the interview type and the eligibility decision by type. Shortened Interviews: Nebraska did not report having shortened interviews. Online Expedited Applications: Online applications can be processed as expedited cases. Application Tracking: Nebraska tracks applications by whether they were submitted online or through some other means. If online, they also track the location of the applicant when filing (home, friend's house, community partner, et cetera). The local offices receive daily reports of the number of cases that are pending and for how long they have been pending. They can view this data in real time under the universal case management system. Local offices also receive monthly reports describing the office's activities during the month. To maintain a high accuracy rate, Nebraska has a team that reviews processed cases, by worker, to check accuracy. Workers are recognized for correct cases. Changes Over Time: Nebraska's ability to track performance is increasing as they modernize their systems. Partnering: Nebraska has both formal and informal arrangements with community-based organizations. The formal arrangements are for partners that receive funding for the services they provide. The regions and local offices also partner with numerous organizations. The regions have community support specialists to work with the partners and do outreach in the community. The partners generally make kiosks available to their clients to apply online for SNAP and other state benefits. They also provide assistance if needed. Some partners received training from the state about the online system. Measures Collected by Partners: Some partners carefully track the number of individuals they serve, and in some cases the number of staff hours involved in assisting clients. Other partners do not have any performance measures. Nebraska tracks only the number of partners. Nebraska (continued) A.131 Desired or Planned Future Measures: Nebraska has additional reports in the design stage. One local office worker desired a report that indicated the percentage of the eligible population that was being served. A.132 Profile: Nevada State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 0 Number of Partners Interviewed: 0 Initiatives Active in State: Call Center, Partnering, Waiver of Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Statewide) Number of Calls (in flow) Statewide Program Access, Customer Service Not reported Not reported Number of Calls Abandoned Statewide Program Access, Efficiency, Customer Service Not reported Not reported Number of Calls Queued Statewide Program Access, Efficiency, Customer Service Not reported Not reported Average Hold Time Statewide Program Access, Customer Service Goal is an average of 10 minutes. No Average Hold Time Statewide Program Access, Customer Service Not reported Not reported Average Calls Handled Statewide Program Access, Customer Service Not reported Not reported Percent of Total Calls Handled Statewide Program Access, Customer Service Goal of 50 percent of calls that are directed to the call center are processed. Not reported Average Staff on Phones Statewide Program Access, Efficiency, Customer Service Not reported Not reported Percent of Calls Completed in the Computer Phone System Statewide Not reported 95 percent of calls resolved by the Voice Response System. No Average Time Prior to Abandonment Statewide Program Access, Customer Service Not reported Not reported PARTNERING (Statewide) Percent of Partners Providing Application Assistance Statewide Program Access, Customer Service No Continuation of the project Number of Partners Statewide Program Access, Customer Service No No Number Accepting Applications Statewide Program Access, Customer Service No No Number Providing Application Assistance Statewide Program Access, Customer Service No No Number Clients Assisted per Partner Statewide Program Access, Customer Service No No Nevada (continued) A.133 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applications Received per Partner Statewide Program Access, Customer Service No No WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) None APPLICATION TRACKING (Statewide) Percent of Applications Approved After Paper Submission to Local Office Statewide Accuracy & Integrity, Efficiency, Customer Service No No Percent of Recertifications Approved After Paper Submission to Local Office Statewide Accuracy & Integrity, Efficiency, Customer Service No No Percent of Community Partner Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Total Applications Approved Statewide Program Access, Customer Service No No Percent of Total Recertifications Approved Statewide Program Access, Customer Service No No Average Benefit Amount (for those eligible) Statewide Program Access, Accuracy & Integrity No No Number of Applications Approved: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Customer Service No No Number of Recertifications Approved: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Customer Service No No Number of Applications Approved: Community Partner Statewide Program Access, Accuracy & Integrity, Customer Service No No Percent of Recertifications Denied Statewide Accuracy & Integrity, Efficiency No No Percent of Complete Applications Denied Statewide Accuracy & Integrity, Efficiency No No Number of Applications Denied: Paper Submission to Local Office Statewide Accuracy & Integrity, Efficiency No No Number of Recertifications Denied: Paper Submissions to Local Office Statewide Accuracy & Integrity, Efficiency No No Nevada (continued) A.134 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applications Denied: Community Partner Statewide Program Access, Efficiency, Customer Service No No Average Processing Time of Application Decision Statewide Accuracy & Integrity, Efficiency Timeliness based on federal policy No Average Processing Time of Recertification Decision Statewide Accuracy & Integrity, Efficiency Federal guidelines No Notes: State collects mandatory accuracy data through QC reviews. Description: General: Nevada runs a demonstration project with two community partner organizations. (In addition to partnering, Nevada operates a call center and has a waiver of face-to-face interviews. Measurement Goals: Nevada primarily measures performance to ensure efficiency customer service, accuracy and integrity, and program access. Call Center: Nevada operates a call center. Call center staff take client information and answer questions. Performance measures were developed with the goal of monitoring effectiveness per the direction of the Nevada legislature. Online System: Nevada does not have an online system. Document Imaging: Nevada does not have document imaging. Kiosks: Nevada does not have kiosks. Waiver of Face-to-Face Interview: Nevada has a waiver to conduct interviews via telephone; however they do not collect any performance measures or aggregate data for this initiative. State staff reported that they conduct face-to-face interviews unless a client requests otherwise. The waiver was sought to alleviate overload in local offices. Shortened Interviews: Nevada does not have a shortened interview initiative. Online Expedited Applications: Nevada does not accept expedited applications online. Application Tracking: Currently, Nevada only accepts paper applications. Community partners may submit applications, and Nevada tracks approval and denials for these applications separately, from a manual report. Along with the required QC reviews, additional reviews are done for employees. Most employees get five cases reviewed per month. Employees who are new or on probation can have all of Partnering: Nevada partners with five community organizations, two of which are part of an FNS demonstration project. Those partners complete the interview and maintain the records of all interviews and submit the applications directly to the local officers. Other partners assist with applications. The state tracks the number of applications submitted by partners, and can calculate the percentage of these applications that are approved or denied. Nevada partners with community organizations to alleviate high local office caseloads. Measures Collected by Partners: The state requires the partners to collect measures. We did not contact any partners as part of this data collection effort. Nevada (continued) A.135 their caseloads reviewed. Changes Over Time: Not reported. Desired or Planned Future Measures: Nevada is in the process of piloting an online application system (ACCESS Nevada). A.136 Profile: New Hampshire State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 0 Number of Partners Interviewed: 0 Initiatives Active in State: Online System, Document Imaging, Partnering, Waiver of Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives ONLINE SYSTEM (Statewide) None DOCUMENT IMAGING (Statewide) Percent of Documents Scanned Statewide Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Average Processing Time Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported PARTNERING (Statewide) Number of Partners Statewide Not reported Not reported WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Error Rate for Cases with Telephone Interviews Statewide Accuracy & Integrity Not reported Not reported Error Rate for Cases with Face-to-Face Interviews Statewide Accuracy & Integrity Not reported Not reported Number of redetermination interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported APPLICATION TRACKING (Statewide) Percent of Total Applications Approved Statewide Program Access, Efficiency, Customer Service Not reported Not reported Percent of Total Recertifications Approved Statewide Program Access, Efficiency, Customer Service Not reported Not reported Average Benefit Amount (for those eligible) Statewide Customer Service Not reported Not reported Percent of Applications Not Completed Statewide Program Access, Accuracy & Integrity, Customer Service Not reported Not reported Total Number of Applications Started Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Total Number of Recertifications Started Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported New Hampshire (continued) A.137 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Total Number of Applications Completed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Total Number of Recertifications Completed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Online Recertifications Denied for Failure to Submit Documentation Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Complete Applications Denied Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Complete Recertifications Denied Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Recertifications Denied Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Average Processing Time of Application Decision Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Average Processing Time of Recertification Decision Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Notes: State collects mandatory accuracy data through QC reviews. Description: General: New Hampshire operates an online system and document imaging system and maintains partnerships with community organizations. The waiver of the face-to-face interview is available at both application and recertification. Measurement Goals: New Hampshire primarily measures performance to ensure customer service, efficiency, accuracy and integrity, and program access. Call Center: New Hampshire does not have a call center. Online System: New Hampshire's Department of Health and Human Services has applications available for download and paper submission and online signed submission. They do not collect any performance data related to the online system. Document Imaging: New Hampshire has document imaging capabilities with electronic case records and remote access to electronic case records. They track the percentage of documents scanned and average processing time. Kiosks: New Hampshire does not have kiosks. New Hampshire (continued) A.138 Waiver of Face-to-Face Interview: New Hampshire tracks the error rate for cases with telephone and face-to-face interviews for the purposes of accuracy and integrity. Shortened Interviews: New Hampshire does not have a shortened interview initiative. Online Expedited Applications: New Hampshire does not accept expedited applications online. Application Tracking: New Hampshire calculates a variety of performance measures and aggregate data related to applications and recertifications, including the percentage not complete, approved, denied, average processing time, and the average benefit of those who are approved. They also track application accuracy measures. Changes Over Time: Not reported. Desired or Planned Future Measures: Not reported. Partnering: New Hampshire maintains partnerships with several community organizations. Partner organizations complete and submit applications. Measures Collected by Partners: New Hampshire does not ask any partners to calculate performance measures. We did not contact partners as part of this study. A.139 Profile: New Jersey State Administered or County Administered: County Number of Local\/County Agencies Interviewed: 0 Number of Partners Interviewed: 0 Initiatives Active in State: Online System, Waiver of Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives ONLINE SYSTEM (Statewide) Number of Online Screenings Started: Online Statewide Customer Service No No Number of Online Screenings Completed: Online Statewide Customer Service No No WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) None APPLICATION TRACKING (Statewide) None Notes: State collects mandatory accuracy data through QC reviews. Description: General: New Jersey has a waiver of the face-to-face interview and an online application system. Measurement Goals: New Jersey measures performance data for customer service purposes. Call Center: New Jersey does not have a call center. Online System: New Jersey operates an online application system and screening tool. The system was first developed by a contractor and is now run by the state. It is possible to pre-populate the online application (OneApp) with this information from the screening tool. Data obtained through the online application is electronically transferred to the eligibility system. New Jersey has the capability to track the number of online screenings started and completed for customer service and outreach purposes. This data is tracked through an automated process and can be monitored down to the county level. Document Imaging: New Jersey does not have document imaging. Kiosks: New Jersey does not have Kiosks. Waiver of Face-to-Face Interview: New Jersey has a waiver of the face-to-face interview. They record the type of interview (telephone, face-to-face, or home visit) in the eligibility system, however, they do not track any performance data for this initiative. Shortened Interviews: New Jersey does not have a shortened interview initiative. New Jersey (continued) A.140 Online Expedited Applications: New Jersey did not report an online expedited application initiative. Application Tracking: New Jersey does not track any performance data related to application tracking. According to the state respondent, some counties may have developed their own systems to track applications. No local county offices were interviewed as a part of this study. Changes Over Time: New Jersey has been using the same application tracking system since the early 1980s. During this fiscal year, New Jersey plans to begin working with community partners. Desired or Planned Future Measures: New Jersey reported that they plan to implement a new eligibility system in Fall 2012. The system will give the state more capabilities than they have with the current system and will be standardized across counties in the state. The state also expressed interest in tracking the number of times a client visits an office prior to receiving benefits. Partnering: New Jersey did not report partnering arrangements. Measures Collected by Partners: Not applicable. A.141 Profile: New Mexico State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 0 Number of Partners Interviewed: 0 Initiatives Active in State: Waiver of Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Percent of Interviews Conducted by Telephone Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Due to the rural nature and mixed populations, a standard benchmark has not yet been determined. No Percent of Interviews Conducted Face-to-Face Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Total Number of Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Applicants Requesting a Phone Interview Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported No Number of Redetermination Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No APPLICATION TRACKING (Statewide) Percent of Applications Approved After Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Yes, not specified No Percent of Recertifications Approved After Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Average Benefit Amount (for those eligible) Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Number of Applications Approved: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Yes, not specified Yes, not specified Number of Recertifications Approved: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Yes, not specified Yes, not specified New Mexico (continued) A.142 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Recertifications Approved: Face-to-Face Interviewed Statewide Program Access, Accuracy & Integrity, Efficiency Not reported Not reported Percent of Applications: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Percent of Recertifications: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Total Number of Applications Completed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Total Number of Recertifications Completed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Applications: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Yes, not specified Yes, not specified Number of Recertifications: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Percent of Complete Applications Denied Statewide Accuracy & Integrity Not sure Not sure Percent of Complete Recertifications Denied Statewide Accuracy & Integrity Not sure Not sure Percent of Initial Applications Denied Statewide Accuracy & Integrity Not sure Not sure Average Processing Time of Application Decision Statewide Accuracy & Integrity, Efficiency, Customer Service No No Average Processing Time of Recertification Decision Statewide Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Notes: State collects mandatory accuracy data through QC reviews. Description: General: New Mexico's SNAP program is administered by the state. New Mexico would like a system where overnight batching and ad hoc reports can function faster. Most state staff do not have access to their own database table for the purposes of running any in house queries. Measurement Goals: New Mexico primarily measures performance to ensure customer service, accuracy and integrity, and efficiency. Call Center: New Mexico does not have a call center. New Mexico (continued) A.143 Online System: New Mexico does not have an online system. Document Imaging: New Mexico does not have document imaging. Kiosks: New Mexico does not have kiosks. Waiver of Face-to-Face Interview: In New Mexico, the waiver is for recertification interviews only. All offices are encouraged by the state to use telephone interviews, unless the client requests an in office interview, to increase access and participation. The state believes the waiver helps minimize barriers to participation and to decrease the workload for eligibility workers. The current eligibility system was revised to account for the waiver so that the eligibility system could run reports based on interview type. Case data are manually entered by SNAP case workers. Automated reports are available for the initiative. Shortened Interviews: New Mexico does not have a shortened interview initiative. Online Expedited Applications: New Mexico does not accept expedited applications online. Application Tracking: New Mexico's application tracking process tracks a number of measures. State staff reported that legislators and stakeholders use the average benefit amount data, which are broken down by household characteristics to make sure certain populations are not being underserved. In New Mexico, staff told us, the state eligibility system is constantly changing. Data for the application process are all manually entered by SNAP workers. Standard monthly reports are automated and ad-hoc reports can be request as needed. State staff reported that they would like some further control over access to ad-hoc reports. Changes Over Time: In New Mexico, caseworker reviews and trainings have been cut because of funding and budget cuts. A 1988 legal settlement resulted in more stringent quality controls than are required by FNS. The data collection system changed to meet the settlement and the changes have to be approved not only by New Mexico SNAP staff, but also by the attorneys in the settlement case. Desired or Planned Future Measures: New Mexico's plans for a document imagining process were in the early stages of implementation and planning at the time of our study. Partnering: New Mexico does not have an initiative for partnering. Measures Collected by Partners: Not applicable. A.144 Profile: New York State Administered or County Administered: County Number of Local\/County Agencies Interviewed: 5 Number of Partners Interviewed: 1 Initiatives Active in State: Call Center, Online System, Document Imaging, Kiosks, Partnering, Waiver of Face\u2010to\u2010Face Interview, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Regionally) Average Time to Process Change Locally Not reported Not reported Not reported Average Queue Time Locally Not reported Not reported Not reported Average Time Prior to Abandonment Locally Not reported Not reported Not reported Number of Calls Abandoned Locally Not reported No No Number of Calls Queued Locally Not reported Not reported Not reported Number of Calls Answered\/Handled Locally Not reported Not reported Not reported Number of Agents Assigned to Phones Locally Not reported Not reported Not reported Average Answer Speed Locally Not reported No No Average Hold Time Locally Not reported Not reported Not reported Percent Calls Abandoned Locally Not reported Not reported Not reported Average Call Duration Locally Not reported Not reported Not reported Average Waiting Time (to Speak to Agent) Locally Not reported Not reported Not reported Average Calls Handled Locally Not reported No No Percent of Total Calls Handled Locally Not reported Not reported Not reported Percent of Changes Processed Locally Not reported No No Number of Changes Received Locally Not reported Not reported Not reported Number of Changes Processed Locally Not reported Not reported Not reported Number of Calls Offered Locally Not reported Not reported Not reported ONLINE SYSTEM (Statewide) Percent of Users Selecting Spanish Language Statewide Program Access, Customer Service No No New York (continued) A.145 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Screened Individuals Potentially Eligible: Online Statewide Program Access, Customer Service No No Percent of Applicants Who Received Help to Apply Online Statewide Program Access, Customer Service No No Percent of Applications Received Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Goal of 20 percent of applications coming in online No Percent of Applicants with Hardship Reason for Requesting Phone Interviews Statewide Program Access, Accuracy & Integrity, Customer Service No No Percent of Applications Received Outside Business Hours Statewide Program Access, Customer Service Not reported Not reported Number of Online Screenings Started: Online Statewide Program Access, Customer Service No No Number of Online Screenings Completed: Online Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Applications Started Statewide Program Access, Customer Service No No Number of Abandoned Applications Statewide Accuracy & Integrity, Efficiency No No Number of Applications in Progress Locally Accuracy & Integrity No No Number of Applications Submitted and Transferred Locally Accuracy & Integrity No No Number of Users Accessing a Specific Page Locally Efficiency, Customer Service No No How Long Users Spent on Each Page of the Online Application Locally Customer Service No No Number of Accounts Created Locally Not reported Not reported Not reported Number of Referrals from 311 Online Locally Not reported Not reported Not reported Average Number Submitted per Month Statewide Efficiency Not reported Not reported Average Level of Benefits Statewide Efficiency Not reported Not reported New York (continued) A.146 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applicants who Received Help to Apply Online Statewide Program Access, Accuracy & Integrity, Customer Service No No Number of Registered Applications Statewide Not reported Not reported Not reported Number of Applications Withdrawn by Applicant Statewide Not reported Not reported Not reported Number of Applications Withdrawn by System Statewide Not reported Not reported Not reported Number of Applications Submitted Outside Business Hours Statewide Not reported Not reported Not reported Number of Applications Abandoned Statewide Not reported Not reported Not reported Number of Applications Not Submitted Statewide Not reported Not reported Not reported DOCUMENT IMAGING (Regionally) Number of Documents Processed Statewide Efficiency Not reported Not reported Number of Documents Scanned Statewide Efficiency Not reported Not reported Percent of Documents Scanned Statewide Efficiency Not reported Not reported Number of Documents Received: Mail Statewide Not reported Not reported Not reported Average Processing Time Locally Not reported No No Number of Tasks Completed Locally Not reported Not reported Not reported KIOSKS (Regionally) Percent of Screenings Resulting in Application Submission: Kiosk Statewide Program Access Not reported Not reported PARTNERING (Statewide) Number Accepting Applications Statewide Not reported Not reported Not reported Number with Terminals\/Kiosks to Submit Application Statewide Not reported Not reported Not reported Number of Partners Statewide Program Access, Efficiency, Customer Service No No Number Providing Application Assistance Statewide Program Access, Efficiency, Customer Service No No New York (continued) A.147 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applications Received per Partner Statewide Efficiency No No WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Error Rate for Cases with Face-to-Face Interviews Statewide Not reported Not reported Not reported Error Rate for Cases with Telephone Interviews Statewide Not reported Not reported Not reported Number of Applicants Requesting an In-Person Interview Statewide Not reported Not reported Not reported Number of Face-to-Face Interviews Statewide Not reported Not reported Not reported Number of Telephone Interviews Statewide Not reported Not reported Not reported Notice of Missed Interview Mailed Statewide Efficiency Not reported Not reported Number of Applicants Requesting a Phone Interview Statewide Not reported Not reported Not reported Number of Missed Interviews Statewide Efficiency Not reported Not reported Number of Redetermination Interviews Statewide Efficiency Not reported Not reported Number of Times Unable to Schedule Interview with Applicant Statewide Efficiency Not reported Not reported Percent of Interviews Conducted by Telephone Statewide Efficiency Not reported Not reported Percent of Interviews Conducted Face-to-Face Statewide Efficiency Not reported Not reported Percent of Telephone Interview Requests Honored Statewide Efficiency Not reported Not reported Percent that are Missed and have Notice Mailed Statewide Efficiency Not reported Not reported Total Number of Interviews Statewide Efficiency Not reported Not reported ONLINE EXPEDITED APPLICATIONS (Statewide) Percent of Applications Approved: Online All Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No New York (continued) A.148 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Expedited Applications Received Online Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Applications Approved: Online All Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Applications Approved: Online Expedited Statewide Efficiency Not reported Not reported Percent of Applications Approved: Online Expedited Statewide Not reported Not reported Not reported APPLICATION TRACKING (Statewide) Percent of Online Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Total Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Total Recertifications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Timeliness Rate Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Applications Approved: Online Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Recertifications Approved After Paper Submission to Local Office Statewide Efficiency Not reported Not reported Percent of Mailed Recertifications Approved Statewide Efficiency Not reported Not reported Percent of Applications Approved After Paper Submission to Local Office Statewide Efficiency Not reported Not reported Number of Recertifications Approved: Paper Submission to Local Office Statewide Efficiency Not reported Not reported Number of Recertifications Approved: Mailed Statewide Efficiency Not reported Not reported New York (continued) A.149 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applications Approved: Paper Submission to Local Office Statewide Efficiency Not reported Not reported Number of Applications Approved: Mailed Statewide Efficiency Not reported Not reported Percent of Kiosk Applications Approved Statewide Efficiency Not reported Not reported Percent of Mailed Applications Approved Statewide Efficiency Not reported Not reported Percent of Community Partner Applications Approved Statewide Not reported Not reported Not reported Percent of Community Partner Recertifications Approved Statewide Not reported Not reported Not reported Percent of Call Center Applications Approved Statewide Not reported Not reported Not reported Percent of Call Center Recertifications Approved Statewide Not reported Not reported Not reported Average Benefit Amount (for those eligible) Statewide Efficiency Not reported Not reported Number of Applications Approved: Kiosk Statewide Efficiency Not reported Not reported Number of Applications Approved: Community Partner Statewide Not reported Not reported Not reported Number of Recertifications Approved: Community Partner Statewide Not reported Not reported Not reported Number of Applications Approved: Call Center Statewide Not reported Not reported Not reported Number of Recertifications Approved: Call Center Statewide Not reported Not reported Not reported Number of Applications Approved: Face-to-Face Interviewed Statewide Not reported Not reported Not reported Number of Recertifications Approved: Face-to-Face Interviewed Statewide Not reported Not reported Not reported Number of Applications Approved: Phone Interviewed Statewide Not reported Not reported Not reported New York (continued) A.150 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Recertifications Approved: Phone Interviewed Statewide Not reported Not reported Not reported Number of Applications Denied: Paper Submissions to Local Office Statewide Efficiency Not reported Not reported Number of Recertifications Denied: Paper Submissions to Local Office Statewide Efficiency Not reported Not reported Number of Applications Denied: Online Statewide Efficiency Not reported Not reported Number of Recertifications Denied: Online Statewide Efficiency Not reported Not reported Percent of Complete Applications Denied Statewide Accuracy & Integrity, Efficiency Not reported Not reported Percent of Complete Recertifications Denied Statewide Accuracy & Integrity, Efficiency Not reported Not reported Percent of Initial Applications Denied Statewide Accuracy & Integrity, Efficiency Not reported Not reported Percent of Recertifications Denied Statewide Accuracy & Integrity, Efficiency Not reported Not reported Number of Recertifications Denied: Mailed Statewide Efficiency Not reported Not reported Number of Applications Denied: Mailed Statewide Efficiency Not reported Not reported Percent of Online Applications Denied for Failure to Submit Documentation Statewide Efficiency Not reported Not reported Percent of Online Applications Denied for Failure to Complete Interview Statewide Not reported Not reported Not reported Number of Applications Denied: Kiosk Statewide Efficiency Not reported Not reported Number of Applications Denied: Community Partner Statewide Not reported Not reported Not reported Number of Recertifications Denied: Community Partner Not reported Not reported Not reported Number of Applications Denied: Call Center Statewide Not reported Not reported Not reported New York (continued) A.151 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Recertifications Denied: Call Center Statewide Not reported Not reported Not reported Percent of Applicants that Appear Eligible for Expedited Benefits Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Applications with Incomplete Information Statewide Efficiency Not reported Not reported Number of Cases with Complete Verification Submitted Statewide Efficiency Not reported Not reported Percent of Recertifications with Incomplete Information Statewide Efficiency Not reported Not reported Number of Recertifications: Community Partner Statewide Not reported Not reported Not reported Number of Applications: Call Center Statewide Not reported Not reported Not reported Number of Recertifications: Call Center Statewide Not reported Not reported Not reported Number of Applications: Faxed Statewide Not reported Not reported Not reported Number of Recertifications: Faxed Statewide Not reported Not reported Not reported Percent of Applications: Faxed Statewide Not reported Not reported Not reported Percent of Recertifications: Faxed Statewide Not reported Not reported Not reported Percent of Applications: Call Center Statewide Not reported Not reported Not reported Number of Applications: Kiosk Statewide Efficiency Not reported Not reported Number of Applications: Mailed Statewide Efficiency Not reported Not reported Percent of Applications: Kiosk Statewide Efficiency Not reported Not reported Percent of Applications: Paper Submission to Local Office Statewide Efficiency Not reported Not reported Percent of Applications: Online Statewide Efficiency Not reported Not reported Percent of Applications: Community Partner Statewide Efficiency Not reported Not reported New York (continued) A.152 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applications: Paper Submission to Local Office Statewide Efficiency Not reported Not reported Number of Applications: Online Statewide Efficiency Not reported Not reported Total Number of Applications Started Statewide Not reported Not reported Not reported Total Number of Applications Completed Statewide Efficiency Not reported Not reported Number of Applications: Community Partner Statewide Efficiency Not reported Not reported Percent of Applications: Mailed Statewide Efficiency Not reported Not reported Number of Recertifications: Paper Submission to Local Office Statewide Efficiency Not reported Not reported Number of Recertifications: Mailed Statewide Efficiency Not reported Not reported Number of Requests for Assistance\/One-page Applications Statewide Not reported Not reported Not reported Percent of Applications Not Completed Statewide Not reported Not reported Not reported Percent of Recertifications: Paper Submission to Local Office Statewide Efficiency Not reported Not reported Percent of Recertifications: Mailed Statewide Efficiency Not reported Not reported Percent of Recertifications: Community Partner Statewide Not reported Not reported Not reported Percent of Recertifications: Call Center Statewide Not reported Not reported Not reported Total Number of Recertifications Started Statewide Not reported Not reported Not reported Total Number of Recertifications Completed Statewide Efficiency Not reported Not reported Average Processing Time of Application Decision Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Average Processing Time of Recertification Decision Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported New York (continued) A.153 Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Description: General: New York is state supervised and county administered. The state use two different eligibility systems; one is in use in one region (New York City) and the other in the rest of the state. The online system was fairly new at the time of our data collection, so most applications were still coming in on paper. One region runs a change center that only handles outgoing calls for interviews. Another county runs a call center. Numerous measures are available statewide through the Commissioner's Dashboard and myWorkspace. Measurement Goals: The local agencies participating in the study primarily use measurement for staffing purposes and to ensure they are processing applications according to state standards. Call Center: The regional call center in New York City is a change center only. It handles outgoing interview calls, and therefore does not track or need any measures of incoming call volume or wait time. Another county has its own call center, serving only its county, which is a sophisticated voicemail system that routes calls as necessary. If more than 16 calls are in progress, new callers will receive a busy signal. The phones flash at worker desks when the call volume gets high (more than five calls in the queue) so that others will pick up phone calls and keep the number in the queue lower. The system can produce a few limited reports, but the data cannot be easily transferred into spreadsheets for further processing. The county is upgrading its system. Online System: The online system, provided by a contractor, allows individuals to submit signed applications and changes, and to check their account balance. The information automatically transfers to the eligibility system. New York City has its own, separate online application portal and they collect their own measures\/data independently of the state. Document Imaging: New York has document imaging in place. Some counties package their documents and send to an offsite state location for imaging, and some counties (including New York City) have their own document imaging capability. In New York City, where SNAP offices have established a paperless office system, staff expressed a wish for clients to be able to scan their own documents and attach the images to their account. Kiosks: Kiosks are relatively new, with only a few counties providing them for applicants. Most kiosks are in the lobby of the local agencies, and they must be located somewhere that there are staff available. Waiver of Face-to-Face Interview: The state is not reporting any information about the waiver, though they do have a place to record when the interview was conducted by phone, face-to-face, or in the home. Shortened Interviews: One local office in New York reported that they have a shortened interview process, but did not provide us with further information. New York (continued) A.154 Online Expedited Applications: The state is incorporating expedited screening questions into the online application. Application Tracking: The state and local agencies are tracking a variety of measures related to applications received, approved, and denied; the help received; and possible difficulties with the application. One local office was sued by an advocacy group for not processing applications in a timely way. Since then that agency has focused on the number of applications, whether the applications are submitted online or on paper, and timeliness of processing. Changes Over Time: At the time of the survey the state was transitioning to a new online application system and new interface for worker processing of applications. One local office we spoke to was working on improvements to the call center and phone system that would enhance their internal tracking capabilities. Desired or Planned Future Measures: The state plans to track applications submitted online versus by other means in 2011. One region would also like to track more related to the telephone interviews, such as the timeliness of the calls and the number of calls required to reach the applicant; they are seeking funding to put this in place. Partnering: Most partnering in the state is related to application assistance. One county we interviewed requires that the partner provide a shortcut to the application on their computers. Counties make arrangements with individual partners, documenting confidentiality and application procedures. Once the partner has a login ID for the system, the partner can see limited information, such as whether an application was approved or denied and reports of the number of applications submitted, pending, approved, and denied, and the denial reasons. Partners do not have access to the eligibility system, and are not paid for application assistance services. The state and counties do not generally require the partners to track anything, though they do track the number of applications coming from partners. Other, informal partners provide assistance now that the state has an online application. Measures Collected by Partners: Partners may collect their own counts of the number of clients served, and have the ability to look at which applications were approved and denied. However, the state and local agencies we spoke to do not require them to collect specific measures. A.155 Profile: North Carolina State Administered or County Administered: County Number of Local\/County Agencies Interviewed: 3 Number of Partners Interviewed: 0 Initiatives Active in State: Call Center, Online System, Document Imaging, Kiosks, Partnering, Waiver of Face\u2010to\u2010 Face Interview, Shortened Interviews, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Statewide) Number of Calls (in flow) Statewide Not reported Not reported Not reported Number of Calls Answered \/Handled Statewide Not reported 11,150 Not reported Number of Calls Abandoned Statewide Not reported Not reported Not reported Total Calls Busy Statewide Not reported Not reported Not reported Percent of Total Calls Handled Statewide Not reported Not reported Not reported Percent Calls Abandoned Statewide Not reported No No Percent of Calls Busy Statewide Not reported Not reported Not reported Average Call Duration Statewide Not reported No No Average Answer Speed Statewide Not reported Not reported Not reported Average Time Prior to Abandonment Statewide Not reported Not reported Not reported Percent of Calls Completed in the Computer Phone System Statewide Not reported No No Number of Transactions Locally None No No Number of LEP (language) Calls Locally None No No Percent of Needs Met through Service Referral or Actions Taken Locally None 84 percent No Average Customer Satisfaction Rating Locally Customer Service 84 percent No Number of Referrals Made Locally None 1,529 Not reported Number of Trips Scheduled Locally None 5,885 Not reported Call Response Locally Efficiency 71 percent No Percent of Changes Processed Locally Not reported Not reported Not reported Number of Changes Processed Locally Not reported Not reported Not reported North Carolina (continued) A.156 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Average Waiting Time (to Speak to Agent) Locally Customer service No No Average Calls Handled Locally Not reported Not reported Not reported Average Staff on Phones Locally Not reported Not reported Not reported Average Queue Time Locally Customer service No No Average Time to De- Queue Locally Not reported Not reported Not reported Average After-Call Work Time by Staff Locally Not reported Not reported Not reported Percent of Queued Calls Handled Locally Not reported Not reported Not reported Number of Calls Queued Locally Not reported Not reported Not reported Number of Agents Assigned to Phones Locally Not reported Not reported Not reported Number of Changes Received Locally Not reported Not reported Not reported Number of Calls Received Locally Not reported Not reported Not reported Number of Hang-ups Locally Not reported Not reported Not reported Number of Calls Not Completed in Computer Phone System Locally Not reported Not reported Not reported Number of Calls Completed in Computer Phone System Locally Not reported Not reported Not reported Number of Calls Transferred to Agent Locally Not reported Not reported Not reported Average Hold Time Locally Not reported Not reported Not reported Average Time to Process Change Locally Not reported Not reported Not reported ONLINE SYSTEM (Statewide) Number of Online Screenings Completed: Online Statewide Program Access Not reported Not reported Number of Applicants Screened as Potentially Eligible Statewide Program Access Not reported Not reported Total number of Website Hits per Month Statewide Not reported Not reported Not reported Number of One-Page Requests for Assistance Submitted Locally Not reported Not reported Not reported North Carolina (continued) A.157 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives DOCUMENT IMAGING (Regionally) Number of Documents Received Locally Not reported Not reported Not reported Number of Documents Received: Mail Locally Not reported Not reported Not reported Number of Documents Processed Locally Not reported Not reported Not reported Number of Documents Scanned Locally Not reported Not reported Not reported Number of Tasks Completed Locally Not reported Not reported Not reported Number of Alerts Processed Locally Not reported Not reported Not reported KIOSKS (Statewide) None PARTNERING (Regionally) Number of Applications Received per Partner Locally Not reported Not reported Not reported Number of Partners Locally Not reported Not reported Not reported Number Accepting Applications Locally Not reported Not reported Not reported Number Providing Application Assistance Locally Not reported Not reported Not reported WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Number of Applicants Requesting a Phone Interview Locally Not reported Not reported Not reported Total Number of Interviews Locally Not reported Not reported Not reported Notice of Missed Interview Mailed Locally Not reported Not reported Not reported SHORTENED INTERVIEWS (Regionally) None APPLICATION TRACKING (Statewide) Percent of Total Applications Approved Statewide Not reported No No Percent of Total Recertifications Approved Statewide Not reported Not reported Not reported North Carolina (continued) A.158 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Average Benefit Amount (for those eligible) Statewide Not reported Not reported Not reported Number of Applications Approved: Paper Submission to Local Office Statewide Not reported Not reported Not reported Percent of Applications Approved After Paper Submission to Local Office Statewide Not reported Not reported Not reported Total Number of Applications Completed Statewide Program Access, Customer Service No No Number of Applications Denied: Paper Submissions to Local Office Statewide Not reported No No Number of Recertifications Denied: Paper Submissions to Local Office Statewide Not reported No No Average Processing Time of Recertification Decision Statewide Efficiency, Customer Service FNS timeliness standards NC has received FNS timeliness awards Average Processing Time of Application Decision Statewide Efficiency, Customer Service One local office asks employees to have approvals processed no later than the 25th day. Another local office reported that 98 percent will be processed within 9 days (regular applications) and 99 percent will be processed within 2 work days (emergency applications). Bonus money from FNS. One local office: director challenged department to process applications more quickly Number of Recertifications Approved: Paper Submission to Local Office Locally Not reported Not reported Not reported North Carolina (continued) A.159 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Recertifications Approved After Paper Submission to Local Office Locally Not reported Not reported Not reported Percent of Mailed Applications Approved Locally Not reported Not reported Not reported Percent of Mailed Recertifications Approved Locally Not reported Not reported Not reported Number of Applications Approved: Face-to-Face Interviewed Locally Not reported Not reported Not reported Number of Recertifications Approved: Face-to-Face Interviewed Locally Not reported Not reported Not reported Number of Applications Approved: Phone Interviewed Locally Not reported Not reported Not reported Number of Recertifications Approved: Phone Interviewed Locally Not reported Not reported Not reported Percent of Cases Reviewed that are Accurate Locally Not reported Food assistance staff will maintain an accuracy rate of 98 percent No Total Number of Applications Started Locally Not reported Not reported Not reported Total Number of Recertifications Completed Locally Not reported Not reported Not reported Number of Applications: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Applications: Faxed Locally Not reported Not reported Not reported Number of Applications: Mailed Locally Not reported Not reported Not reported Number of Requests for Assistance\/One-page Applications Locally Not reported Not reported Not reported Percent of Applications Not Completed Locally Not reported Not reported Not reported Total Number of Recertifications Started Locally Not reported Not reported Not reported North Carolina (continued) A.160 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Recertifications: Paper Submission to Local Office Locally Not reported Not reported Not reported Percent of Initial Applications Denied Locally Not reported No No Percent of Recertifications Denied Locally Not reported No No Percent of Complete Applications Denied Locally Not reported Not reported Not reported Percent of Complete Recertifications Denied Locally Not reported Not reported Not reported Percent of One- Page\/Requests for Assistance Denied Locally Not reported Not reported Not reported Percent of Applications with Incomplete Information Locally Not reported Not reported Not reported Percent of Recertifications with Incomplete Information Locally Not reported Not reported Not reported Number of applications pending decision Locally Not reported Not reported Not reported Number of initial applications approved over 30 days Locally Not reported Not reported Not reported Number of recertifications approved over 30 days Locally Not reported Not reported Not reported Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. ePASS = Electronic Pre-Assessment Screening Service Description: General: North Carolina has an online system with electronic screening assessment and paper application available for download. They also operate a state call center solely for providing information about benefits. Late in 2010, the state obtained the waiver of the face-to-face interview. Local offices mange their own document imaging, call centers, and partnerships with community organizations; implementation varies by county. Measurement Goals: North Carolina typically did not describe measurement goals for their initiatives, but customer service, program access, and accuracy were goals for some measures. North Carolina (continued) A.161 Call Center: North Carolina operates a statewide call center for inquiries on electronic benefits only. They track several performance measures, including percentage of calls handled, abandoned, calls busy, average answer speed, average call duration, and average time prior to abandonment. Local counties operate separate call centers for case management, and implementation varies by locality. Two local county offices interviewed operated call centers. In one county, the call center operations are divided into two sections, consisting of technical workers who answer questions related to SNAP and call center workers who route calls to the correct location. This call center is fairly new and began operating in September 2010. Another county office interviewed for this study operates a call center for all Department of Social Services programs. The call center, as well as the measures and benchmarks were designed by consultants and based on industry standards. Supervisors monitor performance data in real time for staffing purposes and to ensure operations are running smoothly. Real time data are available for the number of callers waiting in each queue, wait time, abandonment, and status of individual workers. Monthly call center reports track measures and whether the office meets the associated benchmark, including call response efficiency, percentage of needs met through referral or action, and percentage customer satisfaction rating. The office reported they are unable to measure incoming call volume due to an issue with both the software and phone server capabilities. Online System: North Carolina has an online system with electronic screening assessment (ePASS) and paper application available for download. At this time, North Carolina does not have electronic application submission. All online applications must be printed and sent to an office via mail, fax, or drop-off. According to the state, information entered in the ePASS screening pre-populates the application. At the state level, North Carolina tracks the number of ePASS screenings completed, number of potentially eligible applications from ePASS screenings, and number of website hits. At the local level, at least one county tracks the number of one-page requests for assistance submitted and the number of applications downloaded. Document Imaging: Document imaging operations in North Carolina are local office specific and implementation varies from county to county. As a result, scanned documents are stored in county- specific systems and there is no cross county access at this time. At least one county tracks performance data, including the number of documents received, and processed. Workers manually track performance data using a spreadsheet. Documents are also coded by the date and as part of an application, recertification, or other. Kiosks: According to the state, some Department of Social Services county offices have computers for completing, scanning, and printing applications. Any performance data tracking would be at the county level. None of the county offices interviewed as a part of this study reported having kiosks. Waiver of Face-to-Face Interview: North Carolina obtained the waiver of the face-to-face interview in late 2010. The type of interview (telephone versus face-to-face) is recorded for the purposes of scheduling and the narrative section of each case. If this information is needed, it must be manually retrieved from each case; there is no easy or quick way to compile statistics. Performance data is tracked at the local level and no data on the waiver is sent to the state. One local office interviewed tracked the Partnering: Some local offices partner with community organizations. No data are sent to the state for monitoring. One county office interviewed tracks the number of partners, number of partners accepting applications, and the number of partners providing application assistance. For partners that accept applications, the local office tracks the number of applications received per partner. Performance data are tracked manually and staff log all applications collected off-site on a spreadsheet. Measures Collected by Partners: We did not contact any of North Carolina's partners as part of this study. North Carolina (continued) A.162 phone interview requests for scheduling purposes. Another county office tracked the total number of interviews, but is unable to distinguish between those conducted over the phone and those conducted face-to-face. Shortened Interviews: North Carolina does not collect any performance data for the shortened interview process. According to one local office, abbreviated interviews are conducted with applicants that applied through The Benefit Bank of North Carolina (a screening tool offered by trained counselors at a range of community partner organizations). These applications are tracked on a log for the purposes of assigning the case to a worker. Online Expedited Applications: North Carolina does not accept expedited applications online. Application Tracking: North Carolina calculates a wide variety of application tracking measures related to applications and recertifications, including approvals, denials, average processing time, and average benefit amount for those eligible. Staff manually enter data into the eligibility system as they process applications. The eligibility system populates the North Carolina data warehouse monthly, but data and reports can be accessed as necessary. In general, county offices use state data to track performance, but may set their own internal benchmarks for performance (for example, two local offices had two different standards for processing time). North Carolina also tracks accuracy measures for the FNS QC process. This data is used to assess agency wide performance goals and to evaluate worker performance. Local county level program integrity teams track accuracy rates separately from the state. The state does not track applications by the method of submission. However, local offices report that they track application submission by drop-off, partner organization (for counties with partners), mail, and fax. Two local offices interviewed report they use a county-specific internal computer system to record whether applications were received in-person, fax, or mail. According to one county office interviewed, the state data system is cumbersome to work with and it is difficult to download data into a more usable format. Recently, this office started using an internally written program to assist them with downloading the data. Changes Over Time: The state office is currently implementing North Carolina Families Accessing Services Through Technology (NC Fast), which is a statewide case management and eligibility system. NC FAST Program consists of three main projects: online verification, service delivery interface, and case management. Four counties have been selected for the pilot program and total state wide rollout is planned for 2013. Document imaging is a new initiative in North Carolina. While functionality varies from county to county, localities are in general still in the early stages of implementation. Likewise, call centers were implemented at varying times throughout North Carolina. One county office interviewed recently established its call center in September 2010. The state runs the eligibility system, FSIS (Food Stamp Information System), but case management systems are run at the county level. Two local offices interviewed used internal case management systems, consisting of in-house computer systems developed by county IT departments, in addition to FSIS. The offices reported using the internal systems since February 2009 and early 2010, respectively. Internal system capabilities vary county to county, but in general the internal computer system stores narratives about the client, as well as household information. Staff log client visits, applications, and client interviews in the system. Scanned documents can also be uploaded and attached to client records. Counties can use the in-house system to produce reports on the number of application taken, number of applications received per day, method by which application was received (mail, fax, in person), and arrival time for in-person applicants. Local county offices report that they must perform some double data entry (into the internal case management system and the state eligibility system). North Carolina (continued) A.163 Desired or Planned Future Measures: North Carolina is interested in developing more measures about how online applications are disposed, especially as functionality increases with the use of NC FAST. Locally, counties reported a desire for additional performance measures. One local office would like to know the number of callers that try to reach the call center. This is currently not tracked due to a lack of call center software capabilities. They are also interested in monitoring the processing time for recertifications. Counties are still in the early phases of developing document imaging and electronic case files. Another county office interviewed indicated that while there are no specific performance measures planned currently, but this may change once the office uses these systems more. An additional respondent expressed interest in tracking agency error by the type of error. A.164 Profile: Ohio State Administered or County Administered: County Number of Local\/County Agencies Interviewed: 10 Number of Partners Interviewed: 0 Initiatives Active in State: Call Center, Online System, Document Imaging, Kiosks, Partnering, Wavier of Face\u2010to\u2010 Face Interview, Shortened Interviews, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Regionally) Average After-Call Work Time by Staff Locally Not reported Not reported Not reported Percent of Queued Calls Handled Locally Not reported Not reported Not reported Number of Calls Receiving Busy Signal\/Unable to Connect with Phone System Locally Not reported Not reported Not reported Average Time to Process Change Locally Not reported Not reported Not reported Number of Hang-ups Locally Not reported Not reported Not reported Number of Changes Processed Locally Not reported Not reported Not reported Percent of Calls Requesting Agent Locally Not reported Not reported Not reported Number of Calls Received Locally Not reported Not reported Not reported Average Answer Speed Locally Not reported No No Average Hold Time Locally Not reported No No Percent Calls Abandoned Locally Not reported No No Average Waiting Time (to Speak to Agent) Locally Not reported No No Average Queue Time Locally Not reported No No Average Time to De- Queue Locally Not reported No No Average Time Prior to Abandonment Locally Not reported No No Number of Calls Abandoned Locally Not reported No No Number of Calls Queued Locally Not reported No No Number of Calls Transferred to Agent Locally Not reported No No Average Call Duration Locally Not reported No No Percent of Calls Completed in the Computer Phone System Locally Not reported No No Ohio (continued) A.165 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Calls Completed in Computer Phone System Locally Not reported Not reported Not reported Percent of Total Calls Handled Locally Efficiency, Customer Service No No Percent of Changes Processed Locally Efficiency, Customer Service No No Number of Applications Processed: Call Center Locally Not reported Not reported Not reported Number of Changes Received Locally Not reported Not reported Not reported Total Number of Calls Handled Locally Not reported No No Number of Clients Requesting Case Status Locally Not reported Not reported Not reported Number of Clients Reporting Case Change Locally Not reported Not reported Not reported Number of Clients Reporting Verifications Turned In Locally Not reported Not reported Not reported Average Calls by Reason for the Call Locally Not reported Not reported Not reported Average Staff on Phones Locally Not reported No No Number of Agents Assigned to Phones Locally Accuracy & Integrity, Customer Service No No Number of Calls Answered\/Handled Locally Accuracy & Integrity, Customer Service No No Number of Calls (in flow) Locally Program Access, Efficiency, Customer Service Not reported Not reported Average Calls Handled Locally Program Access, Efficiency, Customer Service No No Number of Cases Pending Benefit Decision: Call Center Locally Not reported Not reported Not reported Number of Client Calls Returned Locally Not reported Not reported Not reported ONLINE SYSTEM (Statewide) Percent of Applications Received Statewide Not reported Not reported Not reported Average Number of Screenings Completed: Online Statewide Not reported Not reported Not reported Ohio (continued) A.166 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Online Screenings Completed: Online Statewide Not reported Not reported Not reported Number of First-Time Applications Approved Statewide Not reported Not reported Not reported Number of First-Time Applications Denied Statewide Not reported Not reported Not reported Average Number of Screenings Started: Online Statewide Not reported Not reported Not reported Average Number Started per Month Statewide Not reported Not reported Not reported Average Number Submitted per Month Statewide Not reported Not reported Not reported Number of Changes Started Statewide Not reported Not reported Not reported Average Number of Accounts Created per Month Statewide Not reported Not reported Not reported Number of Applications Started Statewide Not reported Not reported Not reported Number of Changes Submitted Statewide Not reported Not reported Not reported Number of Applications Downloaded Statewide Not reported Not reported Not reported Average Number of Changes Submitted per Month Statewide Not reported Not reported Not reported Average Level of Benefits Statewide Not reported Not reported Not reported DOCUMENT IMAGING (Regionally) Average Processing Time Locally Not reported Not reported Not reported Number of Documents Processed Locally Not reported Not reported Not reported Percent of Documents Scanned Locally Not reported 100 percent of documents need to be scanned. No Percent of Documents Received by Mail Locally Not reported No No Percent of Documents Received by Fax Locally Not reported No No Number of Documents Received Locally Not reported Not reported Not reported Ohio (continued) A.167 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Documents Received: Mail Locally Not reported Not reported Not reported Number of Documents Received: Fax Locally Not reported Not reported Not reported Number of Documents Received: Other Electronic Locally Not reported Not reported Not reported Number of Tasks Completed Locally Not reported Not reported Not reported Number of Alerts Processed Locally Not reported Not reported Not reported Number of Documents Scanned Locally Not reported One local office expects documents to be labeled and scanned within 24 hours of receipt. No KIOSKS (Regionally) Number of Uses to Submit Changes Locally Not reported Not reported Not reported PARTNERING (Statewide) Number of Partners Locally Not reported Not reported Not reported Number Accepting Applications Locally Not reported Not reported Not reported Number Providing Application Assistance Locally Not reported Not reported Not reported Percent of Partners Providing Application Assistance Locally Not reported Not reported Not reported Average Cost of Partner Arrangement Locally Not reported Not reported Not reported Number Clients Assisted per Partner Locally Not reported Not reported Not reported Number of Applications Received per Partner Locally Not reported Not reported Not reported Number with Terminals\/Kiosks to Submit Application Locally Not reported Not reported Not reported WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Percent of Interviews Conducted by Telephone Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Ohio (continued) A.168 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Interviews Conducted Face-to-Face Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Error Rate for Cases with Telephone Interviews Statewide Accuracy & Integrity, Efficiency No State: No One local office: Informal achievement award certificates and stickers. Error Rate for Cases with Face-to-Face Interviews Statewide Accuracy & Integrity, Efficiency No State: No One local office: Informal achievement award certificates and stickers. Total Number of Interviews Statewide Program Access, Efficiency, Customer Service No No Number of Redetermination Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Face-to-Face Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Telephone Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Applicants Requesting an In-Person Interview Locally Not reported Not reported Not reported Number of Home Visit Interviews Locally Not reported Not reported Not reported Number of Times Unable to Schedule Interview with Applicant Locally Not reported Not reported Not reported Number of Applicants who Did Not Answer to Type of Interview Requested Locally Not reported Not reported Not reported Percent of Telephone Interview Requests Honored Locally Not reported No No Number of Applicants Requesting a Phone Interview Locally Not reported Not reported Not reported Percent that are Missed and have Notice Mailed Locally Not reported No No Ohio (continued) A.169 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Notice of Missed Interview Mailed Locally Program Access, Efficiency, Customer Service Not reported Not reported Number of Missed Interviews Locally Program Access, Efficiency, Customer Service Not reported Not reported Average Length of Interview Locally Not reported No No SHORTENED INTERVIEWS (Regionally) Number of Shortened Interviews Locally Not reported Not reported Not reported ONLINE EXPEDITED APPLICATIONS (Statewide) Percent of Applications Approved: Online Expedited Statewide Not reported Not reported Not reported Percent of Applications Approved: Online All Statewide Not reported Not reported Not reported Number of Expedited Applications Received Online Statewide Not reported Not reported Not reported Number of Applications Approved: Online Expedited Statewide Not reported Not reported Not reported APPLICATION TRACKING (Statewide) Percent of Total Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Average Benefit Amount (for those eligible) Statewide Program Access No No Number of Recertifications Approved: Faxed Locally Not reported Not reported Not reported Number of Applications Approved: Mailed Locally Not reported Not reported Not reported Number of Recertifications Approved: Mailed Locally Not reported Not reported Not reported Number of Applications Approved: Face-to-Face Interviewed Locally Not reported Not reported Not reported Number of Recertifications Approved: Phone Interviewed Locally Not reported Not reported Not reported Ohio (continued) A.170 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Online Recertifications Approved Locally Not reported Not reported Not reported Percent of Community Partner Recertifications Approved Locally Not reported Not reported Not reported Percent of Mailed Applications Approved Locally Not reported Not reported Not reported Percent of Total Recertifications Approved Locally Not reported Not reported Not reported Number of Applications Approved: Online Locally Not reported Not reported Not reported Number of Recertifications Approved: Online Locally Not reported Not reported Not reported Number of Recertifications Approved: Community Partner Locally Not reported Not reported Not reported Number of Applications Approved: Faxed Locally Not reported Not reported Not reported Percent of Online Applications Approved Locally Not reported Not reported Not reported Percent of Applications Approved After Paper Submission to Local Office Locally Not reported Not reported Not reported Percent of Recertifications Approved After Paper Submission to Local Office Locally Not reported Not reported Not reported Percent of Community Partner Applications Approved Locally Not reported Not reported Not reported Percent of Call Center Applications Approved Locally Not reported Not reported Not reported Percent of Call Center Recertifications Approved Locally Not reported Not reported Not reported Percent of Mailed Recertifications Approved Locally Not reported Not reported Not reported Number of Recertifications Approved: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Applications Approved: Community Partner Locally Not reported Not reported Not reported Ohio (continued) A.171 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Recertifications Approved: Call Center Locally Not reported Not reported Not reported Number of Recertifications Approved: Face-to-Face Interviewed Locally Not reported Not reported Not reported Number of Applications Approved: Call Center Locally Not reported Not reported Not reported Number of Applications Approved: Phone Interviewed Locally Not reported Not reported Not reported Number of Applications Approved: Paper Submission to Local Office Locally Not reported Not reported Not reported Percent of Recertifications: Paper Submission to Local Office Locally Not reported Not reported Not reported Percent of Recertifications: Online Locally Not reported Not reported Not reported Percent of Recertifications: Community Partner Locally Not reported Not reported Not reported Percent of Applications: Mailed Locally Not reported Not reported Not reported Percent of Recertifications: Mailed Locally Not reported Not reported Not reported Number of Applications: Online Locally Not reported Not reported Not reported Number of Recertifications: Online Locally Not reported Not reported Not reported Number of Recertifications: Mailed Locally Not reported Not reported Not reported Number of Requests for Assistance\/One-page Applications Locally Not reported Not reported Not reported Number of Applications: Mailed Locally Not reported Not reported Not reported Number of Recertifications: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Recertifications: Community Partner Locally Not reported Not reported Not reported Ohio (continued) A.172 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applications: Faxed Locally Not reported Not reported Not reported Percent of Applications Not Completed Locally Not reported Not reported Not reported Total Number of Applications Started Locally Not reported Not reported Not reported Total Number of Recertifications Started Locally Not reported Not reported Not reported Number of Applications: Paper Submission to Local Office Locally Not reported Not reported Not reported Total Number of Applications Completed Locally Not reported Not reported Not reported Total Number of Recertifications Completed Locally Not reported Not reported Not reported Percent of Applications: Faxed Locally Not reported No No Percent of Recertifications: Faxed Locally Not reported Not reported Not reported Number of Applications: Community Partner Locally Not reported Not reported Not reported Percent of Applications: Paper Submission to Local Office Locally Not reported No No Percent of Applications: Online Locally Not reported No No Percent of Applications: Community Partner Locally Not reported No No Percent of Applications: Call Center Locally Not reported No No Percent of Recertifications: Call Center Locally Not reported Not reported Not reported Number of Applications: Call Center Locally Not reported Not reported Not reported Number of Recertifications: Call Center Locally Not reported Not reported Not reported Number of Recertifications: Faxed Locally Not reported Not reported Not reported Percent of Complete Applications Denied Statewide Program Access, Accuracy & Integrity, Customer Service No No Percent of Complete Recertifications Denied Statewide Program Access, Accuracy & Integrity, Customer Service No No Ohio (continued) A.173 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Initial Applications Denied Statewide Program Access, Accuracy & Integrity, Customer Service No No Percent of Recertifications Denied Statewide Program Access, Accuracy & Integrity, Customer Service No No Percent of Online Applications Denied for Failure to Submit Documentation Locally Not reported Not reported Not reported Percent of Online Recertifications Denied for Failure to Submit Documentation Locally Not reported Not reported Not reported Number of Applications Denied: Paper Submissions to Local Office Locally Not reported Not reported Not reported Number of Recertifications Denied: Paper Submissions to Local Office Locally Not reported Not reported Not reported Number of Recertifications Denied: Online Locally Not reported Not reported Not reported Number of Recertifications Denied: Community Partner Locally Not reported Not reported Not reported Number of Applications Denied: Faxed Locally Not reported Not reported Not reported Number of Recertifications Denied: Faxed Locally Not reported Not reported Not reported Number of Recertifications Denied: Mailed Locally Not reported Not reported Not reported Number of Applications Denied: Online Locally Not reported Not reported Not reported Number of Applications Denied: Community Partner Locally Not reported Not reported Not reported Number of Applications Denied: Call Center Locally Not reported Not reported Not reported Number of Applications Denied: Mailed Locally Not reported Not reported Not reported Percent of Applicants that Appear Eligible for Expedited Benefits Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Ohio (continued) A.174 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Cases with Complete Verification Submitted Locally Not reported Not reported Not reported Average Processing Time of Application Decision Locally Not reported No No Average Processing Time of Recertification Decision Locally Not reported Not reported Not reported Number of Days Application Pending Locally Not reported Not reported Not reported Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data is listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Description: General: Ohio has implemented numerous modernization initiatives, primarily at the county level. Performance measurement differs among regions, with each county independently determining the information they will track. Counties have access to Ohio's online reporting tool, the Business Intelligence Channel (BIC) that collects performance data from the state's computerized systems (e-Gateway and central registry information system [CRIS-E]). Offices can use this statewide system to pull information regarding applications or see statewide or countywide data. Collabor8 is an inter-county collaborative among eight Ohio counties. The state requested an FNS waiver so that these counties could function as one project area. Participating counties will establish a memorandum of understanding among themselves and with the state. The memorandum will likely incorporate performance standards, including timeliness standards. Clients will be able to move among counties without having to reapply for SNAP benefits. The Collabor8 project is working towards one multi- county call center with a toll free number and there have been discussion of opening up a document imaging system across counties. Counties involved in the project are starting to see data and measures from the other counties within the collaborative. One local office began a performance measurement initiative to help mainly with their error rate. They have a QA unit, but as workers have left they have not been replaced. The QA workers used to review cases prior to authorization, and do a random pull of cases authorized the month before. Now they review intake cases prior to authorization, and do a random pull of cases prior to the month of recertification. While most offices interviewed were satisfied with their collection and use of performance measures and data, a few offices noted challenges. These challenges include: financial difficulties, system failures or glitches, and human error. One office commented that they may have chosen to pull too much information when designing their systems and reports. Measurement Goals: Ohio's performance measures touch on all four FNS goals. They also track performance data in order to monitor staffing, distribute work fairly and evenly, to discern whether clients are using certain initiatives and why, to identify when more training is needed, and for performance reviews. Ohio (continued) A.175 Call Center: Ohio has no statewide call center; each county has the capability to operate its own call center. Eight of the 10 county offices interviewed were developing or had already developed a call center. The characteristics and functionality of the call centers vary by county office. All local offices with call centers collect numerous performance measures and data. Online System: Ohio developed a statewide online application system for implementation in October 2010. Information on this initiative is gathered at the state level primarily, although some local offices track measures or counts of aggregate data. These include: number of applications downloaded, number of applications started, and number of changes submitted. Document Imaging: There is no statewide document imaging system, but approximately half of the counties interviewed have implemented a system. The type of system and its capabilities may differ among county offices. Multiple local offices collect information on their document imaging systems, with the majority focused on aggregate data rather than performance measures. Kiosks: Two of the 10 local offices interviewed in Ohio track kiosk measures. They look at the number of uses to submit changes and lobby wait time. The measure on lobby wait time is not included in the table above because that kiosk's primary function is managing client appointments, not application submission. At the time of the interview, the state had received ARRA funds to buy computer kiosks for each county office, and a few local offices interviewed had already set them up. The state was working to implement kiosks statewide. There are no plans to track applications submitted from these kiosks, but theoretically it is possible via the new online system initiative. Waiver of Face-to-Face Interview: Ohio has a statewide waiver in place, and both state and local offices collect a variety of aggregate data and performance measures on this initiative. Shortened Interviews: In some counties, applicants may receive a fast-track application. Online Expedited Applications: Three local offices from which we collected data record data on online expedited applications, including: percent of applications approved (all online and expedited online), number of expedited applications received online, and number of expedited online applications approved. Application Tracking: Ohio state and local offices collect a number of application tracking measures and data, including information on application accuracy, approvals and denials, application processing and case characteristics, application receipt, and application processing time. Whether the application source is recorded varies by office, with some offices not tracking this and others keeping detailed records. Partnering: Ohio has widespread formal and informal partnering arrangements. One of the state's most active partnerships operates on an annual grant agreement. Though established at the state level, this partner has volunteers that work in some Ohio counties. The partner provides community outreach, information distribution and application assistance using online application software they developed. The state and local offices maintain other partnerships that provide application assistance, station caseworkers at hospitals or community offices, set up outreach initiatives, or have modernized public relations through social networking. Measures Collected by Partners: The grant recipient has extensive capability to track performance but state staff did not know the specifics about its measurement capacity. One partnership provides their local office with the number of applications per month and the type of benefits received. Ohio (continued) A.176 Changes Over Time: Ohio has not changed its data collection or reporting in the past 12 months, beyond developing new measures along with new initiatives. One office has adjusted the phone system numerous times since they began measurement and has been changing the description of the outcome. Desired or Planned Future Measures: State and local officials noted many desired future measures and data, and some offices were in the process of designing and changing measures. The state would like to develop performance measures for the online system, but has not yet (at the time of the interview) nominated specific measures. One local office recommended measures specifically on the effectiveness of the online system. Another local office wants to track measures on the average lengths of case processing, data entry in the CRIS-E system, and mailings. As telephone interviews and initial applications via telephone become more established, one office would like to look at corresponding measures, especially on timeliness. Another is interested in having data on why clients request telephone interviews and client reactions to face-to-face versus telephone interviews. Measures on the number of times per worker that QC errors are found would be helpful to one local office. Two local offices currently include calls to voicemail in their calculation of average calls handled. One local office is concerned that this inclusion is inflating the numbers, and they would like to exclude these types of calls in the future. Finally, one local office would like performance data on the amount of money and time it takes to process a SNAP case. Some local offices had a desire to see reports from other states or other counties, such as: information on benefits, error rate performance measures, information tied to federal standards for timeliness and accuracy, and reasons why other Ohio counties are struggling with timeliness (including information on caseload sizes, worker to case ratio, and length of time to process a case). A.177 Profile: Oregon State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 2 Number of Partners Interviewed: 0 Initiatives Active in State: Online System, Partnering, Waiver of Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives ONLINE SYSTEM (Regionally) None PARTNERING (Regionally) None WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Percent that are Missed and have Notice Mailed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Error Rate for Cases with Telephone Interviews Statewide Accuracy & Integrity Not reported No Error Rate for Cases with Face-to-Face Interviews Statewide Accuracy & Integrity Not reported No Total Number of Telephone Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Redetermination Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Interviews Conducted by Telephone Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Interviews Conducted Face-to-Face Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No APPLICATION TRACKING (Statewide) None Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Description: General: In Oregon, SNAP is state-administered, divided into a number of districts and regions across the state. Each district develops a process to collect performance data, which the state can then access. Districts do not track information at the applicant level, but that information is collected in the Oregon (continued) A.178 case worker's narration. The district office-level is the greatest level of detail Oregon uses to assess program performance. The state noted that their technological capability is very limited. Measurement Goals: Oregon primarily measures performance to ensure customer service, accuracy and integrity, and program access. Call Center: Oregon does not have a call center. Online System: At the time of our data collection, Oregon did not have a statewide online system. One local office we interviewed reported that an online system was being tested in part of the state, with help from a partner agency, and that full rollout of the online system to the rest of the state was planned for spring 2011. Document Imaging: Oregon does not have document imaging. Kiosks: Oregon does not have kiosks. Waiver of Face-to-Face Interview: Oregon modeled its waiver after other states and followed guidance from the Western Region office. The state tracks the error rate for cases with telephone interviews, and if telephone interviews experience higher error rates, the state may conduct targeted training in problem sites or across the board if the problem were statewide. They may also issue statewide transmittals or reminders Shortened Interviews: Oregon does not have a shortened interview initiative. Online Expedited Applications: Oregon did not have the ability to accept expedited applications online at the time of our data collection. Application Tracking: The Oregon SNAP conducts targeted case reviews that are not required by or reported to FNS. Specifically, 16 field reviewers review their own random sample of cases from the field branches. This sample is not associated with the FNS QC sample. In this process, the state sanctions branch offices that are worse than the state average. Sanctions include corrective actions, such as additional training or reviews, to ensure that accuracy improves. Oregon has established this sanction as part of its corrective action plan to avoid sanctions from FNS, and the reviews are intended for internal use as a coaching tool to identify consistent problem trends at the field offices. For FNS QC, Oregon only collects information for the QC sample. This process covers some different performance measures than the quality control process, as well as some overlapping performance measures that they report they approach differently. Each month, quality control and quality assurance staff use several processes to examine all performance measures associated with their sample. One such process is a monthly video-conference, the Quality Assurance Panel Discussion, in which field offices are invited to participate. Policy and training representatives are also present. The discussion focuses on any problems identified in the current month of the sample, as well as how to prevent errors. The quality assurance process collects some of its performance measures for its sample with identifiers that allow them to separate applications by type. Specifically, quality assurance is able to tell which type of application was used (for example, an application for the Self Sufficiency office or an application for day care). Changes Over Time: The state established sanctions for the waiver initiative that has evolved over time. According to the state respondent, the state sanction has been very effective. Oregon now has its Partnering: Oregon does not have an initiative for partnering. However, one local office reported that partner agencies were assisting the state with piloting an online application tool. Measures Collected by Partners: No partner measures were reported. Oregon (continued) A.179 lowest error rate in over 10 years and has maintained an error rate below the national average for the past three to four years. Oregon had not previously ranked below the national average. Desired or Planned Future Measures: At the time of data collection, Oregon was piloting an online system. For 2011, they planned to track whether an application was filed online, but only for the quality control sample. Oregon staff reported that they would like to be able to track which applications were completed using the web-based application for the purpose of comparing accuracy across application type. Such an automated process would require resources beyond Oregon's current capacity. A.180 Profile: Pennsylvania State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 5 Number of Partners Interviewed: 7 Initiatives Active in State: Call Center, Online System, Document Imaging, Kiosks, Partnering, Waiver of Face\u2010to\u2010 Face Interview, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarksa Performance Incentives CALL CENTER (Statewide) Percent of Queued Calls Handled Statewide Efficiency, Customer Service 80 percent is the goal No Number of Calls (in flow) Statewide Efficiency, Customer Service No No Number of Calls Abandoned Statewide Efficiency, Customer Service No No Number of Calls Queued Statewide Efficiency, Customer Service No No Number of Calls Answered\/Handled Statewide Efficiency, Customer Service No No Number of Agents Assigned to Phones Statewide Efficiency, Customer Service No No Number of Calls Receiving Busy Signal\/Unable to Connect with Phone System Statewide Efficiency, Customer Service No No Average Queue Time Statewide Efficiency, Customer Service No No Average Time Prior to Abandonment Statewide Efficiency, Customer Service Not reported Not reported Average After-Call Work Time by Staff Statewide Efficiency, Customer Service 60 seconds is the goal. Some calls are more complex, but this should balance out. No Percent of Calls Requesting Agent Statewide Efficiency, Customer Service No No Number of Calls Transferred to Agent Statewide Efficiency, Customer Service No No Maximum Time to Answer Statewide Not reported Not reported Not reported Maximum Time to Abandon Statewide Not reported Not reported Not reported Total Number of Calls Unanswered Statewide Not reported Not reported Not reported Total Calls Receiving a Queue Full Message Statewide Not reported Not reported Not reported Number of Calls Answered in English (total) Statewide Not reported Not reported Not reported Pennsylvania (continued) A.181 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarksa Performance Incentives Number of Calls Answered in English (in service area) Statewide Not reported Not reported Not reported Number of Calls Answered in English (outside service area) Statewide Not reported Not reported Not reported Average Answer Speed Statewide Efficiency, Customer Service No No Average Hold Time Statewide Efficiency, Customer Service No No Percent Calls Abandoned Statewide Efficiency, Customer Service The goal is 20 percent or less. No Average Call Duration Statewide Efficiency, Customer Service The goal is to keep calls around the 5 minute range No Average Calls Handled Statewide Efficiency, Customer Service The goal is 8 calls per hour (used to be 7, and this was achieved so it was increased). No Percent of Total Calls Handled Statewide Efficiency, Customer Service The goal is 80 percent No Average Staff on Phones Statewide Efficiency, Customer Service The goal is no more that 25 percent of staff off the phones at one time No Number of Calls Received Locally Not reported No No ONLINE SYSTEM (Statewide) Average Number of Screenings Started: Online Statewide Program Access Not sure Not sure Average Number of Screenings Completed: Online Statewide Program Access Not sure Not sure Percent of Users Selecting Spanish Language Statewide Program Access Not sure Not sure Percent of Screened Individuals Potentially Eligible: Online Statewide Program Access Not sure Not sure Percent of Screenings Resulting in Application Submission: Online Statewide Program Access No Not sure Number of Online Screenings Started: Online Statewide Program Access Not sure Not sure Pennsylvania (continued) A.182 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarksa Performance Incentives Number of Online Screenings Completed: Online Statewide Program Access Not sure Not sure Number of Applications Submitted per Month Statewide Program Access No Not sure Number of Applications Started Statewide Program Access Not sure Not sure Number of Applications Submitted With All Application Questions Answered Statewide Program Access Not sure Not sure Number of Online Accounts Created Statewide Program Access Not sure Not sure Number of Log-ins Statewide Program Access Not sure Not sure DOCUMENT IMAGING (Statewide) Ratio of Staff to Documents Scanned Statewide Not reported Not reported Not reported Number of Documents Attached Statewide Not reported Not reported Not reported Percent of Documents Attached in a Timely Manner Statewide Not reported Not reported Not reported Number of Documents Scanned Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Documents Scanned But Not Attached Locally Efficiency No No PARTNERING (Statewide) Number of Screenings per Partner Statewide Program Access, Customer Service Grant payments to partners are based on meeting certain application goals. The goal is specific to the individual partner's contract. The incentive and sanction are tied to grant payments Pennsylvania (continued) A.183 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarksa Performance Incentives Number of Applications Received per Partner Statewide Program Access, Customer Service Grant payments to partners are based on meeting certain application goals. The goal is specific to the individual partner's contract. The incentive and sanction are tied to grant payments KIOSKS (Statewide) None WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Error Rate for Cases with Telephone Interviews Statewide Program Access, Accuracy & Integrity No No Total Number of Interviews Statewide Not reported Not reported Not reported Number of Face-to-Face Interviews Statewide Not reported Not reported Not reported Number of Telephone Interviews Statewide Not reported Not reported Not reported Number of Interview Walk-Ins, New Applications Statewide Not reported Not reported Not reported Number of Interview Walk-Ins, Pending Applications Statewide Not reported Not reported Not reported Average Total Interviews Per Worker Statewide Not reported Not reported Not reported ONLINE EXPEDITED APPLICATIONS (Statewide) None APPLICATION TRACKING (Statewide) Number of Applications Approved: Community Partner Statewide Program Access Not sure Not sure Percent of Total Applications Approved Statewide Program Access Not sure Not sure Percent of Community Partner Applications Approved Statewide Program Access No No Pennsylvania (continued) A.184 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarksa Performance Incentives Number of Expedited Applications Approved Statewide Not reported Not reported Not reported Percent of Applications: Online Statewide Program Access, Efficiency, Customer Service Not reported Counties are ranked according to the percentage of applications they receive online. Internal competition could be motivational among counties. Percent of Applications: Community Partner Statewide Program Access 30 percent target for percent of applications received through COMPASS Not sure Percent Change in Applications Received by County Statewide Program Access Not reported Not reported Number of Expedited Applications Pending Statewide Not reported Not reported Not reported Percent of Applications Not Completed Statewide Not reported Not reported Not reported Number of Applications: Paper Submission to Local Office Statewide Not reported Not reported Not reported Number of Applications: Mailed Statewide Not reported Not reported Not reported Number of Pending Applications Statewide Not reported Not reported Not reported Number of Resurrected Applications Statewide Not reported Not reported Not reported Number of Applications Received Statewide Not reported Not reported Not reported Number of Expedited Applications Denied Statewide Not reported Not reported Not reported Percent of Complete Applications Denied Statewide Program Access Not sure Not sure Number of Applications Denied: Community Partner Statewide Program Access No Not sure Pennsylvania (continued) A.185 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarksa Performance Incentives Number of Applications Denied by Denial Reason Statewide Not reported Not reported Not reported Number of SNAP Cases Closed by Closure Reason Statewide Not reported Not reported Not reported Percent of Initial Applications Denied Statewide Program Access, Efficiency, Customer Service Not sure Not sure Number of Expedited Applications Approved in 5 Days or Less Statewide Not reported Not reported Not reported Number of Expedited Applications Approved or Rejected in More Than 5 Days Statewide Not reported Not reported Not reported Average Processing Time of Application Decision Statewide Program Access, Efficiency, Customer Service State: Not sure One local office tracks the number of pending cases more than 30 days old by worker on a weekly basis. Not sure Percent of Approved Applications Processed Within 7, 15, 30, and 45 Days Statewide Efficiency Not reported Not reported Average Days to Process Approved Applications Statewide Efficiency Not reported Not reported Number of Expedited Applications Pended in 5 Days or More Statewide Not reported Not reported Not reported Number of Days Applications are Pending Statewide Not reported FNS sets a 30 day maximum Not reported Total Number of Pending Cases, by worker Statewide Not reported Not reported Not reported Pending Cases >30 Days, by Worker Statewide Not reported National standard of 30 day timeliness Not reported Number of Applicants Pre-Screened, by Worker Statewide Not reported Not reported Not reported Number of Cases Ready To Go, by Worker Statewide Not reported Not reported Not reported Number of Cases Ready To Go in <25 days, by Worker Statewide Not reported Not reported Not reported Pennsylvania (continued) A.186 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarksa Performance Incentives Number of Cases Ready To Go in >24 Days\/Resurrected, by Worker Statewide Not reported Not reported Not reported Percent of Applicants that Appear Eligible for Expedited Benefits Statewide Program Access, Customer Service Not sure Not sure Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data is listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported statewide. Measure descriptions are unavailable for the online application system, because these calculations are done by a contractor. Pennsylvania staff receive the measures, but do not know how they are determined. Pennsylvania has implemented a statewide call center. An additional call center that serves one specific region in Pennsylvania. aThis column indicates a performance goal. Pennsylvania does not have any official standards or benchmarks because of how collective bargaining agreements are structured. Description: General: The Pennsylvania state office leads all modernization efforts and performance measurement and its related data. Monthly performance reports are sent to the local offices, and further reports are available on Dashboard, an electronic task tracking system available to state staff, but some staff reported these can be cumbersome to use. One office cited the timeliness and error reports as the most valuable. One office noted that technological innovations and access to more data have placed an increased burden on staff, who are presumed to have regularly checked and used this information. Staff reported that even though there may be potentially useful performance measures available in state reports, local office staff must devote their full resources to keeping up with the caseload. Multiple local offices have divided the workload by task, wherein certain staff are responsible for a particular piece of the application processing, rather than being responsible for a specific set of cases. Partnering: Pennsylvania’s partnering arrangements are managed by the state office. Partners provide application assistance and work under a grant agreement that requires them to submit a certain number (which varies by partner) of applications per year. The Pennsylvania state office tracks two aggregate data items on their partnerships: number of screenings completed per partners and number of applications received per partner. Measures Collected by Partners: Pennsylvania partners collect several measures, including information on approvals and denials, screening and application volume, benefit types and amounts, conversion rate, processing time, error rates, and demographic or case characteristics. Pennsylvania (continued) A.187 Measurement Goals: Pennsylvania’s performance measures touch on all four FNS goals. They have also used certain measures for staff training and to test the effectiveness of initiatives, such as the online screening tool. Call Center: The Pennsylvania state office operates a call center, which is staffed by state staff and used in all but one county interviewed. The state tracks a wide variety of data, though they are more concerned with aggregate counts than measures. Local offices receive reports from the state documenting the number of calls from their county processed by the Change Center. These reports are available as needed, but not distributed regularly. In October 2010, one district worked with a vendor to establish a regional (district-wide) call center. Call center staff and the call center computer phone system compile lists of: number of calls (in flow), number of calls abandoned, number of calls queued, and number of calls received. Online System: The online system is called COMPASS. The state office receives performance reports on the online application system from their contractor. They review the percent of users selecting Spanish language, percent of screened individuals potentially eligible, percent of screenings resulting in application submission, the average number of screenings started and completed, and some sets of aggregate data. Document Imaging: Pennsylvania calculates the percent of documents attached in a timely manner and the ratio of staff to documents scanned. They also collect the number of documents attached and the number of documents scanned. One local office records the number of documents scanned but not attached. Kiosks: Pennsylvania does not collect any performance data on their kiosks. Kiosks have been set up in local offices for online application submission, viewing benefit amount, change reporting, or checking application status. Waiver of Face-to-Face Interview: The state office looks at the error rate for cases with telephone interviews. One local office reported that the state makes a few counts of aggregate data available to them. Shortened Interviews: Pennsylvania does not offer a shortened interview. Online Expedited Applications: Clients can submit expedited applications through the state’s online system, but Pennsylvania does not collect any performance data on their online expedited applications. Application Tracking: Pennsylvania reports numerous application tracking measures from the state office, in the areas of application accuracy, receipt, approvals\/denials, and timely processing. Changes Over Time: The contractor operating the computer phone system for the call center changed in 2010. Over the past year, the state has instituted a new requirement that partners use COMPASS. Before, COMPASS was optional and partners could also use paper applications. State staff reported the partnership initiative is easier for the state to manage when partners submit online application. Desired or Planned Future Measures: One local office we interviewed expressed an interest in seeing reports by county for all counties in Pennsylvania, especially QC reports, for comparison purposes. Pennsylvania county offices are evaluated in terms of effective management measures, such as application timeliness, QC errors, and valid client complaint frequency. One county office that participated in the study would like to calculate these measures internally and with greater regularity in order to Pennsylvania (continued) A.188 identify problems preemptively. This effort is limited by staffing constraints, particularly in light of recent staff turnover. They would also like to see reports showing the number of applications that are opened but not completed and are later resubmitted. These applications cause inefficient repetition of initial tasks for clients and staff. Having measures on this might lead to changes in the nature of the annual review or other approaches to caseload renewals. A.189 Profile: Rhode Island State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 0 Number of Partners Interviewed: 0 Initiatives Active in State: Online System, Document Imaging, Partnering, Waiver of Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives ONLINE SYSTEM (Statewide) Percent of Applications Started Online that Were submitted Statewide Program Access, Customer Service No No Number of Applications Submitted Statewide Program Access, Customer Service No No Number of Applications Started Statewide Program Access, Customer Service No No Number of Abandoned Applications Statewide Program Access, Customer Service No No Number of First-Time Applications Approved Statewide Program Access, Accuracy & Integrity, Customer Service No No DOCUMENT IMAGING (Statewide) Number of Documents Processed Statewide Accuracy & Integrity, Efficiency, Customer Service No No Number of Documents Scanned Statewide Accuracy & Integrity, Efficiency, Customer Service No No Number of Tasks Completed Statewide Not reported Not reported Not reported PARTNERING (Statewide) Percent of Partners Providing Application Assistance Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Percent that are Missed and have Notice Mailed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Number of Applicants Requesting a Phone Interview Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Redetermination Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Notice of Missed Interview Mailed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Rhode Island (continued) A.190 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applicants Requesting a Phone Interview Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Applicants Requesting an In-Person Interview Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No APPLICATION TRACKING (Statewide) Percent of Online Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Total Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Total Recertifications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Average Benefit Amount (for those eligible) Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Applications Processed: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service 100 percent of the potentially eligible population- for both paper and electronic applications must be processed No Number of Recertifications Processed: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Applications Approved: Online Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Total Number of Applications Started Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Total Number of Applications Completed Statewide Accuracy & Integrity, Efficiency, Customer Service No No Total Number of Recertifications Completed Statewide Program Access, Accuracy & Integrity, Customer Service No No Percent of Applications: Community Partner Statewide Efficiency Not sure Not sure Number of Applications: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Rhode Island (continued) A.191 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applications: Online Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Applicants that Appear Eligible for Expedited Benefits Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Average Processing Time of Application Decision Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service FNS standard: Within 7 days for cases screened as expedited and within 30 days for non- expedited cases. FNS awards bonuses based on timeliness. Average Processing Time of Recertification Decision Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service FNS standard: Within 7 days for cases screened as expedited and within 30 days for non- expedited cases. FNS awards bonuses based on timeliness. Notes: State collects mandatory accuracy data through QC reviews. Description: General: Rhode Island has a state-administered SNAP program. The state used ARRA funds to support some of its modernization initiatives. Staff reported that a new data warehouse has recently streamlined the state’s reporting abilities. Measurement Goals: Rhode Island primarily measures performance to ensure customer service, accuracy and integrity, and program access and efficiency. Call Center: Rhode Island does not have a call center. Online System: ARRA funds supported some systems development for online applications. Previously, workers had to execute multiple steps to convert information from the contractor-operated online application to the eligibility system. The systems changes have automated some steps in this process. Document Imaging: ARRA funds completely covered the purchase of new fax\/copier\/scanners for each of the offices, and the state reports the initiative would not have been implemented without the funds. Rhode Island indexes all of its scanned files directly into its processing system a procedure they report is both time consuming and necessary. The state is re-evaluating the best way to execute document imaging. Typically, workers use desktop scanners to scan a file and index it to a particular client, recording the document type in the system. Scanned documents are stored in a data warehouse and accessed through a drop down box in the eligibility system. This information is integrated with the electronic case management system. Kiosks: Rhode Island does not have kiosks for application access and submission. Rhode Island (continued) A.192 Waiver of Face-to-Face Interview: ARRA funds supported an upgrade to a digital telephone system, which the state reports would not have been implemented without the use of stimulus funds. On an ad-hoc basis, Rhode Island could request a breakdown of phone versus face-to-face interviews. However, the agency has not done so due to the prohibitive associated costs. Eventually, they hope the new data warehouse will enable internal staff to generate ad-hoc reports at little to no additional cost. No automatically generated report shows whether interviews were missed by type of interview (telephone versus face-to-face), but those data elements are in the system and could be extracted through an ad-hoc report. Shortened Interviews: Rhode Island does not have a shortened interview process. Expedited Online Applications: Rhode Island does not have an expedited online application initiative. Application Tracking: Rhode Island does not have an internal standard for timeliness, but aims to outperform the FNS standard to avoid another court case. The state regularly tracks only the percentage of interviews that are missed and have a notice mailed. A recent federal lawsuit addressed application timeliness in Rhode Island. Rhode Island’s eligibility system includes three different sections: 1) application panels (screening information); 2) statistic panels (interview information); and 3) eligibility panels (benefit calculation). Rhode Island’s old process took the information from the statistic panels, which use interview information, to track timeliness of expedited cases. As a result of the lawsuit, the state changed its focus to the application panels, which use screening information to note whether a case is expedited. The changed system now calculates expedited processing time by starting the processing time clock when a case is determined expedited (which may be when the client is interviewed for a case that did not screen in as expedited). This resolved the issue of some expedited cases appearing untimely when in fact they had been determined expedited after the initial seven-day window had elapsed. Changes Over Time: Rhode Island’s eligibility system uses 30-year-old technology, but most data are now transferred from that eligibility system into the new data warehouse, making ad-hoc reports more accessible. The respondent hopes that a range of employees will be trained to generate ad hoc reports, not just the technology staff who currently handle this. Desired or Planned Future Measures: The state would like to reward offices and individuals for outstanding accuracy but does not have the funding to do so. Individuals without errors in the quality control sample receive a certificate of appreciation. To improve efficiency, Rhode Island would standardize the process of entering applications into the eligibility system. Partnering: Rhode Island has one formal, contractual partner that conducts SNAP outreach on behalf of the state: the University of Rhode Island. The University has annual goals delineated in the contract, and the respondent believes compensation was not dependent on the achievement of those goals or of a certain output. The respondent was unable to provide any further specific details about performance measurement for the University. Rhode Island also has informal partnerships with Community Action Programs via an electronic bridge that transfers data elements from the community action program agencies’ systems into the SNAP online application. That information is reviewed by SNAP staff before being transferred to the SNAP eligibility system. Measures Collected by Partners: We did not contact Rhode Island’s partners as part of this study. A.193 Profile: South Carolina State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 0 Number of Partners Interviewed: 0 Initiatives Active in State: Call Center, Online System, Document Imaging, Partnering, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Statewide) Average Answer Speed Statewide Program Access, Efficiency, Customer Service 5 minutes. Selected because it is the industry standard No Percent Calls Abandoned Statewide Program Access, Efficiency, Customer Service No No Average Call Duration Statewide Customer Service No No Average Waiting Time (to Speak to Agent) Statewide Program Access, Efficiency, Customer Service 5 minutes. Selected because it is the industry standard No Average Calls Handled Statewide None Not sure Not sure Average Time Prior to Abandonment Statewide None Not sure Not sure Average After-Call Work Time by Staff Statewide Program Access Not sure Not sure Number of Calls (in flow) Statewide Program Access, Efficiency, Customer Service Not sure Not sure Number of Calls Abandoned Statewide Customer Service Not sure Not sure ONLINE SYSTEM (Statewide) Percent of Applications Received Statewide Program Access, Customer Service Not sure Not sure Average Number Submitted per Month Statewide Program Access, Customer Service Not sure Not sure DOCUMENT IMAGING (Statewide) Number of Documents Scanned Statewide Customer Service No No PARTNERING (Statewide) Number of Partners Statewide Program Access, Efficiency, Customer Service No Not sure Number Accepting Applications Statewide Program Access, Efficiency, Customer Service No Not sure Number Providing Application Assistance Statewide Program Access, Efficiency, Customer Service No Not sure South Carolina (continued) A.194 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives APPLICATION TRACKING (Statewide) Number of Applications: Online Statewide Program Access, Customer Service No Not sure Number of Recertifications: Mailed Statewide Efficiency, Customer Service Not sure Not sure Notes: State collects mandatory accuracy data through QC reviews. Description: General: In South Carolina, SNAP is state-administered. They have a call center, online system, conduct document imaging, and work with community partners. Measurement Goals: South Carolina primarily measures performance to ensure customer service and program access. Staff reported concern about implementing incentives associated with performance measures. For example, an incentive related to minimizing call duration may have unintended consequences, such as decreased quality of customer service. Call Center: The call center is operated in house and uses the Avaya tracking system. The Avaya call center system is not interfaced with the eligibility system. Call center measures have been in place for two years, and the reports used by the state came packaged with the call center software. One limitation of the current system is that call center agents can sometimes be logged in and ready to receive calls, but the call center software does not recognize that they are logged in. Furthermore, the call center system places restrictions on queued calls. The queue capacity is 120, and the system cuts off additional calls from entering the queue at this point. The system also prevents calls from entering the queue after 4:15 p.m., in order to allow staff to answer all calls remaining in the queue before their workday ends. ARRA funding was used to put extra call center agents on the phones. According to staff interviewed, the demand for increased call center capacity demonstrates a need to maintain a larger phone center staff, even when the funds are exhausted. Online System: The online application system is one year old. The online system is not currently advertised to potential applicants. Staff reported they are surprised by the volume of website hits. Applicants are required to provide (at a minimum) name, address and signature when applying online. Document Imaging: Staff reported that large county offices sometimes struggle with the document imaging process and find it to be a burden. The system counts and tracks the number of documents imaged. Transfer of data on an application first goes to an image, but then the image’s data must be manually entered. Scanned images and cases share numbers, but there is no electronic link between them in the system. Local databases have labels and are assigned case numbers, but they are not linked Partnering: Partnering for application submission is pending because of a grant, but that application access is not yet implemented. South Carolina has two formal contracts. One of the partners is reimbursed quarterly based on the number of the applications they submit. Measures Collected by Partners: We did not contact either partner as part of this study. South Carolina (continued) A.195 to the eligibility system in any way. The documents in the system are available to all the workers, except for folders deemed confidential. Kiosks: South Carolina does not have kiosks. Waiver of Face-to-Face Interview: South Carolina reported that they do not have a waiver of the face-to-face interview. Shortened Interviews: South Carolina reported that they do not have a shortened interview process. Online Expedited Applications: South Carolina does not have online expedited applications. Application Tracking: South Carolina can track the source of applications, but does not run reports or otherwise use the data. The state tracks the quality control measures that are required by FNS. Only the quality control sample is used to look at accuracy measures, not all applications. Changes Over Time: The eligibility system was designed many years ago, but many tracking codes have been added within the last year. Desired or Planned Future Measures: South Carolina would like to institute a process to track the rate at which partners sign up more applicants. South Carolina eventually would like to have all their applications come in online, but staff said the immediate goal is to have as many as possible done this way. A.196 Profile: South Dakota State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 0 Number of Partners Interviewed: 0 Initiatives Active in State: Document Imaging, Waiver of Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives DOCUMENT IMAGING (Statewide) None WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) None APPLICATION TRACKING (Statewide) None Notes: South Dakota did not complete the survey. We were able to interview them to obtain the information described below. Description: General: South Dakota has a document imaging system and a waiver of the face-to-face interview for recertifications. Measurement Goals: South Dakota has very few measures beyond those required by FNS. Call Center: South Dakota does not have a call center. Online System: South Dakota has not implemented an online system. Document Imaging: South Dakota manages the document imaging system in-house. Workers can access the scanned documents from their desktops if they have permission to view that case number. South Dakota does not formally track any performance measures, but some aspects of the document imaging system may be reviewed at times. Kiosks: South Dakota does not have kiosks. Waiver of Face-to-Face Interview: South Dakota conducts all application interviews in person and recertification interviews via phone. The type and number of interviews are captured in the narrative of the case record but are not entered into an automated system for tracking. Shortened Interviews: South Dakota does not use a shortened interview process. Online Expedited Applications: South Dakota does not have online expedited applications. Partnering: South Dakota did not indicate that they partnered with any organizations. Measures Collected by Partners: Not applicable. South Dakota(continued) A.197 Application Tracking: South Dakota tracks the number of pending and approved applications and approval and denial rates for applicants. They also track the number of applications that include elderly or disabled individuals or children. Local offices review cases with errors to identify trends. Changes Over Time: They have not changed state performance measures in at least 20 years. Desired or Planned Future Measures: None reported. A.198 Profile: Texas State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 2 Number of Partners Interviewed: 2 Initiatives Active in State: Call Center, Online System, Document Imaging, Partnering, Waiver of Face\u2010to\u2010Face Interview, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Statewide) Percent of Calls Completed in the Computer Phone System Statewide Efficiency, Customer Service Yes, not specified Not reported Average Answer Speed Statewide Program Access, Customer Service 180 seconds. Not reported Average Hold Time Statewide Program Access, Customer Service Yes, not specified Not reported Percent Calls Abandoned Statewide Program Access, Customer Service No more than 10 percent abandoned in a month (there is also a weekly measure). For every percentage point below the standard, there is a $500 to $1,000 penalty to the contractor. Average Call Duration Statewide Efficiency, Customer Service Yes, not specified Not reported Average Calls Handled Statewide Program Access, Efficiency, Customer Service Yes, not specified Not reported Average Staff on Phones Statewide Program Access, Efficiency, Customer Service Yes, not specified Not reported Number of Hang-ups Statewide Program Access, Efficiency, Customer Service Not reported Not reported Number of Changes Received Statewide Program Access, Efficiency, Customer Service Not reported Not reported Number of Calls Not Completed in Computer Phone System Statewide Program Access, Efficiency, Customer Service Not reported Not reported Number of Calls Abandoned Statewide Program Access, Efficiency, Customer Service Not reported Not reported Number of Calls Queued Statewide Program Access, Efficiency, Customer Service Not reported Not reported Texas (continued) A.199 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Calls Answered\/Handled Statewide Program Access, Efficiency, Customer Service Not reported Not reported Number of Calls Receiving Busy Signal\/Unable to Connect with Phone System Statewide Program Access, Efficiency, Customer Service Not reported Not reported Number of Clients Accessing Computer Phone System Data Statewide Program Access, Efficiency, Customer Service Not reported Not reported Number of Calls Received Statewide Program Access, Efficiency, Customer Service Not reported Not reported ONLINE SYSTEM (Statewide) Percent of Applications Received Statewide Program Access, Efficiency, Customer Service No No Average Number Submitted per Month Statewide Program Access, Efficiency, Customer Service No No Number of Applications Submitted for Each Type of Program Statewide Not reported Not reported Not reported DOCUMENT IMAGING (Statewide) Percent of Documents Imaged Same-Day Statewide Efficiency 98 percent of documents must be imaged same- day Not reported Percent of Documents Imaged by Next Business Day Statewide Efficiency 100 percent of documents must be imaged by the next business day. Not reported Number of Documents Received Statewide Efficiency Not reported Not reported Number of Documents Scanned Statewide Efficiency Not reported Not reported Number of Staff Assigned to Scanning Statewide Efficiency Not reported Not reported PARTNERING (Statewide) Number Clients Assisted per Partner Statewide Program Access, Efficiency, Customer Service No No Texas (continued) A.200 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Applications Received per Partner Statewide Program Access, Efficiency, Customer Service No No Number of Clients Educated Statewide Program Access No No Number of Clients Assisted Statewide Program Access No No WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Percent that are Missed and Have Notice Mailed Statewide Efficiency No No Error Rate for Cases with Telephone Interviews Statewide Accuracy & Integrity No No Error Rate for Cases with Face-to-Face Interviews Statewide Accuracy & Integrity No No ONLINE EXPEDITED APPLICATIONS (Statewide) None APPLICATION TRACKING (Statewide) Percent of Community Partner Applications Approved Statewide Program Access, Efficiency, Customer Service No No Percent of Community Partner Recertifications Approved Statewide Program Access, Efficiency, Customer Service No No Percent of Total Applications Approved Statewide Program Access, Efficiency, Customer Service No No Percent of Total Recertifications Approved Statewide Program Access, Efficiency, Customer Service No No Percent of Applications Not Completed Statewide Program Access, Efficiency, Customer Service No No Percent of Applications: Online Statewide Program Access, Efficiency, Customer Service Not reported No Percent of Recertifications: Online Statewide Program Access, Efficiency, Customer Service Not reported Not reported Texas (continued) A.201 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Online Recertifications Denied for Failure to Submit Documentation Statewide Not reported No No Percent of Initial Applications Denied Statewide Program Access, Customer Service No Percent of Recertifications Denied Statewide Program Access, Customer Service No No Percent of Applicants that Appear Eligible for Expedited Benefits Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Average Benefit Amount (for those eligible) Statewide Program Access, Accuracy & Integrity, Customer Service No No Percent of Applications Processed Timely Locally Efficiency, Customer Service No No Number of Applications Approved: Community Partner Statewide Program Access, Efficiency, Customer Service No No Number of Recertifications Approved: Community Partner Statewide Program Access, Efficiency, Customer Service No No Number of Applications: Online Statewide Program Access, Efficiency, Customer Service No No Number of Recertifications: Online Statewide Program Access, Efficiency, Customer Service No No Number of Applications: Community Partner Statewide Program Access, Efficiency, Customer Service Not reported Not reported Number of Recertifications: Community Partner Statewide Program Access, Efficiency, Customer Service Not reported Not reported Number of Cases with Complete Verification Submitted Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No, but in the process of establishing some No, but in the process of establishing some incentives Number of Applications that are Not Processed Timely Locally Efficiency, Customer Service No No Notes: State collects mandatory accuracy data through QC reviews. Texas (continued) A.202 Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Description: General: Texas has the Texas Integrated Eligibility Redesign System (TIERS), a new eligibility determination system that is being rolled out across the state as an upgrade from the state’s legacy system. The state call center and document imaging center are run by contractor staff. Texas also has a new online application system, built by a contractor, and a variety of other partnering efforts. Measurement Goals: Texas closely monitors a variety of measures from the partners with whom they have formal arrangements. They also review cases beyond those required for QC reviews to identify training opportunities. Call Center: Texas has two call centers, but they do not track data by location\u2014the call centers share the pool of calls. Tied to the call centers is a work management system that operates real-time. If the system fails, it can be difficult for staff to manage the call volume; staff reported that this happens only intermittently, and has not been a problem recently. The call center can receive applications, recertifications, and changes and can be used to schedule interviews and return client phone calls. Clients can sign applications telephonically as well. Call center staff have access to imaged documents to answer client questions about what documents the state has received. Information coming into the call center does not automatically transfer data into TIERS, but creates a digital image from which the eligibility worker can enter the data into TIERS. At the time of our interview, Texas was planning to launch a system to automatically move the data into TIERS. Some information, such as expenses and income, will continue to be entered manually to ensure that the workers review the information carefully. A contractor runs the call center and is required by Texas to report the measures listed in the table. They also conduct monthly reviews of the calls to ensure that the information provided to the callers is accurate. Partnering: Texas has formal agreements with the contractors for the call center, online system, and document imaging. Both formal and informal arrangements govern relationships with community- based organizations that provide application assistance and outreach. One group of partners is focused on the Children’s Health Insurance Program, but they help with SNAP as needed. Another partner assists clients through their own call center, filling out applications for clients, then sending to the client for their signature. The state reimburses partners for the labor, materials, and travel for educating and assisting clients. Texas funds two outreach coordinator positions for one of the partners that is working with other community-based organizations in many counties\u2014this organization is required to serve all of these counties throughout the year. The partners providing application assistance do not have special access to the online system. Texas also has a waiver that allows non-merit employees to conduct the SNAP interview, although state employees continue to determine the benefit. Texas is working with a designated group of community-based organizations to conduct these interviews. Measures Collected by Partners: Texas requires reports from the formal outreach partners about the number of clients assisted and educated, the number of applications distributed, and (for one partner) the counties served in the month. Texas is also working toward tracking which applications come from which partners to identify potential areas for outreach and training. For their own purposes, one of the partners intends to start collecting demographic information\u2014age, number of people in the household, and level of education. Texas (continued) A.203 Online System: Contractors also developed the online system for Texas. Clients can apply online, signing the application electronically if they choose. Reporting for the online system was new at the time of this study, so details about the calculations were not yet available. Texas is also exploring standards for the online system, and staff reported that they plan to attempt to maximize the percentage of online applications. Staff said they are interested in ensuring that the measures can be calculated real-time so supervisors can be responsive to current activity levels. Document Imaging: Everything imaged at the document processing center comes to that center by mail. Texas does not track whether the documents were originally provided by mail or fax or dropped off at the office. Once imaged, the documents are classified as applications or supporting documents. Contractor staff register applications, schedule appointments, and route the documents for state staff to take action. Documents are stored in a repository for the state, allowing workers to search for documents related to a client. The Texas staff meet with the contractor every other week to review the reports and discuss imaging issues and trends. Contractor staff also sample and review the imaged documents for quality control purposes. Kiosks: Texas does not have kiosks. Waiver of Face-to-Face Interview: Telephone interviews as part of the face-to-face waiver are conducted by merit staff, outside the call center. They track whether interviews were conducted in person, through a home visit, or by phone. Shortened Interviews: Texas does not have a shortened interview initiative Online Expedited Applications: Texas accepts expedited applications online, but the state does not collect any data or measures about this initiative. Application Tracking: Texas can track whether applications are received online, by paper, or from a partner, but they do not do so regularly. However, they are not able to track other aspects of processing, such as timeliness, by the source of the application. Local offices in Texas pull cases, in addition to those needed for QC review, for internal quality assurance. They use these for training, accountability, and performance evaluation purposes. State reports track the number of cases handled by each worker Changes Over Time: Texas rolled out a new eligibility determination system, Texas Integrated Eligibility Redesign System (TIERS), beginning in 2009. Texas also reported a new document imaging vendor as of January 2010 and a new online application system built by a contractor. At the time of the interview, state staff described plans to transition partners from hand-delivering or faxing applications to assisting clients with the online application. This would enable the state to automatically track the number of online applications coming in from each partner. Additionally, the layout and production of application tracking reports changed in December 2010, although the calculation of the measures did not change. In the last quarter of 2010, the reports became available on a daily rather than monthly basis. Desired or Planned Future Measures: Texas staff reported that they are interested in tracking why clients come into the office (for example, for an EBT card, to submit an application, and so on). Currently, they only manually track some of the information. Having a report could help with customer flow and improve their services. Staff would like to be able to compare the approval rates of their partners with those of other states to see if they need to improve the services partners provide, and to seek advice from states that have high partner approval rates. A.204 Profile: Utah State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 5 Number of Partners Interviewed: 4 Initiatives Active in State: Call Center, Online System, Document Imaging, Kiosks, Partnering, Waiver of Face\u2010to\u2010 Face Interview, Shortened Interviews, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Statewide) Average Answer Speed Statewide Program Access, Efficiency, Customer Service State: No One local office: Less than 30 minutes. No Percent Calls Abandoned Statewide Program Access, Customer Service No No Average Call Duration Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Average Waiting Time (to Speak to Agent) Statewide Program Access, Efficiency, Customer Service State: No One local office: Reported that the standard is 30 minutes, or 8 calls waiting, at which time more help should be requested. No Average Calls Handled Statewide State: None Local: Program Access, Efficiency, Customer Service State: Within 5 percent of the team average for number of calls No Average Queue Time Statewide Program Access, Efficiency, Customer Service No No Average Time Prior to Abandonment Statewide Program Access, Efficiency, Customer Service No No Average After-Call Work Time by Staff Statewide Accuracy & Integrity, Efficiency, Customer Service State: No One local office: 20 minutes No Number of Calls (in flow) Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Calls Abandoned Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Utah (continued) A.205 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Calls Queued Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Calls Answered\/Handled Statewide Program Access, Efficiency, Customer Service No No Number of Agents Assigned to Phones Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service State: No One local office: each team should staff the queue with at least 5 members. No Number of Calls Received Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Calls Transferred to Agent Statewide None No No Average Hold Time Statewide Program Access, Efficiency, Customer Service One local office: less than 20 minutes No Average Staff on Phones Statewide Local: Program Access, Accuracy & Integrity, Efficiency, Customer Service One local office: phones are staffed with at least 5 people, but expect half of the team which is generally 7 No Number of Calls Receiving Busy Signal\/Unable to Connect with Phone System Statewide Program Access, Customer Service Not sure Not sure Number of Clients Accessing Computer Phone System Data Statewide Program Access, Accuracy & Integrity Not sure Not sure Percent of Calls Completed in the Computer Phone System Statewide None Not reported Not reported Percent of Queued Calls Handled Statewide None Not reported No Percent of Total Calls Handled Statewide None Not reported Not reported ONLINE SYSTEM (Statewide) Percent of Applications Received Statewide Program Access, Customer Service No No Average Number Submitted per Month Statewide Program Access, Efficiency, Customer Service No No Utah (continued) A.206 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Average Number of Changes Submitted per Month Statewide No No Average Level of Benefits Statewide None No No Average Number of Accounts Created per Month Statewide Program Access, Efficiency, Customer Service No No Number of Online Accounts Created Statewide Program Access, Efficiency, Customer Service No No Number of Online Instant Messaging\/Chats Statewide Program Access, Efficiency, Customer Service No No Average Number of Log- ins per Month Statewide Program Access, Efficiency, Customer Service Not sure Not sure DOCUMENT IMAGING (Statewide) Number of Documents Received Statewide None Individual worker performance expectation is based upon a fluctuating workload average. No Number of Documents Received: Fax Statewide None Not reported No Number of Documents Received: Other Electronic Statewide None Individual performance workload measurement. No Number of Documents Scanned Statewide None Individual performance workload measurement. No Percent of Documents Received by Fax Statewide Program Access, Customer Service Workload measurement expectations are found in individual performance plans for those who primarily image documents. No KIOSKS (Statewide) None PARTNERING (Statewide) Number of Partners Statewide Program Access, Efficiency, Customer Service No No Utah (continued) A.207 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Error Rate for Cases with Telephone Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Meet or exceed the federal QC accuracy rates set for all cases No Error Rate for Cases with Face-to-Face Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Meet or exceed federal QC accuracy rates set for all cases No SHORTENED INTERVIEWS (Statewide) Percent of Total Interviews that Are Shortened Statewide Not reported Not reported Not reported Percent of Clients Electing Shortened Interview Statewide Not reported Not reported Not reported ONLINE EXPEDITED APPLICATIONS (Statewide) Percent of Applications Approved: Online Expedited Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Expedited timeframe of 7 days Yes, not specified Percent of Applications Approved: Online All Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure APPLICATION TRACKING (Statewide) Percent of Total Applications Approved Statewide None No No Percent of Online Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Average Benefit Amount (for those eligible) Statewide None No No Percent of Applications: Online Statewide Program Access, Efficiency, Customer Service No No Percent of Recertifications: Mailed Statewide None No No Total Number of Applications Completed Statewide Program Access No No Number of Applications: Online Statewide Program Access, Efficiency, Customer Service No No Number of Recertifications: Mailed Statewide None No No Utah (continued) A.208 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Average Processing Time of Application Decision Statewide Program Access, Efficiency, Customer Service 30 days, but prefer to process applications more quickly. No Percent of Complete Applications Denied Statewide Program Access, Customer Service No No Percent of Applications Approved After Paper Submission to Local Office Statewide Not reported Not sure Not sure Percent of Call Center Applications Approved Statewide Not reported Not sure Not sure Percent of Call Center Recertifications Approved Statewide Not reported Not sure Not sure Percent of Mailed Recertifications Approved Statewide Not reported Not sure Not sure Percent of Total Recertifications Approved Statewide Not reported Not sure Not sure Percent of Applications: Call Center Statewide Not reported Not sure Not sure Percent of Online Applications Denied for Failure to Complete Interview Statewide Not reported Not sure Not sure Percent of Recertifications Denied Statewide Not reported Not sure Not sure Average Processing Time of Recertification Decision Statewide Not reported Not sure Not sure Percent of Applications that Appear Eligible for Expedited Benefits Locally Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Utah (continued) A.209 Description: General: Utah has been modernizing SNAP and other programs for over a decade. Until June, 2009, the offices around the state had different policies and different ways for determining eligibility. In 2009 they consolidated all eligibility services (including SNAP, child care, and financial assistance) under one division. They now have a standardized document imaging center and one phone number for a virtual call center environment. All of Utah’s modernization efforts are initiated and standardized by the state. State staff reported that they are working to improve upon the many existing initiatives, including the online application and their document imaging system. Utah has faced few challenges with their data collection and reporting since they standardized and streamlined their procedures within the last year. While the state noted that they would like reporting to be more automated for certain initiatives, they are fairly satisfied with their current capabilities. Utah is pursuing expanded performance measurement with some of their new initiatives, such as e- notifications and the live-chat feature. Any new measures will be collected in addition to the large number of measures they already track. The state pays to be part of a Call Center Network Group (CCNG), so that they can learn about best practices and recommendations for standards. Measurement Goals: Utah’s performance measures touch on all FNS goals. They also use their performance data for staff development, workload management, responses to requests (from community advocates, legislative workers, or in-house staff), and for monitoring the frequency of use for the online application. Call Center: The call center has one toll-free number, which is the main method of contact for customers to access the program. This includes applications, inquiries, change reporting, and recertification. When a customer calls in, they reach an interactive response system that pulls up their case based on information they enter and routes them to the correct team of agents (the system does not have the functionality to receive information without agent intervention). While all measures are standardized across the state, some local offices interviewed reported additional modernization goals, reporting period, or management targets than the state. Utah sets some of their standards or benchmarks using the team concept, which means setting standards for each worker within a certain range (for example, 5 percent) of the team average. They track wait-time in real time to identify when more staff needed to answer the phones. Online System: Utah has two online systems: one for eligibility workers to interface with cases and to control workflow (E-REP\u2014released in mid-2011) and one for customers to access their account (MyCase\u2014released November 2011). MyCase features include allowing customers to see when tasks will be complete, if documents have been received, and their benefit amount. More features are under development. Utah also offers customers an online chat feature: through the website, they can interact with staff on demand. Utah reports this is more efficient than phone calls because one eligibility worker can handle two to four chats simultaneously. Within the next two years, Utah plans to redesign their online application. They reported plans to expand it to include change reporting, recertifications, instant messaging interviews, and an improved pre-screening tool. Document Imaging: Utah’s document imaging center handles 60 to 75 percent of the statewide documents, and images are stored on a statewide file server. A few locations have the ability to scan Partnering: Utah has a count of the number of partners with whom they have a relationship (usually, this is a written MOU). Partners assist clients and provide computer access, but do not make eligibility determinations. Partners do not provide Utah offices with performance data, though they may gather their own performance data. Measures Collected by Partners: Utah partners did not report any performance measurement. Utah (continued) A.210 and image their own documents if necessary, when mailing will take too long. All paper applications received (whether via fax, scan, email, or state mail) are sent to the centralized Image Operation Center. Utah uses barcoding to help attach correspondence originating with the state back to the case. Kiosks: Utah has more than thirty computer stations throughout the state, termed Job Connection Areas. Clients have the capability to use the kiosks to connect to the Internet for the online application or to download a paper application. Utah does not track any measures for this initiative. Waiver of Face-to-Face Interview: Utah collects the error rate for cases by the type of interview (telephone or face-to face), but these measures are not used unless requested by FNS. Although they do not currently collect it, they would be able to record the type and number of interviews conducted with SNAP applicants. Shortened Interviews: Utah reported that they have a shortened interview initiative, but did not provide us with additional details. Online Expedited Applications: Utah records whether an application is expedited or not, but they do not have any other measures for online expedited applications Application Tracking: Utah tracks a number of application tracking measures, including those required for FNS QC. Utah is aware of the origin of all paper applications while they are processed at the centralized imaging unit, but they do not have a system field to record and save this type of information. Changes Over Time: In the past year there have been no significant changes to the way Utah calculates measures or collects data. They have recently standardized old systems or switched to new systems, but the basic methodology has not changed. Due to one change, Utah does not have as much detail on denial reason, but they are adding back in some of that information. Desired or Planned Future Measures: At the time of the interview, Utah had created MyCase and was trying to encourage more customers to use it to access their information. They have begun to look at numbers based on the homepage but have not put standards in place yet. Additionally, they would like to establish an email notification system for any correspondence sent through the customer homepage (the customer would receive an email when they have a message waiting) and would track data related to this system. Utah would also like to track more information on the live-chat feature. Utah state staff would like to have information on performance measures used by other states, particularly those related to call centers and costs. Due to a lack of standardization, they noted difficulty comparing measures among call centers in other states. Staff in all local offices interviewed were satisfied with the data and reports they received from the state office; they did not name any performance measures that they would like to calculate outside of those they can already access. A.211 Profile: Virginia State Administered or County Administered: County Number of Local\/County Agencies Interviewed: 3 Number of Partners Interviewed: 0 Initiatives Active in State: Call Center, Online System, Document Imaging, Kiosks, Waiver of Face\u2010to\u2010Face Interview, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Regionally) Average Queue Time Regional Efficiency Not reported Not reported Average Time Prior to Abandonment Regional Efficiency Not reported Not reported Number of Calls Answered\/Handled Regional Efficiency Not reported Not reported Number of Agents Assigned to Phones Regional Efficiency Not reported Not reported Average Hold Time Regional Efficiency Not reported Not reported Percent Calls Abandoned Regional Efficiency Not reported Not reported Average Calls Handled Regional Efficiency Not reported Not reported Average Staff on Phones Regional Efficiency Not reported Not reported Number of Calls Abandoned Regional Efficiency Not reported Not reported Number of Calls Queued Regional Efficiency Not reported Not reported ONLINE SYSTEM (Statewide) Percent of Applicants Who Received Help to Apply Online Statewide Program Access, Customer Service No No Number of Applications Started Statewide Not reported Not reported Not reported Total Number of Applications Submitted per Month Statewide Not reported No No Number of Abandoned Applications Statewide Not reported No No Number of Applicants who Received Help to Apply Online Statewide Not reported Not reported Not reported DOCUMENT IMAGING (Regionally) None KIOSKS (Regionally) None Virginia (continued) A.212 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives PARTNERING (Statewide) Number of Partners Statewide Not reported Not reported Not reported WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Percent of Interviews Conducted Face-to-Face Statewide Program Access, Customer Service No No Number of Denials Due to Missed Interview Statewide Program Access, Accuracy & Integrity, Customer Service No No Number of Redetermination Interviews Statewide Efficiency Not reported Not reported Number of Missed Interviews Statewide Efficiency Not reported Not reported Notice of Missed Interview Mailed Statewide Efficiency Not reported Not reported Total Number of Interviews Statewide Efficiency Not reported Not reported ONLINE EXPEDITED APPLICATIONS (Statewide) None APPLICATION TRACKING (Statewide) Percent of Total Applications Approved Statewide Not reported No No Percent of Total Recertifications Completed Statewide Not reported No No Number of Applications Online Locally None Not reported Not reported Average Benefit Amount (for those eligible) Statewide Not reported No No Number of Applications Approved: Online Statewide Not reported Not reported Not reported Number of Applications Approved: Face-to-Face Interviewed Statewide Not reported Not reported Not reported Number of Applications Approved: Phone Interviewed Statewide Not reported Not reported Not reported Percent of Applications Not Completed Statewide Not reported Not reported Not reported Virginia (continued) A.213 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Total Number of Applications Started Statewide Not reported Not reported Not reported Total Number of Recertifications Started Statewide Not reported Not reported Not reported Total Number of Applications Completed Statewide Not reported Not reported Not reported Total Number of Recertifications Completed Statewide Not reported Not reported Not reported Percent of Complete Applications Denied Statewide Program Access No No Timeliness Rate Statewide Customer Service No Because of a court order, if the agency is below 97percent for regular processing time of applications, the local agency is required to submit a corrective action plan for each case that was out of compliance. Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Description: General: Virginia’s SNAP is state supervised and locally administered. They have a 60-day application process rather than a 30-day process. The online system is available statewide, and the call center is available in some counties. Some counties are also beginning to incorporate document imaging and kiosks. Measurement Goals: Virginia’s primary goal for the measures related to efficiency. Call Center: The call center operates in one county, and they consider it to be a pilot. The call center handles change reporting, transfers callers to agent, and makes outgoing calls. The county’s primary measures for tracking are the total number of calls, number of abandoned calls, and the average queue time. They use the measures primarily to distribute work and modify office procedures for handling calls. Virginia (continued) A.214 Online System: The online system provides a screening tool and allows applicants to submit signed application. At the time of this data collection, the state had a manual process for entering the online information into the eligibility system. The state sends the electronic version of the application to the appropriate local agency; the local agency has to retrieve and download the application and upload it into the eligibility system. In November 2010, the system will become automated and information will go directly into the eligibility system. Virginia developed a variety of reports to measure the system’s performance, but they are not used due to questions about the accuracy of the data. The reports primarily measured program access and efficiency. Document Imaging: At the time of our data collection, one county was imaging documents and another county was preparing to image documents. The county did not report any measures related to document imaging. Kiosks: At the time of the interview, one county had just recently put a few computers in their lobby for applicants. Waiver of Face-to-Face Interview: Virginia has a waiver of the face-to-face interview at both application and recertification. Their system records whether the interview is conducted by phone or face- to-face. Shortened Interviews: Virginia does not have a shortened interview process. Online Expedited Applications: The eligibility system automatically determines if an application should be expedited. The state believes they could provide measures on approval status, possibly by some demographic characteristics, but it’s not regularly done now. Application Tracking: Virginia believes they could be doing more measuring, such as differentiating between applications and recertifications, but it would require a shift in priorities. They also have an internal database for tracking applications that works in tandem with an appointment scheduler. This information is currently used to make staffing decisions across offices. Changes Over Time: None reported. Desired or Planned Future Measures: Virginia would like to see more readily available data on accuracy rates for states, particularly high performing ones. A local agency is interested in tracking what percentage of applicants are repeat applicants versus first-time applicants. Partnering: Virginia is in the process of developing outreach plans with several community-based organizations. No partnerships completed for SNAP outreach but the state reported performance measures will likely include expenditures, number of screenings completed, number of applications, and final disposition of applications Measures Collected by Partners: Not applicable. A.215 Profile: Washington State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 7 Number of Partners Interviewed: 4 Initiatives Active in State: Call Center, Online System, Document Imaging, Kiosks, Partnering Arrangements, Waiver of Face\u2010to\u2010Face Interview, Shortened Interviews, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Statewide) Number of Calls (in flow) Statewide Program Access, Efficiency, Customer Service No No Number of Calls Abandoned Statewide Program Access, Efficiency, Customer Service No No Number of Calls Answered\/Handled Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Number of Agents Assigned to Phones Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Number of Calls Receiving Busy Signal\/Unable to Connect with Phone System Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Average Answer Speed Statewide Program Access, Efficiency, Customer Service 5 minutes No Average Hold Time Statewide Program Access, Efficiency, Customer Service 8 minutes No Percent Calls Abandoned Statewide Program Access, Efficiency, Customer Service No No Average Call Duration Statewide Program Access, Efficiency, Customer Service No No Average After-Call Work Time by Staff Statewide Program Access, Efficiency, Customer Service No No Number of Changes Received Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Changes Processed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Calls Received Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Cases Pending Benefit Decision: Call Center Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Applications Processed: Call Center Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Washington (continued) A.216 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Client Calls Returned Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of documents processed within 30 days Statewide Program Access, Accuracy & Integrity, Customer Service 90% of documents processed within 30 days No Standard of Promptness for Applications and Recertifications Statewide Program Access, Accuracy & Integrity, Customer Service 95% of applications processed within standard of promptness No Percent of Total Calls Handled Locally Program Access, Customer Service Not reported Not reported Average Staff on Phones Locally Program Access, Customer Service Not reported Not reported Average Time Prior to Abandonment Locally Program Access, Customer Service Not reported Not reported Percent of Queued Calls Handled Locally Program Access, Customer Service Not reported Not reported Number of Calls Queued Locally None Not reported Not reported ONLINE SYSTEM (Statewide) Percent of Applications Received Statewide Program Access, Efficiency, Customer Service No No Number of Changes Submitted Statewide Program Access, Accuracy & Integrity, Customer Service No No Number of Online Applications Submitted Locally Program Access, Customer Service No No DOCUMENT IMAGING (Statewide) Percent of Documents Scanned Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service 100% of customer documents scanned. Mandated by Governor due to lawsuit. No Washington (continued) A.217 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Documents Scanned within Standard of Promptness Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service 100% of non- completed documents scanned within 1 business day. 100% of completed documents scanned within 3 business days, mandated by governor. No Number of Documents Received Statewide Efficiency No No Number of Documents Processed Statewide Efficiency No No Number of Documents Scanned Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Tasks Completed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No KIOSKS (Statewide) Number of Uses to Submit Changes Statewide Program Access, Customer Service No No PARTNERING (Statewide) Number of Partners Statewide Program Access, Customer Service No No Number Accepting Applications Statewide Program Access, Customer Service No No Number Providing Application Assistance Statewide Program Access, Customer Service No No Number with Terminals\/Kiosks to Submit Application Statewide Not reported Not reported Not reported WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Number of Face-to-Face Interviews Locally Customer Service Not reported Not reported SHORTENED INTERVIEWS (Statewide) None ONLINE EXPEDITED APPLICATIONS (Statewide) None Washington (continued) A.218 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives APPLICATION TRACKING (Statewide) Percent of Total Applications Approved Statewide Not reported Not reported Not reported Percent of Total Recertifications Approved Statewide Not reported Not reported Not reported Percent of Applications: Online Statewide Program Access, Customer Service No No Percent of Recertifications: Online Statewide Program Access, Customer Service No No Number of Applications: Online Statewide Program Access, Customer Service No No Number of Recertifications: Online Statewide Program Access, Customer Service No No Average Processing Time of Application Decision Statewide Efficiency, Customer Service No No Average Processing Time of Recertification Decision Statewide Efficiency, Customer Service Yes, not specified. Yes, not specified. Average Benefit Amount (for those eligible) Locally Not reported No No Percent Errors where Information was Incorrect Locally None Not reported Not reported Total Number of Applications Completed Locally None Not reported Not reported Total Number of Recertifications Completed Locally None Not reported Not reported Percent of Complete Applications Denied Locally Program Access, Customer Service No No Percent of Initial Applications Denied Locally Program Access, Customer Service No No Percent of Applicants that Appear Eligible for Expedited Benefits Locally None Not reported Not reported Number of Cases with Complete Verification Submitted Locally None Not reported Not reported Percent of Applications Pending Locally Program Access No No Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. Washington (continued) A.219 If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. ACD = Automatic Call Distribution: also known as call center software, which sorts callers into queues by the reason for their call. VDN = Vector Directory Number: phone numbers of agents who are connected to different parts (vectors) of the call center queue. CSO = Community Service Office: Washington’s name for local SNAP offices. Description: General: Washington has each modernization initiative that we have reviewed for the study. They launched their online system in the early 2000s. They recently redesigned their service delivery, and one outcome was an alignment of the use of technology across the state. The call centers now use the Avaya system to track the same data across states. Documents are scanned at four indexing sites in the state. Washington partners with community organizations and has kiosks available for client use at both local offices and community partners. The state has both a waiver of the face- to-face interview and a shortened interview process. Measurement Goals: Washington measures performance with the goal of improving program access, efficiency, customer service, and accuracy and integrity. Call Center: The state has a series of regional call centers. The state uses the Avaya Call Management System. The system automatically tracks a wide variety of data for calls received, handled, and abandoned, as well as performance data to monitor staff, such as number of agents assigned to phones and after call work time by staff. Hold time and answer speed are also tracked and have associated performance standards. Washington developed performance data by visiting existing call centers in both private industry and other government agencies. Call center supervisors also requested performance measures they thought would be most useful for management purposes. Online System: Washington operates the online system and tracks the percent of applications received, number submitted, and number of changes submitted. Local offices can query reports as needed. Staff manually enter information obtained through the online application system into the eligibility system. At the time of data collection, Washington was in the process of upgrading their online system. Also, at the time of interview, it was not possible to use the screening tool to pre-populate the application; however, two local offices are testing this functionality as part of a pilot program. Document Imaging: Washington tracks the number of documents received, processed, and scanned; the state also tracks the percentage of documents scanned overall (for which they set a standard of 100 percent), and within standard of promptness. Non-completed documents must be scanned within one business day, while completed documents must be scanned within three business days. The state maintains these benchmarks to comply with standards mandated by the governor. When offices receive a document, it is sent via US mail to one of four indexing sites in the state to be sorted, Partnering: Washington tracks the total number of partners, as well as the number accepting applications, providing application assistance, and hosting kiosks to submit applications. Washington partners with approximately 500 community- based organizations (as direct contractors or as subcontractors). Partnerships at the state level are governed by memoranda of understanding. Local offices also partner, and this contact is monitored and reported to the state. There are four levels of partnership, depending on the equipment and services, ranging from a computer with self-service access to online applications and providing equipment and staff resources for a variety of assistance and advice. Measures Collected by Partners: Partners in Washington collect performance data, including the number of applications and application approval rate. Washington (continued) A.220 scanned, and indexed. The indexing process both identifies the origin of the document and the type of document. Once indexed, the document is automatically loaded into the barcode document imaging system, where it is linked to the case of the client who submitted it. This process takes a maximum of 2.5 days. Scanned documents are stored on mega-servers with multiple backups. Case managers and workers assigned to the case have access to the documents. Kiosks: Washington has kiosks available at the local Community Service Offices (CSOs) for all clients. Clients can use the kiosks to complete a screening for benefit eligibility, submit a change of circumstance, or complete and submit an application or recertification. Staff are on hand at the CSO as a resource to answer questions and walk clients through the application process, resulting in a higher level of customer service. Washington is currently unable to track if an application originated at a kiosk, but the functionality has been requested by at least one local office. The online application contains an optional question allowing clients to enter who helped them complete the application; however, Washington does not use this information for any purpose. Waiver of Face-to-Face Interview: Washington has a waiver to conduct interviews via telephone. Locally, Washington tracks the number of face-to-face interviews, but at the state level the type of interview is not tracked. Missed interviews or other problems completing the interview are recorded in the case narrative, but are not tracked. Shortened Interviews: Washington assigns cases to an interview type based on how error prone the household is expected to be. Simple cases that get the shortened interview are often single-person or small households or Medicaid applicants. More difficult cases, such as those with more people, self- employment income, or an accompanying TANF application are sent through the longer interview process. Washington did not report collecting any measures related to the shortened interview. Online Expedited Applications: Expedited applications can be submitted through the online system. At the state level, demographic information or approval status for expedited cases is available on an ad-hoc basis. Application Tracking: Washington tracks the percentage of applications approved, percentage submitted online, and the average processing time. They also track accuracy measures required by FNS. Washington developed application tracking performance data many years ago. Some measures are based on federal requirements, while others were developed by quality assurance or policy and operations staff in the state office. Data are collected at the state level, and the reports are then shared with local offices. Changes Over Time: Washington delivered approximately 17 kiosks to outreach partners with funding from an FNS grant to improve SNAP access among the working poor population. Washington installed the online application system in 2000. Since then, they have tracked the percentage of application received online to monitor program access. At the time of data collection, they were working with private industry and customer focus groups to develop a new online application. Desired or Planned Future Measures: Washington is unable to track the type of interview, but staff reported that they would like to have that capability. For online application, they are interested in tracking the number of applications by source, including from community partners, home computers, and from the office lobby. This capability would assist outreach efforts and help to identify the most active partners. To increase customer service, Washington is interested in monitoring client outcomes. This capability should be available next year using quality monitoring software. Washington also plans to upgrade the call center with virtual hold software, allowing customers can hang up if there is a wait to speak with an agent. The system will call them back when the next agent is available. In addition, respondents in Washington stated a desire for additional performance data to monitor staff. Additional performance data would allow supervisors to identify training needs and oversee staff work. To capture this type of information, Washington plans on integrating the current call management system with Workforce Management Software to develop staffing models. A.221 Profile: West Virginia State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 0 Number of Partners Interviewed: 0 Initiatives Active in State: Call Center, Online System, Wavier of the Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Statewide) Number of Calls Answered\/Handled Statewide Program Access, Efficiency, Customer Service At least 40 calls per day, per worker. No Number of Agents Assigned to Phones Statewide Efficiency, Customer Service 40 contacts per day, per agent. All agents are automatically on the phones during high volumes. No Percent of Changes Processed Statewide Program Access, Efficiency, Customer Service Goal is 100 percent. No Average Hold Time Statewide Program Access, Efficiency, Customer Service Not sure No Percent Calls Abandoned Statewide Program Access, Efficiency, Customer Service Less than 20 percent dropped calls No Average Call Duration Statewide Accuracy & Integrity, Efficiency, Customer Service No No Average Waiting Time (to Speak to Agent) Statewide Program Access, Efficiency, Customer Service Less than 5 to 7 minutes. Yes, not specified Average Calls Handled Statewide Accuracy & Integrity, Efficiency, Customer Service Minimum of 40 calls per day, per worker. No Percent of Total Calls Handled Statewide Accuracy & Integrity, Efficiency, Customer Service 80 percent of clients. Goal to drop no more than 30 percent. They plan to reduce this number to 20 percent. No Average Queue Time Statewide Program Access, Efficiency, Customer Service No No Number of Calls (in flow) Statewide Program Access, Customer Service No No Number of Calls Abandoned Statewide Program Access, Customer Service Not reported No West Virginia (continued) A.222 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Calls Queued Statewide Program Access, Customer Service If more than eight calls in queue or if wait time is greater than eight minutes, all workers are instructed to log onto the phones. No Number of Calls Received Statewide Program Access, Customer Service Not reported No Number of Hang-ups Statewide Program Access, Efficiency, Customer Service Not sure No Number of Changes Received Statewide Accuracy & Integrity, Customer Service 40 contacts per day, per agent. No Number of Changes Processed Statewide Accuracy & Integrity, Customer Service 40 contacts per day, per agent. No Number of Calls Transferred to Agent Statewide Program Access, Efficiency, Customer Service No No ONLINE SYSTEM (Statewide) Number of Online Accounts Created Statewide Program Access, Customer Service No No Number of Log-ins Statewide Program Access, Customer Service No No WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Percent of Interviews Conducted by Telephone Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Percent of Interviews Conducted Face-to-Face Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Total Number of Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Redetermination Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Face-to-Face Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Telephone Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No West Virginia (continued) A.223 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Interviews Conducted by Telephone for Redeterminations Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percent of Closures Related to Telephone Interview Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported APPLICATION TRACKING (Statewide) Number of Applications Approved: Face-to-Face Interviewed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Recertifications Approved: Face-to-Face Interviewed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Total Number of Applications Started Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Total Number of Recertifications Started Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Total Number of Recertifications Completed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Notes: State collects mandatory accuracy data through QC reviews. West Virginia has two call centers, Northern and Southern. Description: General: West Virginia has an online system, call center, and waiver of the face-to-face interview. Enhanced call center functionality is planned. Measurement Goals: West Virginia primarily measures performance to ensure customer service, accuracy and integrity, efficiency, and program access. Call Center: West Virginia has two call centers: Northern and Southern. State staff reported that the Southern call center is advanced and has automated functionality. It became operational in May 2010. The Northern call center is in the process of converting to the same operations as the Southern call center; funds for this conversion have been allocated (but not awarded). Data for the measures is calculated by the systems and provided in reports. Thus, many details of the calculations were not available. Online System: West Virginia tracks the number of online applications and the number of log-ins. Some demographic information is transferred from the online system to the mainframe eligibility system. A contractor developed the system that the state now maintains. Some reports are available to system staff while others are available by caseload to supervisors. West Virginia (continued) A.224 Document Imaging: West Virginia does not have document imaging. Kiosks: West Virginia does not have kiosks. Waiver of Face-to-Face Interview: West Virginia received guidance from FNS and looked at other state operations for procedural advice for the waiver of the face-to- face interview, with the goal of maintaining a high level of customer service. West Virginia tracks the number of applications and recertifications that are completed by a phone interview or a face-to-face interview. They also track the percent of case closures related to telephone interviews, which was monitored regularly until it was determined that the error rate was not increasing. This information is now reported on demand. Data collection for this measure is automatic, but report production is not. The office aims for the failure rate for telephone interview to be less than that for face-to-face; however, there is no specific target. All initial applications are face-to-face interviews, except for hardship interviews. Redetermination application interviews that only apply to SNAP benefits are completed on the telephone. The percent of interviews conducted by telephone for redetermination was monitored regularly until they determined that the error rate was not increasing. This information is now reported on demand. Shortened Interviews: West Virginia does not have a shortened interview initiative. Online Expedited Applications: West Virginia did not report that they have developed the initiative for online expedited applications. Application Tracking: West Virginia tracks the number of approvals for face-to-face interviews, in comparison to telephone interviews. The number of applications, type of application, and number of applications by county are tracked on a worker manual contact sheet. These are transferred to a common report by region submitted to the state monthly. West Virginia also collects data for the FNS QC process. Changes Over Time: West Virginia implemented a call center in May 2010. Desired or Planned Future Measures: State staff reported that, by 2014, the eligibility system in West Virginia will be completely web-based, which they expect will increase efficiency. They are also creating an automated supervisory review process. This process will gather data and measure performance on a much larger sample than quality control, based on worker or supervisor review. Partnering: West Virginia does not have partnering arrangements. Measures Collected by Partners: Not applicable. A.225 Profile: Wisconsin State Administered or County Administered: County Number of Local\/County Agencies Interviewed: 9 Number of Partners Interviewed: 6 Initiatives Active in State: Call Center, Online System, Document Imaging, Kiosks, Partnering, Waiver of Face\u2010to\u2010 Face Interview, Shortened Interviews, Online Expedited Applications, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives CALL CENTER (Statewide) Average Hold Time Statewide Customer Service No No Average Waiting Time (to Speak to Agent) Statewide Not reported One local office: Under 10 minutes (and no more than 10 people in the queue) One local office: No official standard but would like to see this time to be under 1 minute (currently ranges from 30 seconds to 2 minutes). No Average Time Prior to Abandonment Statewide Not reported Not reported Not reported Average Calls Handled Statewide Not reported No No Percent of Total Calls Handled Statewide Not reported State: No One local office: 80 percent of calls answered. State: No One local office: Posted in the call center as a worker incentive Number of Calls (in flow) Statewide Not reported Not reported Not reported Number of Calls Abandoned Statewide Not reported Not reported Not reported Number of Calls Queued Statewide Not reported Not reported Not reported Number of Calls Answered\/Handled Statewide Not reported One local office: 45 calls answered a day per worker (during a full work day) One local office: Posted in the call center as a worker incentive Number of Agents Assigned to Phones Statewide Not reported Not reported Not reported Wisconsin (continued) A.226 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Changes Processed Statewide Not reported Not reported Not reported Average Time to Process Change Statewide Not reported One local office: Informal goal of average time between 5 and 7 minutes to process changes. Another local office: No No Number of Changes Received Statewide Not reported Not reported Not reported Number of Changes Processed Statewide Not reported Not reported Not reported Number of Cases Pending Benefit Decision: Call Center Statewide Not reported Not reported Not reported Number of Applications Processed: Call Center Statewide Not reported Not reported Not reported Average Answer Speed Statewide Customer Service No One local office: Goal is to answer 95 percent of calls within 30 seconds. No Percent Calls Abandoned Statewide Not reported Not reported Not reported Average Call Duration Statewide Not reported No No Average Time to Process Changes and Documents Locally Not reported One local office: Informal goal to stay within 10 days, ideally 2-3 days. No Percent of Calls Completed in the Computer Phone System Locally Not reported Not reported Not reported Percentage of Calls Within Their Categories of Time Intervals Locally Not reported One local office: 95 percent of calls within 30 seconds No Average After-Call Work Time by Staff Statewide Not reported No No Wisconsin (continued) A.227 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Calls Receiving Busy Signal\/Unable to Connect with Phone System Locally Not reported Not reported Not reported Average Time to De- Queue Locally Not reported Not reported Not reported Number of Calls Received Locally Not reported Not reported Not reported Number of Calls Completed in Computer Phone System Locally Not reported Not reported Not reported Average Completion Length Locally Not reported No No Percent of Screened Individuals Potentially Eligible: Call Center Locally Not reported Not reported Not reported Percent of Screenings Resulting in Applicant Receiving Benefits: Call Center Locally Not reported Not reported Not reported Percent of Screenings Resulting in Application Submission: Call Center Locally Not reported Not reported Not reported Average Staff on Phones Locally Not reported Not reported Not reported Average Queue Time Locally Not reported Not reported Not reported Percent of Queued Calls Handled Locally Not reported Not reported Not reported Number of Hang-ups Locally Not reported Not reported Not reported Percent of Calls Requesting Agent Locally Not reported Not reported Not reported Number of Calls Transferred to Agent Locally Not reported Not reported Not reported ONLINE SYSTEM (Statewide) Average Number of Screenings Started: Online Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Average Number of Screenings Completed: Online Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No Not sure Percent of Users Selecting Spanish Language Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No Not sure Percent of Screened Individuals Potentially Eligible: Online Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No Not sure Wisconsin (continued) A.228 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Online Screenings Started: Online Statewide Not reported Not reported Not reported Number of Online Screenings Completed: Online Statewide Not reported Not reported Not reported Percent of Applicants Who Received Help to Apply Online Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Yes, not specified Not sure Percent of Applications Received Statewide Program Access, Accuracy & Integrity, Efficiency No Not sure Average Number Started per Month Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No Not sure Average Number Submitted per Month Statewide Program Access, Accuracy & Integrity, Efficiency No Not sure Percent of Multiple Applications Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Percent of Applicants with Hardship Reason for Requesting Phone Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Average Level of Benefits Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of One- page\/Requests for Assistance Submitted Statewide Not reported Not reported Not reported Number of Applications Started Statewide Not reported Not reported Not reported Number of Abandoned Applications Statewide Not reported Not reported Not reported Number of Clients Requesting Help to Apply Online Statewide Not reported Not reported Not reported Number of Applicants who Received Help to Apply Online Statewide Not reported Not reported Not reported Number of Applicants who did not Receive Help to Apply Online Statewide Not reported Not reported Not reported Average Number of Log- ins per Month Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No Not sure Average Number of Accounts Created per Month Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No Not sure Wisconsin (continued) A.229 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Average Number of Changes Submitted per Month Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No Not sure Number of Online Accounts Created Statewide Not reported Not reported Not reported Number of Changes Started Statewide Not reported Not reported Not reported Number of Changes Submitted Statewide Not reported Not reported Not reported DOCUMENT IMAGING (Statewide) Number of Documents Scanned Statewide Accuracy & Integrity, Efficiency No No Number of Documents Received Statewide Not reported Not reported Not reported Number of Documents Processed Statewide Not reported Not reported Not reported Number of Tasks Completed Statewide Not reported Not reported Not reported Number of Alerts Processed Statewide Not reported Not reported Not reported Percent of Documents Scanned Locally Not reported One local office: 100 percent of documents are required to be scanned. No Average Processing Time Locally Not reported No No KIOSKS (Statewide) Number of Screenings Completed: Kiosk Statewide Not reported Not reported Not reported Number of Uses to Access Online Account Statewide Not reported Not reported Not reported Number of Uses to Submit Changes Statewide Not reported Not reported Not reported PARTNERING (Statewide) Percent of Partners Providing Application Assistance Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Average Cost of Partner Arrangement Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Wisconsin (continued) A.230 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Partners Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Number Accepting Applications Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Yes, not specified Yes, not specified Number Providing Application Assistance Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Yes, not specified Yes, not specified Number Clients Assisted per Partner Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Yes, not specified Yes, not specified Number of Applications Received per Partner Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Yes, not specified Yes, not specified Number with Terminals\/Kiosks to Submit Application Statewide Not reported Not reported Not reported WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) Percent of Telephone Interview Requests Honored Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service 50 percent No Error Rate for Cases with Telephone Interviews Statewide Accuracy & Integrity, Efficiency 50 percent Yes, not specified Error Rate for Cases with Face-to-Face Interviews Statewide Accuracy & Integrity, Efficiency 50 percent Yes, not specified Total Number of Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Number of Applicants Requesting a Phone Interview Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Number of Redetermination Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Percent of Interviews Conducted by Telephone Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service State: Not sure One local office: No more than 50 percent of the caseload One local office: No State: Yes, not specified Two local offices: No Wisconsin (continued) A.231 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Interviews Conducted Face-to-Face Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service State: At least 50 percent face-to-face One local office: No State: Not sure One local office: No Percent that are Missed and have Notice Mailed Statewide Program Access, Accuracy & Integrity, Customer Service No State: Yes, not specified One local office: No Number of Face-to-Face Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Number of Telephone Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Yes, 50 percent Yes, not specified Number of Home Visit Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not sure Not sure Number of Missed Interviews Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Yes, not specified Yes, not specified Notice of Missed Interview Mailed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Yes, not specified Yes, not specified Number of Applicants Requesting an In-Person Interview Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Number of Applicants who Did Not Answer to Type of Interview Requested Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Not reported Not reported Percentage of Interviews Conducted by Phone Without Documented Hardship Reason Statewide Not reported State; According to the conditions of the waiver, the waiver of the requirement to document hardship status was limited to 50 percent of the caseload. No Number of Times Unable to Schedule Interview with Applicant Locally Not reported Not reported Not reported Wisconsin (continued) A.232 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives SHORTENED INTERVIEWS (Regionally) Percent of Total Interviews that Are Shortened Locally Not reported Not reported Not reported Percent of Clients Electing Shortened Interview Locally Not reported Not reported Not reported Number of Shortened Interviews Locally Not reported Not reported Not reported Number of Clients with Shortened Interviews Locally Not reported Not reported Not reported ONLINE EXPEDITED APPLICATIONS (Statewide) Percent of Applications Approved: Online Expedited Statewide Not reported Not reported Not reported Percent of Applications Approved: Online All Statewide Not reported Not reported Not reported Number of Expedited Applications Received Online Statewide Not reported Not reported Not reported Number of Applications Approved: Online Expedited Statewide Not reported Not reported Not reported APPLICATION TRACKING (Statewide) Percent of Total Applications Approved Statewide Not reported Not reported Not reported Percent of Total Recertifications Approved Statewide Not reported Not reported Not reported Average Benefit Amount (for those eligible) Statewide Not reported Not reported Not reported Number of Applications Approved: Face-to-Face Interviewed Statewide Not reported Not reported Not reported Number of Recertifications Approved: Face-to-Face Interviewed Statewide Not reported Not reported Not reported Number of Applications Approved: Phone Interviewed Statewide Not reported Not reported Not reported Number of Recertifications Approved: Phone Interviewed Statewide Not reported Not reported Not reported Wisconsin (continued) A.233 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Number of Total Applications and Recertifications Approved Statewide Not reported Not reported Not reported Number of Second Party Reviews Locally Not reported One local office: A minimum of 34 per month Not reported Number of Applications Approved: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Recertifications Approved: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Applications Approved: Call Center Locally Not reported Not reported Not reported Number of Recertifications Approved: Call Center Locally Not reported Not reported Not reported Total Number of Applications Started Statewide Not reported Not reported Not reported Total Number of Recertifications Started Statewide Not reported Not reported Not reported Total Number of Applications Completed Statewide Not reported Not reported Not reported Total Number of Recertifications Completed Statewide Not reported Not reported Not reported Percent of Applications Not Completed Statewide Not reported Not reported Not reported Percent of Applications Received Online Statewide Not reported Not reported Not reported Number of Applications: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Recertifications: Paper Submission to Local Office Locally Not reported Not reported Not reported Number of Applications: Mailed Locally Not reported Not reported Not reported Number of Recertifications: Mailed Locally Not reported Not reported Not reported Number of Requests for Assistance\/One-page Applications Locally Not reported Not reported Not reported Wisconsin (continued) A.234 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Applications: Call Center Locally Not reported Not reported Not reported Percent of Recertifications: Call Center Locally Not reported Not reported Not reported Number of Applications: Call Center Locally Not reported Not reported Not reported Number of Recertifications: Call Center Locally Not reported Not reported Not reported Percent of Online Applications Denied for Failure to Complete Interview Statewide Not reported Not reported Not reported Percent of Online Recertifications Denied for Failure to Complete Interview Statewide Not reported Not reported Not reported Percent of Complete Applications Denied Statewide Not reported Not reported Not reported Percent of Complete Recertifications Denied Statewide Not reported Not reported Not reported Percent of One- page\/Requests for Assistance Denied Statewide Not reported Not reported Not reported Percent of Initial Applications Denied Statewide Not reported Not reported Not reported Percent of Recertifications Denied Statewide Not reported Not reported Not reported Number of Applications Denied Statewide Not reported Not reported Not reported Number of Recertifications Denied Statewide Not reported Not reported Not reported Number of Applications Denied: Paper Submissions to Local Office Locally Not reported Not reported Not reported Number of Recertifications Denied: Paper Submissions to Local Office Locally Not reported Not reported Not reported Number of Applications Denied: Call Center Locally Not reported Not reported Not reported Number of Recertifications Denied: Call Center Locally Not reported Not reported Not reported Wisconsin (continued) A.235 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Applications with Incomplete Information Statewide Not reported Not reported Not reported Percent of Recertifications with Incomplete Information Statewide Not reported Not reported Not reported Percent of Applicants that Appear Eligible for Expedited Benefits Statewide Not reported One local office: No One local office: No Number of Cases with Complete Verification Submitted Statewide Not reported Not reported Not reported Number of Applicants that Appear Eligible for Expedited Benefits Locally Not reported Not reported Not reported Number of Applications with Incomplete Information Locally Not reported No No Average Processing Time of Application Decision Statewide Not reported One local office: From January 1 Oct 30 2010, county was required to meet 70 percent timeliness. This benchmark increased to 90 percent for subsequent months. No Average Processing Time of Recertification Decision Statewide Not reported One local office: From January 1 Oct 30 2010, county was required to meet 70 percent timeliness. This benchmark increased to 90 percent for subsequent months. Not reported Percent of documents processed within 30 days Locally Not reported Not reported Not reported Applications in Intake Mode over 30 days Locally Not reported Not reported Not reported Notes: State collects mandatory accuracy data through QC reviews. Wisconsin (continued) A.236 Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. All measures reported by Milwaukee County have been marked as Statewide because the state has responsibility. However, not all localities within Wisconsin may be required to collect these measures. Description: General: Wisconsin modernized SNAP, which it calls FoodShare, by establishing many initiatives, primarily at the county level. The state and counties actively collect performance measures and data. Milwaukee County and a centralized Enrollment Services Center have been fully administered by the state since January 2010 in response to a lawsuit. Wisconsin staff said they are generally satisfied with the accuracy of their data. County offices described difficulties using call center software for reporting. One county stressed that it took time to get the system operational to the point where they could pull data elements and generate reports. Another office has had difficulty accessing historical data. On the whole, county offices expressed a desire to receive more reports from the state and to have greater system functionality in order to break down data, automatically calculate measures, or retrieve needed reports. Some Wisconsin counties used ARRA funding to support their programs and modernization initiatives. One local office used funds to expand their county call center to take on regional work. They also used ARRA funding to pay for kiosk phone lines, which are linked to the change center. After receiving ARRA funds, another county implemented a call center, kiosks, and expedited online applications. Additionally, funds were used to support the state takeover of FoodShare in one county and fund eligibility workers at the local level. Though no performance measures were created specifically based on ARRA funding, one respondent said that reporting increased as a result of the state takeover of FoodShare. One county has set up a new workflow. Under this arrangement, three workers are dedicated to specific tasks (answering the phone, processing changes, answering general questions) and do not manage caseloads. This allows other caseworkers more time to process cases and handle detailed client phone calls. Six months of data will be evaluated this winter to determine the success of the workflow plan. The office plans to set benchmarks based on these data once they know their averages for certain measures (such as average processing time for scanning and average time to process changes). Measurement Goals: Wisconsin’s performance measures touch on all four FNS goals. They also track performance data in order to track licensing and partnership costs, monitor workers, see the impact of an initiative, and for staffing purposes. All training staff and managers at the Milwaukee call center are state employees, while the eligibility workers are still county employees. Call Center: Wisconsin has established locally run call centers in each county office. The state and county offices collect numerous measures and sets of aggregate data on their call centers. Partnering: Wisconsin offices work with several partners at the state and county levels. Wisconsin primarily collects aggregate data on their partnering arrangements. The terms of the state takeover in Milwaukee established a partnering contract that designated one partner as a community access point . Measures Collected by Partners: Wisconsin partners collect many aggregate data items, including information on application assistance, volume of screenings and applications, outreach, hold times, and use of scanning capabilities for verification through ACCESS. Wisconsin (continued) A.237 Online System: Wisconsin has a statewide online application system, ACCESS, which was developed with a contractor. The state office tracks percent of users selecting Spanish language, percent of screened individuals potentially eligible, percent of applicants who received help to apply online, percent of applications received, percent of multiple applications, and percent of applicants with hardship reason for requesting phone interviews. Staff noted that some outreach projects may offer informal incentives for use of the online application, but those incentives are unknown to SNAP staff. Document Imaging: All Wisconsin counties have document imaging capabilities, which fall under three categories: scan first, process first, or hybrid. Scanned documents are stored in the statewide electronic case file system. While only the number of documents scanned is tracked at the state level, multiple county offices track information beyond this. They collect average processing time, numbers of alerts and documents processed, number of documents received, number of documents scanned, number of tasks complete, and percent of documents scanned. Local offices track the date that documents were received, in order to manage workload. In these offices, documents are scanned as they come in and processing time is determined by looking at the date noted on the documents. One office reported a state-mandated 10-day processing time frame and another reported a 30-day scanning timeframe for process first documents. Kiosks: Wisconsin county offices participating in the study, with one exception, have established kiosks for application access and submission. Some kiosks also provide telephones and\/or application assistance at the kiosks. Some local offices we surveyed collect aggregate data, including: number of screenings completed at a kiosk, number of uses to access online account, and number of uses to submit changes. Waiver of Face-to-Face Interview: All Wisconsin counties are included under the waiver. Both the state and local offices collect aggregate data and performance measures on this initiative. Shortened Interviews: Some local offices have a shortened interview available. One office reviews the percent of total interviews that are shortened, percent of clients electing shortened interviews, and number of clients with shortened interviews. Online Expedited Applications: Some counties we interviewed track information about online expedited applications, such as: percent of applications approved (online expedited and online all), number of online expedited applications approved, and number of online expedited applications received. Application Tracking: Wisconsin local offices we interviewed collect information on application accuracy, approvals and denials, application processing and case characteristics, application receipt, and application processing time. Whether the application source is recorded varies by office; the state has detailed records of application origin, while some counties keep limited or no record of this. Changes Over Time: Some local offices participating in the study have adopted new call center systems and have begun collecting new data as part of the software they are using. In one office, the wait time was previously reviewed weekly. This stopped when the division manager position became vacant. Since the office has hired a new manager, they are anticipating they will monitor this monthly. For nine months in 2009, staff at one local call center would fill out reports on the types of calls they received (people calling with questions or with changes), how long the calls took, and how problems were resolved. Tracking this information became difficult for staff and the call center supervisor was not receiving accurate information. The office abandoned this completely and now uses reports pulled from the call center software. Desired or Planned Future Measures: Wisconsin state staff reported they are interested in seeing how QC data affects reporting. Local offices would like to see: customer satisfaction measures; data on a regional change center, broken down by county; records of documents by origin; measures on Wisconsin (continued) A.238 abandonment rate; current reports on application processing time; and measures on accuracy rates for the waiver of face-to-face interview. Various offices mentioned specific measures, such as: dropped call rate, the point (number of minutes) at which calls are dropped, number of Spanish-language calls, percent of applications approved and denied, number of applications received per month, number of changes per case, and average call duration. Multiple offices noted that developing some of these measures could be possible with more staff time or a greater number of staff. Wisconsin would like to see all states report on participation rate by federal poverty level. Some offices mentioned that any measures would be useful to help streamline reporting, gain efficiencies, and help state to state communication. One office thinks it would be useful to have the ability to cross-match information with other states. This will help them address instances where client does not provide the correct state of residence or work. A.239 Profile: Wyoming State Administered or County Administered: State Number of Local\/County Agencies Interviewed: 2 Number of Partners Interviewed: 0 Initiatives Active in State: Partnering, Waiver of Face\u2010to\u2010Face Interview, Application Tracking Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives PARTNERING (Other) None WAIVER OF FACE-TO-FACE INTERVIEW (Statewide) None APPLICATION TRACKING (Statewide) Percent of Applications Approved After Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Follow the federal standard. If it drops below the federal standard then we explore and monitor further and implement corrective action if needed. No Percent of Recertifications Approved After Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Federal guidelines No Percent of Total Applications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Federal guidelines No Percent of Total Recertifications Approved Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Follow the federal benchmark No Number of Applications Approved: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Follow the federal benchmark No Number of Recertifications Approved: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Follow the federal benchmark No Percent of Applications: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Wyoming (continued) A.240 Measure or Aggregate Data Collected Measure Implemented Statewide or Locally Modernization Goal(s) Performance Standards and Benchmarks Performance Incentives Percent of Recertifications: Paper Submission to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service No No Total Number of Applications Completed Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Follow the federal benchmark No Percent of Complete Applications Denied Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Federal standards for timeliness No Percent of Complete Recertifications Denied Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Federal standards for timeliness No Number of Applications Denied: Paper Submissions to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Federal standards for timeliness No Number of Recertifications Denied: Paper Submissions to Local Office Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Federal standards for timeliness No Average Processing Time of Application Decision Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Follow federal regulations around timeliness No Average Processing Time of Recertification Decision Statewide Program Access, Accuracy & Integrity, Efficiency, Customer Service Federal standards for timeliness No Notes: State collects mandatory accuracy data through QC reviews. Table includes all measures indicated as calculated by state and local offices. If the local office indicated the same measure as a state, the local response was deleted and performance data are listed as Statewide. If a local office reported that they use a measure they receive from a statewide system, the measure is reported as statewide. Description: General: Wyoming has a waiver of the face-to-face interview. Applications are available for download on the state website. Local offices may partner informally with community based organizations. Measurement Goals: Wyoming tracks applications ensure program access, accuracy & integrity, efficiency, and customer service. Call Center: Wyoming does not have a call center. Online System: Wyoming does not have an online system. Document Imaging: Wyoming does not have document imaging. Kiosks: Wyoming does not have kiosks. Wyoming (continued) A.241 Waiver of Face-to-Face Interview: Wyoming does not track any performance data related to the waiver of the face-to-face interview. Recently, the computer system has been modified to record an indicator for the type of interview. Shortened Interviews: The state respondent indicated they do not have a shortened interview process. Online Expedited Applications: Wyoming does not accept expedited applications online. Application Tracking: Wyoming tracks a range of measures and aggregate data related to applications and recertifications, including the percentage submitted, completed, approved, and denied, and the average processing time. Application tracking has been a part of the eligibility system since its inception in 1983. Wyoming also tracks the accuracy measures required by FNS. When a client misses an interview, the office sends a \”notice of missed interview\” letter, advising the client to reschedule. Any time a notice is generated the notice history keeps a record of what notice is sent and the date. Some notices are generated automatically by data entered elsewhere into the system; most are generated by the caseworker. Changes Over Time: Wyoming has been collecting the application measures since 1983. The eligibility system is old and it difficult to make changes or find knowledgeable programming staff. Desired or Planned Future Measures: Recently, the computer system has been modified to record include an indicator for the type of interview. Data by interview type has not yet been reported, but may be in the future. Wyoming is planning new caseload reporting. The report will include timeliness information, such as the date an application has been started or processed. Partnering: At least one local office partners informally with community organizations. This office presents information on their programs to leadership groups, advocacy organizations, and senior centers. Partners refer clients to the local office for services. At the state level, Wyoming partners with JP Morgan to maintain EBT cards. Measures Collected by Partners: We did not contact Wyoming’s partners as part of this study. Page 2 performance standards for 39, and of those, few States assigned benchmarks to gauge and incentives to encourage their success. The performance measures used by the States were also categorized according to FNS’ four modernization goals of Program Access, Accuracy and Integrity, Efficiency, and Customer Service. In general: \uf0b7 For Program Access, States seek to monitor overall access (rather than just for the initiative) and improve program access through efficiencies and correct use of procedures. \uf0b7 For Accuracy and Integrity, States determine staff training needs and ensure that staff uses correct procedures, in addition to tracking overall accuracy. \uf0b7 For Efficiency, States track the initiative usage, seek to improve the customer experience, determine staffing needs and ensure that staff follows procedures. \uf0b7 For Customer Service, States track initiative-specific efficiency and customer experiences, determine staffing needs and ensure that staff follows procedures. This research suggests that performance measurement is a useful and important tool to integrate into new State modernization initiatives, and specifically that: \uf0b7 Performance measurement is most effective when built into the initiative during the design phase. \uf0b7 Establishment of an expected level of performance (standard) for each measure is important to gauge success. \uf0b7 Strong performance measures and standards align with goals for improving overall program outcomes. Summary States and local SNAP offices are collecting a large number of measures. Standards for these measures are less common, and benchmarks and incentives are almost nonexistent. Most measures are used for technology-related initiatives. Measurement in the early stages of implementation can indicate whether an initiative is performing its function as designed. As initiatives mature, measures can be used to study trends and identify potential problems through periodic reviews of reports. For More Information U.S. Department of Agriculture, Food and Nutrition Service, Office of Research and Analysis, Performance Measurement for Supplemental Nutrition Assistance Program Modernization Initiatives: Integrated Report, by Laura Castner, Amy Wodarek O’Reilly, Kevin Conway, Maura Bardos, Emily Sama-Miller, Mathematica Policy Research, Inc. FNS Project Officer: Rosemarie Downer, Alexandria, VA: 2012 (available online at www.fns.usda.gov\/ora). U.S. Department of Agriculture, Food and Nutrition Service, Office of Research and Analysis, Performance Measurement for Supplemental Nutrition Assistance Program Modernization Initiatives: Individual State Findings, by Kathleen Keefe, Emily Sama- Miller, Laura Castner, Maura Bardos, Elizabeth Clary, Sarah Wissel, and Larry Vittoriano, Mathematica Policy Research, Inc. FNS Project Officer: Rosemarie Downer, Alexandria, VA: 2012 (available online at www.fns.usda.gov\/ora). The U.S. Department of Agriculture (USDA) prohibits discrimination in all of its programs and activities on the basis of race, color, national origin, age, disability, and where applicable, sex, marital status, familial status, parental status, religion, sexual orientation, political beliefs, genetic information, reprisal, or because all or part of an individual’s income is derived from any public assistance program. (Not all prohibited bases apply to all programs.) Persons with disabilities who require alternative means for communication of program information (Braille, large print, audiotape, etc.) should contact USDA’s TARGET Center at (202) 720-2600 (voice and TDD). To file a complaint of discrimination, write: USDA, Director, Office of Adjudication, 1400 Independence Avenue, SW, Washington, DC 20250-9410; or call (866) 632-9992 (Toll-free Customer Service), (800) 877- 8339 (Local or Federal relay), or (866) 377-8642 (Relay voice users) or (800) 845-6136 (Spanish Federal- relay). USDA is an equal opportunity provider and employer. Nutrition Assistance Program Report Series Office of Research and Analysis Family Nutrition Programs Performance Measurement for Supplemental Nutrition Assistance Program Modernization Initiatives: Integrated Report Final Report United States Food and Department of Nutrition Agriculture Service December 2012 Non-Discrimination Policy The U.S. Department of Agriculture (USDA) prohibits discrimination in all of its programs and activities on the basis of race, color, national origin, age, disability, and where applicable, sex, marital status, familial status, parental status, religion, sexual orientation, political beliefs, genetic information, reprisal, or because all or part of an individual’s income is derived from any public assistance program. (Not all prohibited bases apply to all programs.) Persons with disabilities who require alternative means for communication of program information (Braille, large print, audiotape, etc.) should contact USDA’s TARGET Center at (202) 720-2600 (voice and TDD). To file a complaint of discrimination, write: USDA, Director, Office of Adjudication, 1400 Independence Ave., SW, Washington, DC 20250-9410 or call (866) 632-9992 (Toll-free Customer Service), (800) 877-8339 (Local or Federal relay), (866) 377-8642 (Relay voice users). USDA is an equal opportunity provider and employer. Performance Measurement for Supplemental Nutrition Assistance Program Modernization Initiatives: Integrated Report Final Report Authors: Laura Castner Amy Wodarek O’Reilly Kevin Conway Maura Bardos Emily Sama-Miller Submitted by: Submitted to: Mathematica Policy Research Office of Research and Analysis 1100 1st Street, NE, 12th Floor Food and Nutrition Service Washington, DC 20002-4221 3101 Park Center Drive Telephone: (202) 484-9220 Alexandria, VA 22302-1500 Facsimile: (202) 863-1763 Project Director: Project Officer: Laura Castner Rosemarie Downer This study was conducted under Contract number AG-3198-D-08-0105 with the Food and Nutrition Service. This report is available on the Food and Nutrition Service website: http:\/\/www.fns.usda.gov Suggested Citation: U.S. Department of Agriculture, Food and Nutrition Service, Office of Research and Analysis, Performance Measurement for Supplemental Nutrition Assistance Program Modernization Initiatives: Integrated Report, by Laura Castner, Amy Wodarek O’Reilly, Kevin Conway, Maura Bardos and Emily Sama-Miller, Project Officer: Rosemarie Downer, Alexandria, VA: 2012. United States Food and Department of Nutrition Agriculture Service Family Programs Report December 2012 iii ACKNOWLEDGMENT This report was prepared by Laura Castner, Amy Wodarek O’Reilly, Kevin Conway, Maura Bardos, and Emily Sama-Miller of Mathematica Policy Research. Several additional staff members contributed substantially to the report as interviewers: Lauren Akers, Elizabeth Clary, Kathleen Keefe, Elizabeth Laird, Alexander Millar, Antoniya Owens, Sarah Wissel, and Larry Vittoriano. The manuscript was prepared by Jackie McGee. The authors would like to thank Rosemarie Downer, Anita Singh, John Knaus, and regional staff from the U.S. Department of Agriculture’s Food and Nutrition Service (FNS). We also greatly appreciate the invaluable efforts of the state and local SNAP staff and partnering agencies in responding to our survey and participating in telephone and in-person interviews. v CONTENTS EXECUTIVE SUMMARY …………………………………………………………………………………… xiii I INTRODUCTION ………………………………………………………………………………… 1 A. What Is Modernization?…………………………………………………………………. 3 B. Why Modernize? ………………………………………………………………………….. 3 C. What Modernization Initiatives Have States Put in Place? …………………….. 4 D. Why Is Measurement Important? …………………………………………………….. 5 E. How Are Measures Currently Used for Federal Oversight? …………………… 5 F. How Have States Used Performance Measures? …………………………………. 6 1. Early Stages of Implementation ………………………………………………… 7 2. Maintenance Stage of Implementation ……………………………………….. 7 3. Real Time \/ Daily Measures ……………………………………………………… 8 4. Focus on Quality and Timeliness …………………………………………….. 10 G. How Are Measures Assessed? ………………………………………………………. 11 H. Study Objectives and Research Questions ………………………………………. 12 II METHODS ………………………………………………………………………………………. 15 A. Phase 1: Preliminary Assessment and Instrument Development …………. 15 1. Introduction ………………………………………………………………………… 15 2. Who We Interviewed ……………………………………………………………… 15 3. What We Learned ………………………………………………………………….. 16 4. A Limited Set of Initiatives to Examine ……………………………………… 17 5. Data Collection Instruments …………………………………………………… 18 B. Phase 2: Respondent Selection and Data Collection …………………………. 19 1. Selecting Respondents ………………………………………………………….. 19 2. Responses Received ……………………………………………………………… 21 3. Level of Response ………………………………………………………………… 21 III SUMMARY OF PERFORMANCE MEASURES ACROSS INITIATIVES …………………. 23 A. Purpose of Measures by Modernization Goal …………………………………… 23 Contents Mathematica Policy Research vi III (continued) B. Key Performance Measures and Gaps, by Initiative …………………………… 25 1. Call Center ………………………………………………………………………….. 25 2. Online System ……………………………………………………………………… 27 3. Document Imaging ……………………………………………………………….. 28 4. Kiosks ………………………………………………………………………………… 29 5. Partnering (for Application Assistance) …………………………………….. 29 6. Waiver of Face-to-Face Interview ……………………………………………… 30 7. Shortened Interviews …………………………………………………………….. 30 8. Expedited Applications\u2014Submitted Online ……………………………….. 31 C. General Gaps in Performance Measurement ……………………………………. 31 D. Use of Standards, Benchmarks, and Incentives ………………………………… 34 E. Relationship of Measures to SNAP Lifecycle ……………………………………. 36 IV LIMITATIONS TO USE OF MEASURES AND STANDARDS FOR CROSS- STATE COMPARISONS ………………………………………………………………………. 39 A. Contextual Limitations ……………………………………………………………….. 39 1. Initiative Scope and Features ………………………………………………….. 39 2. System Limitations ……………………………………………………………….. 40 3. Terminology Differences ……………………………………………………….. 41 4. Varying Measurement Motivations …………………………………………… 41 B. Data Limitations ………………………………………………………………………… 41 1. Accuracy …………………………………………………………………………….. 42 2. Validity ……………………………………………………………………………….. 42 3. Reliability ……………………………………………………………………………. 42 V OPPORTUNITIES AND CHALLENGES FOR UNIFORMLY MEASURING INITIATIVES …………………………………………………………………………………….. 43 A. Criteria for Inclusion in the Proposed Set of Measures ……………………… 45 1. Addressing Goals for Modernization ………………………………………… 45 2. Feasibility as a Reporting Requirement …………………………………….. 46 B. Proposed Set of Measures ……………………………………………………………. 46 1. Application Tracking …………………………………………………………….. 46 2. Call Centers ………………………………………………………………………… 47 3. Online Systems ……………………………………………………………………. 49 4. Document Imaging ……………………………………………………………….. 50 5. Partnering (Application Assistance) …………………………………………. 52 6. Waiver of Face-to-Face Interview ……………………………………………… 53 Contents Mathematica Policy Research vii VI CONCLUSION ………………………………………………………………………………….. 55 A. Summary of Study Findings …………………………………………………………. 55 B. Setting Measurement Requirements: Recommendations and Limitations ……………………………………………………………………………….. 56 C. Next Steps ………………………………………………………………………………… 56 REFERENCES ………………………………………………………………………………………………… 57 GLOSSARY …………………………………………………………………………………………………… 59 APPENDIX A: PERFORMANCE MEASURES AND AGGREGATE DATA ix TABLES II.1 Modes of Data Collection for Each Respondent Type ……………………………… 20 III.1 Types of Measures States Are Collecting to Assess Progress Toward Modernization Goals ………………………………………………………………………… 24 III.2 Number of Measures States Are Collecting to Assess Progress Toward Modernization Goals …………………………………………………………….. 26 III.3 Top Performance Measures: Call Center ………………………………………………. 27 III.4 Top Performance Measures: Online System ………………………………………….. 28 III.5 Top Performance Measures: Document Imaging ……………………………………. 29 III.6 Top Performance Measures: Kiosks …………………………………………………….. 29 III.7 Top Performance Measures: Partnering ……………………………………………….. 30 III.8 Top Performance Measures: Waiver of Face-to-Face Interview ………………….. 30 III.9 Top Performance Measures: Shortened Interviews …………………………………. 31 III.10 Top Performance Measures: Expedited Applications\u2014Submitted Online …………………………………………………………………………………………… 31 III.11 Number of States Able to Track Application Status, by Application Source …………………………………………………………………………………………… 34 III.12 Use of Standards, Benchmarks, and Incentives ……………………………………… 35 III.13 Measures and Reported Purpose by Stages of the Lifecycle …………………….. 37 V.1 Recommended Application Tracking Measures …………………………………….. 47 V.2 Recommended Call Center Measures ………………………………………………….. 48 V.3 Recommended Online System Measures ……………………………………………… 50 V.4 Recommended Document Imaging Measures ……………………………………….. 51 V.5 Recommended Partnering Measures …………………………………………………… 52 V.6 Recommended Measures for Waiver of the Face-to-Face Interview ……………. 53 Tables Mathematica Policy Research x A.1 Measures and Aggregate Data for Call Centers …………………………………….A.3 A.2 Measures and Aggregate Data for Online Systems ………………………………..A.5 A.3 Measures and Aggregate Data for Document Imaging ……………………………A.7 A.4 Measures and Aggregate Data for Kiosks …………………………………………….A.8 A.5 Measures and Aggregate Data for Partnering ……………………………………….A.9 A.6 Measures and Aggregate Data for Waiver of Face-to-Face Interview ………..A.10 A.7 Measures and Aggregate Data for Shortened Interview ………………………..A.11 A.8 Measures and Aggregate Data for Complete Expedited Interviews Online …………………………………………………………………………………………A.12 A.9 Measures and Aggregate Data for Application Tracking ……………………….A.13 xi FIGURES I.1 Number of Real-Time and Daily Call Center Measures Calculated by States ……………………………………………………………………………………………… 9 I.2 Number of Real-Time and Daily Application Tracking Measures Calculated by States ………………………………………………………………………… 10 II.1 Number and Combination of Respondents, by State ………………………………. 21 xiii EXECUTIVE SUMMARY Over the last decade, several factors have come together and substantially affected the administration of the Supplemental Nutrition Assistance Program (SNAP). For example, the number of participants rose from 17 million people in fiscal year 2000 to 40 million in fiscal year 2010. At the same time, state budget crises led to staff cuts and federal agencies have emphasized increasing access to SNAP and participation among eligible individuals, while also increasing accuracy, and have provided incentives for states who achieve this. The U.S. Department of Agriculture Food and Nutrition Service (FNS), with responsibility for overall administration of SNAP, has supported the states in these efforts by providing them with several policy options to better tailor the program to the states’ specific needs. At the same time, substantial growth has occurred in the acceptance and use of technology for accessing a wide range of services. States have responded to these factors by seeking out ways to improve the efficiency of their service delivery with fewer staff and without affecting accuracy\u2014collectively referred to as modernization. Modernization initiatives can include the following types of activities: policy simplification, restructuring organizational and administrative functions, partnering with other organizations, and the use of technology. Major changes to any program do not always go smoothly, however. Without methodical introduction of the changes and careful monitoring of how well the initiatives are working to achieve the states’ goals, service delivery can be affected negatively. This report is an exploratory study intended to add to the growing body of knowledge about how states are modernizing their programs by focusing specifically on how they monitor and measure the success of discrete aspects of their initiatives. For example, when SNAP offices institute call centers to handle inquiries from participants, eliminating the need for participants to talk to a specific person who may not be available when they call, how does the state know if participants’ calls are getting through to the call center? Does the state know how many participants hang up because they were on hold for too long? Does it know if a participant has to call back repeatedly because the person who answered the phone could not answer a question? In this report, we present our findings on how states are measuring the performance of a specific set of modernization initiatives, which we selected in consultation with FNS. We discuss the range of measures, identify those most commonly used across sites, compare and contrast how the measures are defined across sites, and discuss standards and incentives associated with the measures. We also suggest a standard set of measures for each initiative that FNS and states could consider using to systematically track the performance of each initiative. Methods To conduct the study, we used a two-phase approach. In the first phase, we interviewed a broad group of stakeholders (including SNAP staff at the national, regional, and state levels, and advocates) to obtain their perspectives on state performance measurement activities, identified the key modernization initiatives to include in the study, and developed the data collection instruments. During the second phase, we collected data from 44 states and the District of Columbia, more than 100 local agencies within those states, and more than 50 community and business partners who work with the states and local agencies. Data collection with the state and local agencies included an Executive Summary Mathematica Policy Research xiv electronic survey through which we gathered information about their modernization initiatives and requested the names and definitions of each performance measure they collect for each initiative. We followed up on the information they provided through in-depth telephone interviews. We also conducted telephone interviews with staff from the community and business partners about the performance measures states ask them to collect and those they collect for their own purposes. Background The primary reason states have given for choosing to modernize the delivery of SNAP was increased staff caseloads (Rowe et al. 2010). A survey, conducted in 2008, also identified the economic downturn, state legislation, and staff turnover at the local level as influencing their modernization decisions. Modernizing because of increased caseloads can be categorized as striving to achieve efficiency\u2014one of four goals FNS typically uses to categorize modernization efforts. The others are program access, accuracy and integrity, and customer service, all of which are described here: Efficiency. Efficiency relates to delivering the program at the lowest cost possible without sacrificing the other goals. Program Access. The goal is to ensure that benefits can be accessed by individuals eligible to receive them and that equal program access is available for protected classes of clients. It is affected by the location and operating hours of local SNAP offices, the modes through which applications can be submitted, the level of effort needed to complete the application process, and other factors. Accuracy and Integrity. Maintaining the integrity of SNAP requires that staff accurately determine both eligibility and the benefit amount for households as well as minimize fraud. Ensuring accuracy can include policies about reporting and verifying information. Any change to the responsibilities of and expectations for program staff could inadvertently lead to changes in program integrity. Customer Service. Improving customer service involves ensuring the quality of staff interactions, minimizing the length of time needed to complete the application process, and improving the ease and clarity of that process. However, steps taken to ensure payment accuracy and efficiency can reduce customer satisfaction. New technologies and other major changes to the traditional client services approach also can affect customer satisfaction. Many types of modernization activities fit with these goals. In this study, however, we focus on eight key initiatives: call centers, online systems, document imaging, kiosks, partnering, waiver of face-to-face interviews, shortened interviews, and online expedited applications. We also discuss measurement associated with tracking application submissions, approvals\/denials, and accuracy. Key Study Findings Across the initiatives, we identified the measures the states have in place; examined how they align with FNS’s goals for modernization; and verified the states’ purposes for collecting the measures, how commonly they were used across states, and how often they set an expected level of performance. The key findings are as follows: Executive Summary Mathematica Policy Research xv States and local offices are collecting a large number of measures. Across the eight modernization initiatives we examined, states and local offices reported using 89 different performance measures to assess their activities. We asked the SNAP offices to identify the modernization goals that the measures address. We then categorized the measures within each goal according to their use. For example, among measures that states and local offices identified as assessing progress toward program access, we found that they fit into several categories based on their use: monitoring SNAP as a whole; monitoring the initiative; improving access to SNAP by maximizing staff time and program costs or by having effective procedures; ensuring correct use of procedures; and ensuring that customers are able to make effective use of the initiative. In Table 1, we show the number of different measures associated with each goal, further categorized by each measure’s use (note that some measures may be included under more than one goal). The number of measures for each theme yields insights into which areas state and local offices focus their measurement activities: For Program Access, states seek to monitor overall access (rather than just at the initiative level) and improve program access through efficiencies and correct use of procedures. With Accuracy and Integrity as a goal, states determine staff training needs and ensure that staff use correct procedures, in addition to tracking overall accuracy. For Efficiency, states track the initiative usage, seek to improve the customer experience, reduce staff time, and ensure that staff are following correct procedures. For Customer Service, states track initiative-specific efficiency and customer experiences, determine staffing needs, and ensure that staff are following correct procedures. Standards are less common and benchmarks and incentives are rare. Performance measurement typically includes the use of standards, benchmarks, and incentives to assess performance of the selected measures. Standards and benchmarks establish the criteria against which performance is measured. A standard is the desired outcome of an activity\u2014for example, answering a call within three minutes\u2014while a benchmark is the desired rate of success\u2014for example, answering at least 80 percent of calls within three minutes. An incentive is a motivator for performance and either can be rewards and public recognition for the highest performers and\/or some combination of sanctions and technical assistance for the lowest performers. If no standards exist, there is no way of knowing if an activity is meeting its goals. Executive Summary Mathematica Policy Research xvi Table 1. Types of Measures States Are Collecting to Assess Progress Toward Modernization Goals FNS’s Modernization Goal Purpose of Measure Number of Measuresa Number of States Collecting Measures Program Access Monitor combined result of initiatives to increase program access 43 27 Improve program access through maximizing staff time and agency cost efficiencies 37 20 Ensure correct procedures are being used by staff 34 17 Monitor use of initiative 30 12 Improve program access through establishing effective procedures 21 14 Determine if customers are able to use the initiative 5 3 Accuracy and Integrity Track accuracy related to the program as a whole as opposed to a specific initiative 84 32 Assess general accuracy used in eligibility determinations as part of FNS quality reviews 24 20 Determine staff training needs and ensure correct procedures are used by staff 17 16 Reduce overall errors or improve overall accuracy 12 10 Identify overall error rate or trends, not related to a specific initiative 9 11 Identify error rate or trends for a specific initiative 5 8 Reduce errors or improve accuracy for a specific initiative 3 2 Efficiency Track usage or system load 73 26 Reduce staff time and agency costs 47 21 Improve customer experience 46 20 Determine staffing needs and ensure that correct procedures are used by staff 30 19 Assess initiative-specific accuracy used in eligibility determinations as part of FNS quality reviews 20 12 Customer Service Assess general customer service 61 26 Increase efficiency\/improve customer experience for a specific initiative 52 24 Identify barriers to program access for specific initiative 40 12 Determine staffing needs and ensure that correct procedures are used by staff 32 17 Assess general accuracy used in eligibility determinations as part of FNS quality reviews 27 13 Eliminate or reduce barriers to program access for customers 8 6 Note: Totals are based on responses from the 45 states that participated in the study. Each respondent indicated its own purpose for collecting the measure, and those can differ across the state and local offices within a state. The measures in this table include those reported by both offices, so a measure may be captured more than once. a Numbers in this column include performance measures reported for each initiative and application tracking. Executive Summary Mathematica Policy Research xvii Of the 89 measures being collected, states reported developing performance standards for 39 measures and, of those, a few are assigned benchmarks and incentives to gauge their success. In Table 2, we present, by initiative, the number of measures with standards or benchmarks and summarize how states use them. Table 2. Use of Standards, Benchmarks, and Incentives Initiative (Number of Measures Using Standards or Benchmarks) Use of Standards, Benchmarks, and Incentives Call Center (20) Standards: Evaluate performance of individual call center agent or a group of agents Incoming Calls Benchmarks: Assess volume of calls through specifying waiting and call duration times Incentives: Determine high or low staff performance review; penalize vendor poor performance (when contracting with a vendor) Standards: Track incoming call handling Phone System Benchmarks: Resolve maximum number of calls within phone system or via agent Incentives: Not reporteda Standards: Track processing of all changes and their processing time Change Reporting Benchmarks: Not provided Incentive: Determine high staff performance review rating Online System (3) Standards: Track portion of applications received online Online Application Benchmarks: Increase usage of application option Incentives: Not used Not used Online Screening and Online Account Access Document Imaging (6) Standards: Assess quantity of documents scanned, by document type, within specified timeframe Benchmarks: Scan all documents within a certain timeframe Incentives: Satisfaction of achieving mandate; penalize vendor poor performance (when contracting with a vendor) Kiosks (0) Not used Partnering\u2014Application Assistance (2) Standards: Monitor partners providing application assistance Benchmarks: Ensure that all partners provide assistance Incentives: Pay partners for approved applications for which they provided assistance Waiver of Face-to-Face Interviews (6) Standards: Monitor the proportion of interviews conducted face-to-face and telephonically to track utilization Benchmarks: Ensure that the percentage of telephone interviews does not exceed waiver requirement Incentives: Not reporteda Shortened Interviews (0) Not used Expedited Applications (2) Standards: Process expedited applications Benchmarks: Process expedited applications within 7-day timeframe required by FNS Incentives: Not reporteda Note: Totals are based on responses from the 45 states that participated in the study. a Not reported indicates that the respondent identified the use of an incentive but did not describe the specific incentive utilized. Executive Summary Mathematica Policy Research xviii Measures most commonly are used for technology-related initiatives. The most common measures are those for call centers and online systems (see Table 3). The technology built into these systems makes it easier to collect and report performance measures. Thus, states with more technology-driven initiatives have many more performance measures than states without those initiatives. Table 3. Number and Use of Measures States Are Collecting, by Initiative Initiative Number of States with Initiative Number of Different Measures for Initiative Use of Measure Call Center 28 34 Identify staffing needs Identify need to train staff Assess efficiencies and inefficiencies of operations Assess customer service Online System 33 27 Demonstrate program access and increased participation Track utilization of online application feature, including making comparisons with paper applications Document Imaging 24 8 Track system operations Manage staff workload Kiosks 14 1 None reported Partnering 34 3 Track approvals Track partners providing assistance Track partner assistance provided\u2014from monitoring to evaluating effectiveness and reimbursing them for assistance provided Enhance program access Waiver of Face-to- Face Interview 40 12 Show effectiveness of interview option Determine accuracy, including making comparison with face-to-face interviews Comply with federal waiver requirement for performance measure calculations Shortened Interviews 12 2 None reported Online Expedited Applications 20 2 Increase program access Track approvals Note: Totals are based on responses from the 45 states that participated in the study. In most states, staff are driven by federal accuracy and timeliness requirements. Staff at all levels of SNAP agencies indicated that states are not meeting their mission if they do not provide benefits to those eligible, get the beneficiaries the right benefit amount, or achieve these goals in a timely fashion. At times, states may sacrifice the result for certain operational measures of individual initiatives so they can ensure that other, more critical goals, such as FNS timeliness standards, are achieved. If FNS were to require additional measures, states likely would divert resources to meet those requirements. When incentives are available, such as for high accuracy and timeliness rates, states are motivated to outperform other states so as to receive the incentive. Executive Summary Mathematica Policy Research xix Use of measures changes over time: first to assess whether the initiative is working as designed; then to monitor trends; and finally, to allocate resources in real time. In the early stages of implementation, reports can indicate that each piece of the initiative is performing its particular function as planned. The measures provide proof of success to stakeholders and can help secure funding for continued operations. As initiatives mature, performance measurement allows for monitoring. Once staff are confident that the initiative is working as designed, they can analyze trends and identify potential problems through periodic reviews of reports. However, too much monitoring can create unnecessary work for staff if they need to perform additional tasks purely for tracking purposes. As measurement matures, real-time and daily reports help to allocate limited staff resources. Frequent and timely measurement of operations allows managers to assign more workers to the area most in need at a particular moment, such as assigning more agents to answer phones during periods of high call volume, or assigning more eligibility workers to process applications if too many are in danger of failing to meet the timeliness standard. Limitations While many states are collecting performance measures, several factors related to modernization and performance measures affect cross-state comparisons. Two particular aspects are contextual limitations (related to the context and environment in which the initiatives operate and their functions) and data limitations (related to the ability to calculate accurate, valid, and reliable measures). Contextual Limitations. State and local offices operate and implement modernization initiatives in different contexts, which influence their choice of performance measures and standards and their observable outcomes. Common contextual limitations include the following: Initiative Scope and Features. The size and breadth of an initiative, especially whether it serves the entire state or just a region, and what services are provided through that initiative. System Limitations. The inability of eligibility and reporting systems to capture and report measures of interest to each state. Terminology Differences. The variations in how measures are worded and defined across states show that states are measuring different performance aspects. Varying Measurement Motivations. Motivations for using measures and standards vary across states, with some choosing measures to manage staff workloads and others setting standards to comply with legislative or other mandates. Data Limitations. While contextual limitations impact cross-state analysis of measures, the data used to calculate the measures also set limits to performance evaluations and can jeopardize interpretations of individual measures. Performance measures are subject to three main types of data limitations: accuracy, validity, and reliability. Accuracy. When data are used to calculate performance measures, how they are entered impacts the quality of the measure. Data entered automatically are likely subject to fewer errors and less subjectivity. In many cases, however, staff must manually track data for measures, such as the time spent on a call or the number of documents scanned in a day. Manual entries and calculations are more subject to error. Executive Summary Mathematica Policy Research xx Validity. This refers to whether or not states actually are measuring what they intend to measure, or if their measure includes data they do not intend to capture. Respondents sometimes indicated validity concerns regarding the measures they reported, but did not elaborate. Reliability. Data reliability addresses the consistency of how measures are calculated over time and across localities or states. To be reliable, a measure must not have changed recently\u2014either in how it is defined or calculated. In addition, the definition must reflect the same concept regardless of where it is being collected. For example, hold time in one call center must reflect the same period of time in another, which we did not always find to be the case. Setting Measurement Requirements: Recommendations and Limitations Measurement improves the chances that an initiative will succeed, both at implementation and during ongoing operations. It also improves the chances even more when measures are built into initiatives during the design stage. Developing measures helps states clearly articulate the purpose of the initiative and set a preliminary expected level of performance. Developing measures and stating goals also can help ensure that the initiative is helping them achieve their mission of providing accurate benefits in a timely fashion. With many states developing and launching broadly similar initiatives, sharing lessons learned and discussing expectations for performance can help later-adopting states avoid the pitfalls faced by the early-adopting states. However, each state operates SNAP within its own set of circumstances. States differ in the computer capabilities supporting their eligibility systems, assistance programs included in those systems, combinations of modernization initiatives, functionalities for the modernization initiatives, and abilities to calculate measures. Given the unique environment in which each state operates, FNS and state staff must be cautious in setting expectations for a modernization initiative based on the performance achieved by other states. Based on our research, we make the following recommendations for FNS and states when launching new initiatives: 1. Performance of initiatives should be measured\u2014states acknowledge that where measurement occurs, performance improves. 2. Performance measurement should be built into the initiative during the design stage. 3. States should have an expected level of performance (standard) for each measure. 4. The performance measures and standards should be tied to improving overall program outcomes. However, because of the limitations noted above, and because of the exploratory nature of this study, we did not develop a detailed understanding of every measure in every state; therefore, we do not recommend a set of performance measures for each initiative for use across all states in all situations. Instead, we present (see Table 4) a limited set of measures that appears to be useful within a broad range of state operational contexts. States implementing new initiatives could review this list for suggested measures. Although we hope that most will be appropriate for the states, some measures may not fit within a particular state’s context, while others may be a close fit, but need slight modification to be appropriate. States will need to make these decisions based on their specific goals for implementation, the functionality of their initiative, the other initiatives they have in place, Executive Summary Mathematica Policy Research xxi the unique features the initiative may entail, and the integration of SNAP with other assistance programs. Table 4. Recommended Measures Measure Name Definition Rationale Comments Application Tracking Percent of Applications Submitted by Source Number of applications submitted through each mode (mail\/fax, in- person, online\/kiosk, call center, through a partner), divided by total number of applications. (Maximum of five measures, depending on activities in a state.) Helps the state to track the means by which applicants are applying. Even states that can track whether an application was submitted online often cannot track when it was delivered by mail separately from those dropped off at the office. These two may need to be combined. Percent of Applications from Each Source that Are Approved Number of applications submitted through each mode and subsequently approved, divided by number of applications submitted by mode. (Maximum of five measures, depending on activities in a state.) Helps the state to understand whether one mode of application tends to be used by more people who are not eligible or do not complete the process. Excludes from both the numerator and denominator the number of applications in process during the period data are collected. Call Center Calls Received per Active Case Number of calls, divided by number of active cases. Demonstrates volume of calls in relation to caseload size, making the measure more comparable across states and regions. Include in the numerator and denominator only the cases in the area covered by the call center. States with online account access may have much lower call volume; states with more functionality may have higher call volume. High numbers of calls from nonparticipants would not be reflected in the denominator. Average Number of Calls Handled per Agent Number of calls handled by agents, divided by number of agents. Demonstrates load on individual agent Need to set standards based on functionality of systems\u2014 those with automated systems likely will see lengthier calls and thus have a lower expectation for number of calls possible per agent. Percent of Calls Abandoned Number of calls abandoned before reaching automated system or agent. Potentially demonstrates dissatisfaction of callers with amount of time required to wait for service. If recorded information is played during hold time, it may answer the caller’s question and not truly be an abandoned call, even though it would be recorded as such. Average Time to Answer by Automated Total time callers wait to access automated system, divided by total number Potentially demonstrates dissatisfaction of callers with amount of time Will not apply to call centers with no automated response unit. Executive Summary Mathematica Policy Research Table 4 (continued) xxii Measure Name Definition Rationale Comments Response Unit of calls into system. required to wait for service. Average Waiting Time to Speak to Agent Total time callers wait before talking to an agent, divided by total number of calls into system. Potentially demonstrates dissatisfaction of callers with amount of time required to wait for service. Should not include time spent being directed by automated unit; clock should not stop if an agent answers only to place caller on hold. Online System Percent of Applications Received Online Number of applications submitted online, divided by the total number of applications received. Demonstrates use of online system in relation to other modes of application. States may differ in how they count unsigned applications that must be printed and sent to the applicant for approval because an unsigned application does not constitute a completed application. Percent of SNAP Caseload with an Online Account Number of active SNAP accounts, divided by size of SNAP caseload. Demonstrates use of system by entire caseload. Defining active as accounts associated with ongoing cases. Average Number of Logins per Account Number of logins, divided by number of active accounts. Demonstrates use of system by those who have accounts. Defining active as accounts associated with ongoing cases. Average Number of Changes Submitted per Account Number of changes submitted, divided by number of active accounts. Potentially demonstrates that system is difficult to use or not meeting clients’ needs. Defining active as accounts associated with ongoing cases. Document Imaging Percent of Documents Scanned within 24 Hours Number of documents successfully scanned within 24 hours of receipt, divided by total number of documents to be scanned received in the same 24-hour period. Demonstrates ability of system to process documents efficiently. All documents should be included\u2014even those that are damaged\u2014to track percent requiring manual scanning. Some states are not attempting to achieve scanning within 24 hours, so a longer timeframe may be preferable for them. Percent of Documents Attached to Case within One Business Day of Being Scanned Number of documents attached to case record by close of business on the business day following imaging of the document, divided by number of documents imaged the previous day. Documents need to be accessible to case workers quickly for the system to work well. They are not accessible until they are attached to the file. Not all documents may have enough information in them to attach to a case immediately. Low attachment rate could reflect errors by the person submitting the document, as well as lack of ability of staff to attach documents to cases. Does not reflect how many documents never get attached to a case. Percent of Orphaned Documents Number of documents scanned exactly four weeks prior but that are still not attached to a case, divided by daily average of scanned documents. Tracks extent to which scanned documents could not be tied to a specific case and thus are unusable for purposes of eligibility determination and\/or recertification. Difficult to set the time period over which to measure\u2014a worker may be able to attach the document days or weeks later if the sender provides information that can assist the worker in identifying the case. Executive Summary Mathematica Policy Research Table 4 (continued) xxiii Measure Name Definition Rationale Comments Partnering Number of Partners Providing Application Assistance Number of partners who have formally declared they are working with the state, even though there may be no formal agreement on how they conduct this activity or how many people they are trying to serve. Helps the state to track the organizations, eases communication of changes in procedures. Definitions of application assistance may differ across states, making this more of a gauge for internal purposes than for comparison across states. Number of Clients Assisted by Partners Report from partners of number of clients they assisted with the application. Helps the state understand the use of the partners by their residents and how much need there is for providing continued training to partner staff. Not all those receiving assistance ultimately will submit their applications. Waiver of Face-to-Face Interview Percent of Application Interviews Conducted by Telephone Number of application interviews conducted by telephone, divided by number of application interviews. Demonstrates the use of the waiver at application. Percent of Recertification Interviews Conducted by Telephone Number of recertification interviews conducted by telephone, divided by number of recertification interviews. Demonstrates the use of the waiver at recertification. Error Rate for Cases with Telephone Interviews Number of cases in error that included a telephone interview, divided by number of cases in error. Demonstrates the ability of the eligibility worker and applicant to communicate all necessary information over the phone rather than through an in-person meeting. Error rates generally are calculated through formal quality control reviews, and some states can separate the error rate based on the interview type. For this measure to be accurate and unbiased, it likely would require a change in the QC review process\u2014the sampling would need to account for both types of interviews. 1 I. INTRODUCTION Over the last decade, states have initiated a wide range of enhancements to their administration and operation of the Supplemental Nutrition Assistance Program (SNAP). These enhancements, generally captured through the term modernization, can include increasing the use of technology and the Internet, specializing staff roles, expanding office hours, aligning eligibility policies across multiple assistance programs, partnering with community-based organizations, and other initiatives. Recently, modernization activities have received substantial attention from stakeholders: state legislators and administrators are interested in ways to reduce costs, advocacy agencies watch for potential negative impacts on service delivery, community partners seek to link clients with all services for which they are eligible, and the federal government must ensure the accuracy and timeliness of benefit delivery. The underlying question for the stakeholders, then, is How well are the initiatives working? In this report, we examine how states are determining the performance of their initiatives. We identify the measures states have in place for each initiative and how these measures compare across states. However, it is helpful to first review certain factors that have led to and influenced modernization initiatives in the states, which include the following: Number of SNAP participants rose sharply. The number of participants in SNAP more than doubled over the last 10 years. In fiscal year 2000, more than 17 million people participated in an average month; in fiscal year 2010, more than 40 million did. State budget crises led to staff cuts and smaller operating budgets. Almost all states have experienced budget shortfalls, and because states generally cannot run a budget deficit, they must find ways to balance their budgets. At least 44 states and the District of Columbia have achieved this in part by making cuts in their workforce\u2014 eliminating or not filling jobs, imposing furloughs, or through other means (Johnson et al. 2011). In addition, states pay approximately 50 percent of the costs to administer SNAP, so they look beyond staff cuts to find ways to reduce the SNAP operating budget. States are attempting to integrate multiple assistance programs. States can find additional cost savings in integrating programs such as Temporary Assistance to Needy Families (TANF), Supplemental Security Income (SSI), and Medicaid. This sort of integration may allow them to use one computer system to determine eligibility for all programs rather than maintaining different systems for each program. Similarly, providing one phone number through which clients can access all programs and having a computer direct the calls to the right department can reduce the number of state staff needed to answer phones. Federal performance efforts led to national SNAP participation goals. In July 2002, the Office of Management and Budget required the U.S. Department of Agriculture (USDA) and other federal agencies to complete the Performance Assessment Rating Tool (PART). Using the PART, agencies had to establish specific goals and show progress over time in meeting them. One of the goals of the Food and Nutrition Service (FNS), the USDA agency responsible for SNAP, is to improve program participation rates. Introduction Mathematica Policy Research 2 FNS increased pressure on states to raise their participation rates without decreasing accuracy. Because they administer their own programs, FNS relies on states to increase their participation. In 2003, FNS established monetary rewards to encourage states to increase participation. In 2009, it split $12 million dollars among the eight states with the most improved and highest rates of participation. However, FNS also must ensure that states are not providing benefits to people who are not eligible, so it also established both sanctions for those with high rates of error and rewards for those with low rates of error, with the highs and lows being determined in relation to other states. FNS provided several policy options for states to tailor programs to their particular needs. FNS has had a long-standing policy allowing states to apply for waivers regarding certain policies. In addition, in the early 2000s, FNS began to give states increased flexibility in setting policies. For example, states may align their asset rules with other assistance programs to make applying for multiple programs easier for residents, simplify the conditions under which SNAP participants report changes in their income, and provide support for those leaving cash assistance by allowing their SNAP benefits to remain constant for a period of time. Technology is more widespread and its improvements have increased its use. The past decade has seen remarkable growth in the use of the Internet, both in the amount of general information available and the ability of individuals to access or submit information unique to them, such as through online banking or online benefit account access. States increasingly accept electronic or telephonic signatures. Several states now allow electronic or telephonic signatures when residents submit required information by phone or Internet. Increased need among residents, decreased availability of staff, smaller state budgets, and monetary incentives related to quality have led states to find ways to improve the efficiency of their service delivery without affecting accuracy. Monetary incentives to increase participation have led them to improve their customer service and access. Flexibility in setting policy, paired with technological improvements, has given states many options for exactly how they achieve the improvements. Major changes to programs do not always happen smoothly, however. Without a methodical introduction of the changes and careful monitoring of how well they are achieving the states’ goals, service delivery can be severely and negatively affected. This report is an exploratory study intended to add to the growing body of knowledge on how states are modernizing their programs by focusing specifically on how they are monitoring and measuring the success of discrete aspects of their efforts. For example, when SNAP offices initiate call centers to handle inquiries from participants, eliminating the need for callers to talk to a specific person who may not be available when they call, how does the state know if the people were able to get through to the call center? Does the state know how many participants hang up because they were on hold for too long? Does it have information on whether a participant has to call back repeatedly because the person who answered the phone was not able to answer a question? We present our findings on how states are measuring performance of a specific set of modernization initiatives, which we selected in consultation with FNS. The findings are based on data obtained through surveys, telephone interviews, and in-person interviews with state and local Introduction Mathematica Policy Research 3 SNAP administrators and organizations that work for or partner with SNAP agencies. We discuss the range of measures, identify those most commonly used across states, compare and contrast how the measures are defined across states, and discuss standards and incentives associated with the measures. We also suggest a standard set of measures for each initiative that FNS and states could consider using to systematically track the performance of each. In the remainder of this chapter, we first review how we define modernization for the purpose of this report and why states develop modernization initiatives. We then discuss how both the federal government and states use performance measurement of initiatives. Finally, we review how we will assess measures and discuss the objectives and research questions for the study. In Chapter II, we discuss the methods and sources of data. We provide a summary of the measures in Chapter III\u2014tying them to FNS’s goals for modernization and the states’ purposes for measurement, and discuss limitations in cross-state comparison in Chapter IV. In Chapter V, we present a set of measures for each initiative for FNS and the states for consideration when implementing the initiatives. Finally, in Chapter VI, we summarize our findings and the use of performance measurement in the states. Additionally, we provide a glossary of terms used throughout the study\u2014including descriptions of modernization initiatives and their functionalities. In Appendix A, we provide a list of all of the measures states are collecting, by initiative (using the word measure in its broadest sense\u2014a wide range of data collected to assess performance). A. What Is Modernization? The term modernization refers to more than just using technology to accept SNAP applications or convey information to applicants and participants. Rather, modernization refers to a broad set of changes that can be adopted to improve efficiency and access to the program and\/or enhance customer service. In some states, modernization refers to a sweeping set of changes\u2014 essentially a reconfiguration of the business model for administering SNAP. In other states, it refers to a handful of changes but not a fundamental shift in the state’s business model. Modernization changes are not always adopted statewide. Some states pilot test modernization changes in parts of a state. Additionally, in about one-fifth of the states, SNAP is administered by counties. In those states, some counties have instituted their own modernization initiatives. B. Why Modernize? The primary reason states have given as to why they chose to modernize the delivery of SNAP was increased caseloads for their staff (Rowe et al. 2010). The survey conducted by Rowe et al. in 2008, also identified that the economic downturn, state legislation, and staff turnover at the local level influenced their modernization decisions. Introduction Mathematica Policy Research 4 Modernizing SNAP because of increased caseloads can be viewed as an effort to achieve efficiency\u2014one of four primary FNS goals. For example, Florida completely redesigned its business model in 2004 to improve efficiency in response to a need to cut costs drastically in a very short period of time. The other goals are to improve program access, increase accuracy and integrity, and improve customer service. Utah, for example, rolled out an integrated software management system in 2003 that improved access to several assistance programs, including SNAP, by automatically assessing a person’s eligibility for all related programs at one time. C. What Modernization Initiatives Have States Put in Place? Along with classifying the goals for modernization into four categories, we also can classify the components of modernization into four categories: Policy Simplification. Policy simplification can improve operational efficiency by reducing the time needed for staff to collect and process information. It also can increase program access and customer service by reducing the burden of applying and participating. Examples include combining applications across programs, reducing the need for documentation to verify information, increasing the use of telephone and\/or shortened interviews at application and recertification, and simplifying income reporting policies. Restructuring of Organizational and Administrative Functions. Streamlining organizational and administrative functions can improve operational efficiency. Examples include specializing staff roles and eliminating the caseworker model, centralizing functions, closing local offices, and expanding or reducing office hours. Partnering with Other Organizations. States can reduce administrative costs and increase program access by partnering with commercial businesses and nonprofit community-based partners. Business partners often take over specific support activities, such as sorting and mailing or developing computer systems, saving the agency from having to employ staff with these specialized roles. Community-based partners typically provide access to computers and\/or application assistance. They often have a strong presence within a certain geographic area or with a specific demographic. FNS Categorization of Goals for Modernization Program Access. The goal is to ensure that benefits can be accessed by individuals who are eligible to receive them. It is affected by the location and operating hours of local SNAP offices, the modes through which applications can be submitted, the level of effort needed to complete the application process, and other factors. Accuracy and Integrity. Continued support for SNAP requires that staff accurately determine both eligibility and the benefit amount for households. Ensuring accuracy can include policies about reporting and verifying information; ensuring integrity may include minimizing fraud and providing equal program access for protected classes of clients. Any change to the responsibilities of and expectations for program staff could inadvertently lead to changes in program integrity. Efficiency. Efficiency relates to delivering the program at the lowest cost possible without sacrificing the other goals. Customer Service. Improving customer service involves ensuring the quality of staff interactions, minimizing the length of time needed to complete the application process, and improving the ease and clarity of that process. Steps taken to ensure payment accuracy and efficiency can reduce customer satisfaction. New technologies and other major changes to the traditional client services approach also can affect customer satisfaction. Introduction Mathematica Policy Research 5 Technology. Many states are including technology as part of their modernization efforts. States can use technology to improve operating efficiency, increase program access, and increase customer service for clients who prefer to use the technology. Examples include online systems, automated eligibility systems, automated response units (phone systems), telephone call centers and change centers, electronic signatures, data matching, document imaging, and electronic learning. From their 2008 survey, Rowe et al. (2010) found that almost every state either had implemented or piloted some type of initiative related to improving customer access, implemented a technological change, or initiated an information-sharing process. They found that half of the states had initiated organizational changes, online applications, electronic record-keeping, and call centers. D. Why Is Performance Measurement Important? Major changes to systems and policies, especially when they are implemented quickly due to an immediate need or affect a large group of people, often are accompanied by unexpected challenges that can lead to problems or delays in service delivery. If the problems are not caught early, states may face a number of consequences: problems causing high error rates have led to sanctions by FNS, and issues with timeliness have led to lawsuits and increased oversight and scrutiny by state legislators. Partly in reaction to the issues some states have faced in modernizing the administration of SNAP, the 2008 Farm Bill requires that the USDA identify standards for major changes in operations and collect data that can be used to identify and correct problems relating to integrity and access, particularly by certain vulnerable households. In its proposed rule to meet this requirement, FNS has identified several measures that could be collected if a state were to implement a major change. For example, for states implementing online applications, FNS’ reporting requirements may include the number of applications submitted, approved, and denied online; the processing time for approved applications; and demographic information on the households using online applications. For states requiring reporting through change centers, requirements may include the number of changes received; the average time to process changes; and\/or the number of changes processed (Federal Register, 2011). Measuring individual aspects of modernization efforts may not ensure that every state initiative will work exactly as planned as soon as it is implemented, but building in automated checks on the system will help state officials to identify issues sooner and may prevent any issues that arise from having substantial negative consequences for the already at-risk population. E. How Are Measures Currently Used for Federal Oversight? For decades, FNS has required states to collect several performance measures. States are expected to process applications within 30 days of receipt of a signed application, and applications identified for expedited processing must be completed within 7 days. The states with the best timeliness rates receive financial rewards. FNS measures the accuracy of benefit issuance and eligibility determination through the SNAP Quality Control data and provides financial rewards to states with the best or most improved accuracy rates. States with accuracy rates that consistently perform worse than other states may be subject to sanctions. FNS also measures program participation among the low-income population through the Program Access Index and financially rewards states with the highest and most improved indices. Introduction Mathematica Policy Research 6 Arizona staff reported that after their implementation of document imaging, they needed to prove they were saving staff time. To do so, they developed a staffing efficiency model to measure the work times on document imaging tasks (as well as those they replaced, such as filing). The state then provided the governor’s office with quarterly reports documenting that the imaging system not only worked effectively but also showed clear cost savings resulting from the staff time savings. FNS has contracted with Mathematica to look at outcome measures in several states. In 2006 2007, we studied Florida’s Automated Community Connection to Economic Self-Sufficiency (ACCESS Florida) and examined how caseload size, administrative costs, client access to benefits, and program accuracy changed with its introduction (Cody et al. 2008). We currently are conducting a study to look at similar measures in Florida and four additional states: Georgia, Massachusetts, Utah, and Washington. However, FNS generally has not required that states meet specific performance standards on individual initiatives, instead leaving it to the states. States conduct measurements at the initiative level to determine if each specific activity is working in the way intended. Once they are satisfied the initiative is performing at an acceptable level, they often stop measuring and focus only on the broader outcome measures. In requiring states to achieve a certain level of performance on the broader outcome measures, FNS has been acting as a monitor\u2014that is, consistently reviewing the measures to be sure states are meeting a certain level and implementing consequences when they do not meet these levels. In conducting an exploratory study such as this one and providing a wealth of information about the range of measures, FNS is acting in a facilitating role. It is providing information to the states with the goal of helping them plan better systems and avoid some of the pitfalls other states have experienced. If FNS chooses to put a set of standards in place for individual initiatives, it will be moving to the monitoring role at the individual initiative level as well. F. How Have States Used Performance Measures? States use performance measurement differently at different stages of implementation. In the early stages, states measure an initiative to determine whether it is working as designed and meeting initial expectations. Later, states review reports to monitor the performance and ensure that acceptable levels of operations are sustained over time. The intensity of the monitoring will vary over time, however. When states are confident that the initiative is working as desired, they generally shift their attention and resources to other issues. If, while monitoring the reports, they identify issues that need to be addressed, their attention will return to the initiative and they generally will review the measures more frequently until these issues are resolved. On the other hand, when individual initiatives need less monitoring, managers\u2014particularly local managers\u2014tend to want more immediate access to results of measures. They become less interested in reports about what has happened in the past and instead want to have reports that tell them the state of activities at that exact moment: instead of wanting to know, for example, the average wait time for callers over the past week or month, they want more real-time data from these measures. For example, they may want data on how many people are waiting on hold at the present moment and how long they have been waiting. If the numbers are too high, the managers can reassign staff. Once the measures return to more desirable levels, those staff can return to their previous activities. Introduction Mathematica Policy Research 7 1. Early Stages of Implementation Emerging from discussions with states, particularly during our site visits, is the fact that early on in the life of an initiative, SNAP state and local managers face pressure to justify the changes being made\u2014whether they be policy changes, such as a waiver of face-to-face interviews; technology changes, such as online applications, a call center, or document imaging; or partnerships with community organizations or businesses. Given that significant resources are needed to implement these changes and that numerous internal and external stakeholders have an interest in the outcome of the initiatives, the pressure to demonstrate success is often quite strong. Much of the pressure derives from the fact that states need to show they actually have achieved their key objective\u2014for example, reducing overall costs, reducing staff workloads, improving service delivery to clients, or some combination of these. Measurement, then, often is the tool by which states can establish proof of concept. In other words, measurement helps answer the question of whether an initiative is doing what it is intended to do. For example, states that have implemented a document imaging system want proof that the scanners can effectively image a document that is readable, that the imaged document can be indexed or coded properly, and that it will be attached to the correct case file. It is also important for the state to show that the centralization of document imaging services actually achieved cost savings by requiring fewer workers to handle the same or increasing levels of imaging. In addition to proving the concept of the initiative, measurement during this early stage of implementation helps states to demonstrate its success, making it easier for funders to continue its funding. It also may make it easier for them to introduce other initiatives in the future. Equally important is that measurement helps states ensure that they are not creating unintended consequences for clients or other processes. 2. Maintenance Stage of Implementation Once proof of concept is confirmed, the focus of states and managers shifts to monitoring the modernization initiatives. During this maintenance stage, managers routinely review static reports\u2014 weekly, monthly, or quarterly\u2014to track performance during that period, as well as compare results over time. The purpose of these reviews is to assess how well the initiative is working and ensure that the results the state has achieved initially can be maintained. Analysis of static reports also can help states establish standards for the future. In this respect, states will review past reports and ask: How did we do last month? Last quarter? Or last year? Then they will ask if they can do better. These standards, however, are usually informal and not rigorously defined or tracked; they act mainly as guides by which managers judge progress. However, managers and staff eventually face time or resource constraints, forcing them to make choices about how frequently and intensely they will continue to monitor the static reports. When most initiatives tend to operate at a satisfactory level, the act of reviewing reports becomes less revealing\u2014and compelling\u2014over time. So, while managers continue to produce the reports, their focus tends to shift to reviewing them only to identify red flags that would cause them concern about the initiative’s operations. Furthermore, the red flags that seem to concern the managers most involved with the initiatives are those that could directly affect quality; for example, the ability to We benchmark against ourselves in terms of what’s possible, where we can push the envelope. \u2014State SNAP Director Introduction Mathematica Policy Research 8 correctly determine eligibility or deliver the right benefit amount, or timeliness, such as the ability to meet the 30-day deadline for eligibility determination. 3. Real Time \/ Daily Measures As static reports become less revealing, the trend is for states to want real-time data. The desire for more frequent data may seem at odds with many states’ views that they do not have the time or resources to review static reports. However, daily and real-time measures provide state staff, particularly local office managers, with information that they can act on immediately. Having information on a daily or real-time basis allows managers to focus on the now, which enables them to track staff workloads most effectively, identify obstacles to effective flow as they occur, and quickly identify potential problems with cases that are most time-sensitive or those considered as highest risk. Most important, having access to this frequency of data enables managers and staff to take immediate action in response to conditions as they currently exist and not how things were last week, last month, or last quarter. In this study, we find that 30 states have at least one performance measure or aggregate data calculated on a real-time or daily basis. Overall, the 30 states average 13 performance measures and aggregate data calculated based on these immediate timeframes. The initiatives that have the greatest number of real-time and daily measures are focused primarily in two areas\u2014of the 381 (non-unique) performance measures and aggregate data calculated at this frequency, 173 are related to the call center (Figure I.1) and another 113 are related to application tracking (Figure I.2). Introduction Mathematica Policy Research 9 Figure I.1. Number of Real-Time and Daily Call Center Measures Calculated by States Note: Of the 45 responding states, 28 had call centers, 19 (shown in blue and green) calculate real- time or daily measures and 9 do not (shown in yellow, along with those who do not have call centers). Of the 9 states, 4 calculate longer-term measures, another 3 states calculate measures but did not report the period of time, and 2 states did not report calculating any measures. Counts are based on reported frequencies. Most states had one or more measures for which they did not report a frequency. Introduction Mathematica Policy Research 10 Figure I.2. Number of Real-Time and Daily Application Tracking Measures Calculated by States Note: Of the 45 responding states, 20 (shown in blue and green) identified application tracking measures that they calculated real-time or daily. Of those 20, 15 states identified 1 to 5 measures, 4 states identified 6 to 10 measures, and 1 state identified more than 16 measures. Counts are based on reported frequencies. Most states had one or more measures for which they did not report a frequency. 4. Focus on Quality and Timeliness States also made clear that their primary concern is to ensure quality and timeliness. Staff at all levels of SNAP agencies indicated that states are not meeting their mission if they do not provide benefits to those who are eligible, get the beneficiaries the right benefit amount, or achieve these goals in a timely fashion. At times, then, states may allow performance to drop for certain operational measures so they can ensure that other, more critical goals are achieved. For example, one state noted that it made the strategic decision not to shift resources to call centers when it noticed that the average answer speed was declining. Instead, the state kept its resources in areas that best ensured benefits were being provided to eligible clients in a timely manner. Introduction Mathematica Policy Research 11 G. How Are Measures Assessed? Until now, we have used the word measure in its broadest sense; that is, as the wide range of data that can be collected to assess an initiative’s or program’s performance. In fact, in performance measurement terminology, there are three key types of measures used to assess performance: data elements, aggregate data, and performance measures. A data element is the lowest level of measure and refers to a field that can track a specific instance of an activity. For instance, if the state tracks whether an application comes to it through an online system or through traditional means (fax, mail), the field in which it would store the information is the data element. The next level is aggregate data, or counts of data elements. Using the same example, the total number of applications that come through the online system and the total that come through traditional means are both aggregate data. Finally, performance measures are calculations using related data elements, such as the percentage of all applications that come to the state through its online system and the percentage that come through traditional means. A performance measure, used in this very strict sense, is normalized in such a way to make it comparable with other measures. For example, the normalization makes it comparable with other measures of the initiative in the same state or with the same measure in another state. Not all measures are equally appropriate for assessing performance. The measures themselves need to be assessed, which we have done according to the following six dimensions: 1. Validity: Does the measure accurately reflect performance toward a given goal? The connection between the measure and the goal needs to be clear. 2. Level of Detail: What are the measure’s unit of analysis and frequency? How often it needs to be measured will depend on its use\u2014measures that help allocate staff need to be available at high levels of frequency. Measures for trend analysis may be reported less often. 3. Reliability and Comparability: Are the data elements accurate, as well as the aggregate data used to construct the measure? A measure may appear to reflect the desired information, but if the means by which the data are captured is subject to error, it may not be a valid measure. The data also need to be collected similarly across states for comparisons to be valid. 4. Efficiency: How much time, effort, and money are needed to construct the measure? A measure that requires manual reporting or substantial post-processing of data elements is not likely to be sustainable. 5. Performance Standards: Is it reasonable to set an expectation for performance? Some measures may be used simply to track trends, but measures used to assess performance typically are paired with an expected result. Performance Measurement Terminology 1. Data Element is a field in a database that stores an instance of an activity or characteristic and is the basis of aggregate data and performance measure calculations. 2. Aggregate Data are counts of data elements, such as the number of observations. 3. Performance Measures are calculations using related pieces of aggregate data to assess an activity and typically are displayed as percentages and averages. 4. Performance Standard includes a standard and a benchmark. A standard is the desired outcome of an activity\u2014for example, answering a call within three minutes, while a benchmark is the desired rate of success\u2014for example, answering at least 80 percent of calls within three minutes. Introduction Mathematica Policy Research 12 6. Performance Initiatives: Is it reasonable to attach incentives or disincentives to the measure? Will staff performance reviews be tied to their ability to meet the standards? Will offices be rewarded if they meet the standard for a given period of time or penalized if they fail to meet it? Answers to these questions may differ across states, even for measures that appear to be the same. A complete understanding of measures, and the ability to compare across states, requires an assessment at this level. H. Study Objectives and Research Questions In this report, we seek to discuss the range of performance measures states have in place for specific initiatives\u2014assessing them according to the dimensions described above and comparing and contrasting their definitions and the level of performance that states meet. We also propose a small set of measures for each initiative that (1) aims not to be overly burdensome for the states to collect, (2) consists of reliable and valid measures, and (3) assists FNS and the states in assessing the likely success of the initiative. We also identify the gaps in this set of measures. In a separate report prepared as part of this study (Keefe et al. 2012), we provide a profile of initiatives and measures by state. Specifically, we address the following objectives and questions: a. Describe Each State’s SNAP Performance Measures and Standards in Detail: Which SNAP modernization performance measures do states have in place? What gaps exist in the performance measures currently used? Which standards are in place to gauge performance? What gaps exist in state performance standards? What is the rationale for the performance standards states use, and has the rationale changed over time? How were the performance measures developed? Have performance measures changed over time? What similarities and differences in performance measures and performance standards exist across states? What alternatives are there for states in measuring performance? b. Provide a Detailed Description of How Performance Measures Are Implemented, What Results Are Observed, and How the Results Are Used in Each State Are states using performance measures that are reliable and valid? How are performance measures operationalized across states? To what extent are different performance measures used? Using these performance measures, what results are observed? Introduction Mathematica Policy Research 13 Does performance vary? If so, on what basis? Are performance data used to gauge performance and make decisions? If so, how? What opinions do stakeholders offer about measures in place and potential additions? What similarities and differences occur across states? c. Assess the Pros and Cons of Applying Alternative Performance Measures and Standards Across States Which criteria should be considered in choosing to implement individual performance measures and standards? How well do the alternative measures and standards meet these criteria? Which measures appear to be more burdensome or require sophisticated technology to collect? Is there a best set of performance measures for program access, payment accuracy, efficiency, and customer service? Do these measures provide added value to the existing SNAP measures? Is there a set of measures and standards that are generally applicable? Under what circumstances? 15 II. METHODS To answer the research questions, we used a two-phased approach. During Phase 1, we interviewed a broad group of stakeholders to obtain their perspective on state performance measurement activities, identified the key modernization initiatives to include in the study, and developed the data collection instruments. During Phase 2, we developed a process to select the respondents and conducted data collection from state and local SNAP offices and organizations partnering with them. In this chapter, we describe the activities that occurred in these two phases. A. Phase 1: Preliminary Assessment and Instrument Development 1. Introduction Because of the exploratory nature of the study, Phase 1 involved developing insights on the performance measures and standards states use to assess their modernization initiatives. Based on our interviews with a broad group of SNAP stakeholders, we worked with FNS to refine the data collection objectives, identified the key modernization initiatives to include in the study, and developed a framework for assessing modernization initiatives for use in developing the data collection instruments. This section describes our approach to these activities. 2. Who We Interviewed We interviewed stakeholders with a wide perspective on state modernization and measurement activities, as well as those with insight on a specific aspect. They fell into four main groups: FNS, national human services organizations, state SNAP agencies, and community-based organizations and business partners. Questions for these stakeholder groups targeted their particular experience and perspective in the following areas: National and Regional FNS Office Staff. We interviewed staff members in the national office and all seven regional offices. We obtained an understanding of FNS’s goals for the project and learned more about state modernization initiatives and efforts to measure them. National Human Service Organizations. We solicited input from organizations researching and\/or working with states as they modernize. We identified their concerns about state modernization initiatives and what they would like to see from the study. They also discussed their insights on state and partner measurement activities. State SNAP Offices. We had in-depth discussions with state agencies that focused on the scope, functionalities, and measurement activities (including standards and benchmarks) of their modernization initiatives. Community-Based Organizations and Business Partners. We discussed these organizations’ concerns and knowledge of state-based measurement activities, state measurement of partnering activities, and their internal measurement capabilities. In addition to interviewing these groups, we reviewed related documents to supplement the feedback we received. These documents included call center statistics, online application utilization rates, applicant surveys, and SNAP reports, which we obtained from stakeholders and state websites. Methods Mathematica Policy Research 16 They provided an account of modernization initiatives and descriptions of specific performance measurement activities, including calculation methodologies and performance standards and incentives for selected measures. Also, we reviewed preliminary data collected for the study Enhancing Supplemental Nutrition Assistance Program (SNAP) Certification: SNAP Modernization Efforts (Rowe et al. 2010) to extract information that states reported on these activities. 3. What We Learned Using the findings from the interviews and document analysis, we drafted a preliminary inventory of state performance measurement activities. This inventory helped us assess the extent of measurement activities for specific modernization initiatives. Four themes emerged about how states were developing their approach to measuring performance: Official SNAP measures motivate existing performance measurement activities. States emphasize measurement of four SNAP measures: the payment error rate (the percentage of approved households with benefits calculated incorrectly), the negative error rate (the percentage of applying households incorrectly denied benefits), timeliness (the percentage of applications processed within 30 days), and the program access index (the percentage of the low-income population receiving benefits). FNS reporting requirements and performance bonuses motivate states to pay close attention to their performance in these areas. Some states developed corollary performance measures to identify the sources of errors or delays to improve their performance. States collect performance measures and data but may not use the information to their full advantage. Although computer systems retain a large volume of information, not all states are using the data to assess the initiatives’ overall impact on SNAP administration. State financial and technical resource and time constraints hinder the development of measures to assess progress in meeting program goals. Even in states with performance measures, we uncovered few performance standards against which they are benchmarking their efforts. Technology-related modernization initiatives facilitate current and future performance measurement. Building performance measures and tracking results is easier for technology-related modernization initiatives, as compared to others. We saw more performance measures with technology-related initiatives because systems are designed to track key indicators to ensure their functioning. Some states used this to their advantage to develop program performance measures. Other states simply may need to determine how to extract the data according to concrete measure specifications and leverage them for their performance needs. States make limited use of performance measures to assess modernization initiatives’ impacts on program access. For some stakeholders, it is vital to track where applicants are falling out of the system and their characteristics. (For example, do elderly individuals abandon online applications more often or at earlier stages than applicants of other ages?) States appear to be doing this only on a very limited basis. States have been tracking application and recertification approvals and denials and identifying errors, but preliminary research did not find that they tracked individuals’ characteristics, such as age, along with the approval, denial, and error rates. Methods Mathematica Policy Research 17 4. A Limited Set of Initiatives to Examine The objective of the study is to build a comprehensive inventory of the performance measurement activities used by states and local offices to monitor the performance of SNAP modernization initiatives. The comprehensive inventory includes specific measures and associated performance standards to enhance FNS’ understanding of how agencies collect, analyze, and use performance data to monitor their modernization activities. While our preliminary inventory included all initiatives FNS includes in its definition of modernization, we identified a subset of eight initiatives, in consultation with FNS, based on their wide use across states and their importance to FNS. To request states to submit detailed accounts of performance measures used for all initiatives would have been too difficult and time-consuming for agencies with limited staff resources, especially those already overwhelmed with growing caseloads. Limiting the number of initiatives was important to balance the time required and give staff the opportunity to provide well thought-out and thorough answers to questions. Below we list the eight selected initiatives, which fall into three categories\u2014technology, partnering with other organizations, and policy simplification: Technology – Call centers, including computer phone system, change center, interviews, and application processing – Online systems, including the online screening tool, online application, and online account – Document imaging – Kiosks used for application access and submission Partnering with other organizations for application assistance Policy simplification – The waiver of face-to-face interviews – A shortened interview process – The capacity for clients to complete expedited applications online The 12 initiatives included in the preliminary but not the comprehensive assessment includes the following: 1. Automatic case management 2. Data matching 3. Telecommuting 4. Staff e-training 5. Allowing clients 60 instead of 30 days to complete applications 6. Improving computer system characteristics using upgrades and modifications 7. Expanding use of electronic benefit transfer Methods Mathematica Policy Research 18 8. Outreach activities 9. Office changes 10. Organizational changes: transferring and sharing functions 11. Organizational changes: staff functions 12. Training Also, due to the emphasis that states place on existing FNS measures and the centrality of application processing to SNAP operations, we developed a performance measures module focused on the receipt and processing of applications and recertifications. This module, referred to as application tracking, expanded on the FNS-required performance measures regarding application approvals and denials by including performance measures related to the source of the application or recertification. 5. Data Collection Instruments The data collection plan involved a combined methods approach, using surveys, phone interviews, and in-person interviews and observations to collect detailed information for the inventory and provide states with an interactive response approach to describe and demonstrate their use of performance measures. We designed an electronic survey for state and local SNAP agencies that provided us with information about the presence and functionality of initiatives in the state and basic information about the performance measures the state collects. We then followed up with phone interviews to clarify survey responses and obtain more details about their select performance measures. For partner organizations, we developed an interview protocol focusing on their partnering activities with the state or local SNAP agencies as well as their performance measurement activities. After completing surveys and interviews in the state, we visited 10 states to conduct in-person interviews and observe their use of the performance measures. Below, we describe the content of the survey, interview, and site visit data collection instruments. Electronic Survey. We designed an electronic survey to collect details on the performance measures used for the modernization initiatives. For each of the eight initiatives, we provided states and local agencies with a list of known performance measures, aggregate data, and data elements collected during Phase 1. Respondents identified which of the eight modernization initiatives they had in place and their characteristics. Then, based on their responses, the survey revealed only the portion of the performance measures list relevant to their initiative. To augment this list and allow us to identify other areas of measurement, we gave states the opportunity to include additional performance measures. This option allowed us to build a more comprehensive inventory. For each performance measure and aggregate data, respondents answered questions on (1) how (and how frequently) they calculate the measure or element, (2) their purpose for collecting it and how it relates to FNS’s SNAP program goals, (3) their assessment of the accuracy and uses of the measure or element, and (4) the performance incentives or standards the respondent’s organization uses with respect to the measure. The survey also included a section for all respondents about application tracking, separated into six key steps in the applications process: (1) receipt, (2) processing and case characteristics, (3) approvals, (4) denials, (5) accuracy, and (6) processing time. Methods Mathematica Policy Research 19 Respondents also had the opportunity to add additional application tracking measures not on the preliminary list. Interview with State and Local SNAP Agencies. The interview protocol includes a series of questions to better understand the characteristics of the modernization initiative and to follow-up on the survey responses to obtain further detail about the performance measure calculations for a subset of the performance measures being calculated. These questions assess the six areas of the measure framework we developed: (1) validity, (2) level of detail, (3) reliability and comparability, (4) efficiency, (5) performance standards, and (6) performance incentives. Interview with Partner Organizations. The partner-only interview protocol focused on both the measures partners report to the state or local office and internal measures they use to assess their own performance. The interview questions on the measures assessed the same six qualities as those in the SNAP agency interview. Site Visits with Select States. The site visits to states included interviews with staff and tours of the state SNAP office, a local SNAP office, and a partner office to better understand the usage of performance measures for daily SNAP operations. We developed a semi-structured protocol for the interviews, which was tailored to each state after reviewing findings from the surveys and interviews. We also requested that the states send us 12 months’ worth of actual performance data for the performance measures they reported tracking, if possible. B. Phase 2: Respondent Selection and Data Collection With the data collection instruments developed, we identified and selected respondents and conducted the data collection. In this section, we describe the process, the modes of data collection by respondent type, the response rate, and our approach to following through with respondents. 1. Selecting Respondents Collecting information from every state and the District of Columbia provides FNS with the broadest view of the use of SNAP performance measures and standards across the country. All state SNAP agencies received an invitation to participate in the study and, in most states, a subset of their county and local SNAP offices and partner organizations also received one. Responses collected from the sample of county and local SNAP offices supplemented the state-level picture by determining the extent to which local-level offices are conducting additional performance measurement activity, as well as giving a front-line perspective on performance data generation and usage. All state, county, and local SNAP office respondents also received a request to submit 12 months’ worth of performance data for each measure their office calculates. The process we used to identify individuals to contact varied by the respondent type. State SNAP Offices Selection. All states SNAP offices and the District of Columbia received a survey, request for interview, and request for performance data. Local Office Selection. Initially, we planned to survey, interview, and request performance data from 150 local offices from the 19 states identified as having a relatively high number of the key initiatives and that together covered the range of Methods Mathematica Policy Research 20 initiative functionalities and regions of the country. We also factored in the performance on the outcome measures for which FNS provides financial incentives (timeliness, accuracy, and participation levels). In county-administered\/state-supervised states, we requested from the state contact information for 13 local offices and in state- administered states we requested it for 7. (When necessary, we randomly sampled from among the county and local offices.) During data collection, we often found little variation within state-administered states regarding their measurement of modernization initiatives because they primarily followed direction from the state. As a result, to enhance the data collection effort, we modified our approach and requested contact information for local offices in all states as opposed to only the subset of states. The target numbers changed to five for county-administered states and two for state-administered states. When contacting the states to obtain the local office contact information, we suggested offices based on a random selection (to simplify the process for states), but states also had the opportunity to identify specific local offices they wanted as part of this study. Switching the approach to local office selection allowed us to obtain wider understanding of the modernization initiatives and their measurement activities within more states. It also increased the opportunity for the discussions of modernization initiatives not utilized at the state-level. Partner Organization Selection. We targeted three types of partner organizations\u2014 national businesses, local businesses, and community-based organizations. During the interview with the office, we obtained further details on these partnering activities and requested the contact information. In selecting the partners for the interviews, we focused on those that were doing a wide variety of measurement activities or had unique partnering arrangements. Site Visit State Selection. We selected states for site visits in two stages. Before beginning data collection, seven states stood out in terms of their implementation of modernization initiatives and likelihood of having developed a wide range of performance measures. During data collection, after conducting numerous interviews and becoming more informed about the types and prevalence of performance measures across states, we identified three additional states. Deferring the decision for the remaining states allowed us to identify modernization activities that states had undertaken but were not known by the regions or well publicized. It also allowed us to uncover innovative approaches to performance measurement and particularly interesting lessons learned by states. Table II.1 lists the three respondent types and the modes through which each responded to the study. Table II.1. Modes of Data Collection for Each Respondent Type Respondent Type Electronic Survey Telephone Interview Performance Data Site Visit State SNAP Offices X X X X County\/Local SNAP Offices X X X X Partner Organizations X X X Methods Mathematica Policy Research 21 2. Responses Received During the nine-month data collection period from August 2010 to April 2011, we received feedback from 217 respondents representing 44 of the states and the District of Columbia. Figure II.1 depicts the states that contributed to the study and the total number of respondents within each. There were 15 states where respondents included the state and local SNAP offices and partner organizations. Of these 15, 10 also participated in the site visits. There were 19 states where both the state and local offices responded, and 11 where the state was the only respondent. Figure II.1. Number and Combination of Respondents, by State 3. Level of Response The number of solicited respondents differed from our planned approach. We sought participation from all 50 states and the District of Columbia as originally planned. However, 6 did not have time available to contribute, resulting in a total of 45 states participating in the study. We accommodated the schedules of respondents to make the request as minimally burdensome as possible, which included both allowing states to complete an interview only and extending the data collection period. Despite our best attempts, some state and local office respondents still were unable to participate due to their administrative responsibilities. When we changed our approach to selecting local office respondents during data collection, most state offices we interviewed provided the contact information upon request. We did not request this information from the six state offices that chose not to respond to the initial survey. We Methods Mathematica Policy Research 22 completed a survey, telephone interview, or both a survey and interview with 117 offices, out of 137 originally contacted. Although we collected contact information for numerous partner organizations, it would not have been fruitful to interview all of them. Many partner activities involved conducting SNAP outreach and providing application assistance. After conducting a substantial number of interviews with these types of partners and identifying the typical measurement activities, we chose to continue interviewing organizations with activities beyond outreach and application assistance. We sent participation requests to 71 and interviewed 55. 23 III. SUMMARY OF PERFORMANCE MEASURES ACROSS INITIATIVES Across the eight modernization initiatives included in our comprehensive inventory, states reported using 89 different performance measures to assess their activities. Of these 89 measures, states reported developing performance standards for 39 and, of those, assigning benchmarks and incentives to gauge the success of the standard to only a few. A wide variety of factors influence states as they develop performance measures for their modernization initiatives, including FNS’s goals for modernization and operational considerations, such as staffing and understanding utilization of the initiative. Barriers to additional measure development include insufficient resources (financial and time) and the technical incapacity of some legacy eligibility systems to track and store data related to modernization initiatives. In this chapter, we describe the purpose of measures according to FNS’s modernization goals; key state performance measures for each initiative; general gaps in performance measurement; states’ use of standards, benchmarks, and incentives; and how the measures relate to the SNAP lifecycle of application processing. Appendix A lists all measures and aggregate data that were identified by the states for each of the initiatives. A. Purpose of Measures by Modernization Goal Using FNS’s four goals for modernization\u2014program access, accuracy and integrity, efficiency, and customer service\u2014we asked states and local offices to identify the modernization goal that a measure addressed. We then categorized the measures within each goal according to the measure’s use. For example, among measures that states and local offices identified as assessing progress toward program access, we found that they fit into several categories based on their use: monitoring SNAP as a whole, monitoring the initiative, improving access to SNAP by maximizing staff time and program costs or having effective procedures, ensuring correct use of procedures, and ensuring that customers can make effective use of the initiative. In Table III.1, we show the number of different measures associated with each goal, further categorized by the measure’s use (note that some measures may be included under more than one goal). The number of measures for each of these themes yields insight into the focus areas state and local offices choose for their measurement activities. The list below highlights the themes for each goal: Program Access. Monitoring overall program access (rather than just at the initiative level) and improving it through efficiencies and correct use of procedures, have a large number of measures, but customer use of the initiative has fewer. Accuracy and Integrity. Aside from tracking overall accuracy, determining staff training needs and ensuring that staff use the correct procedures are the primary areas in which state and local offices focus their measurement activities. Efficiency. Tracking initiative usage, improving customer experience, reducing staff time, and following correct procedures are the areas in which the most measurement occurs. Performance Measures Summary Mathematica Policy Research 24 Table III.1. Types of Measures States Are Collecting to Assess Progress Toward Modernization Goals FNS’s Modernization Goal Purpose of Measure Number of Measuresa Number of States Collecting Measures Program Access Monitor combined result of initiatives to increase program access 43 27 Improve program access through maximizing staff time and agency cost efficiencies 37 20 Ensure correct procedures are being used by staff 34 17 Monitor use of initiative 30 12 Improve program access through establishing effective procedures 21 14 Determine if customers are able to use the initiative 5 3 Accuracy and Integrity Track accuracy related to the program as a whole as opposed to a specific initiative 84 32 Assess general accuracy used in eligibility determinations as part of FNS quality reviews 24 20 Determine staff training needs and ensure correct procedures are used by staff 17 16 Reduce overall errors or improve overall accuracy 12 10 Identify overall error rate or trends, not related to a specific initiative 9 11 Identify error rate or trends for a specific initiative 5 8 Reduce errors or improve accuracy for a specific initiative 3 2 Efficiency Track usage or system load 73 26 Reduce staff time and agency costs 47 21 Improve customer experience 46 20 Determine staffing needs and ensure that correct procedures are used by staff 30 19 Assess initiative-specific accuracy used in eligibility determinations as part of FNS quality reviews 20 12 Customer Service Assess general customer service 61 26 Increase efficiency\/improve customer experience for a specific initiative 52 24 Identify barriers to program access for specific initiative 40 12 Determine staffing needs and ensure that correct procedures are used by staff 32 17 Assess general accuracy used in eligibility determinations as part of FNS quality reviews 27 13 Eliminate or reduce barriers to program access for customers 8 6 Note: Totals are based on responses from the 45 states that participated in the study. Each respondent indicated its own purpose for collecting the measure, and those can differ across the state and local offices within a state. The measures in this table include those reported by both offices, so a measure may be captured more than once. a Numbers in this column include performance measures reported for each initiative and application tracking. Performance Measures Summary Mathematica Policy Research 25 Customer Service. Tracking initiative-specific efficiency and customer experience, as well as determining staffing needs and following correct procedures, are the areas in which states concentrate their measurement activities. In Table III.2, we present similar information about the use and purpose of the measure, but focusing on each initiative. In the 28 states with a call center, for example, we find 34 different measures that states and local offices are using to identify training needs, assess operations efficiency, and assess customer service. Within the online systems, they are tracking how much a system is being used and making comparisons of its use with other ways of submitting applications. In Table III.2, we also show how states and local offices viewed these measures in relation to FNS’s modernization goals. In most cases, respondents matched a measure with more than one goal. They selected the goal of Accuracy and Integrity least often but identified the remaining three goals about equally. The respondents cited goals for all of the measures used for these initiatives: online account access, waiver of face-to-face interviews, and expedited online applications. For call centers, they cited all goals equally except for Accuracy and Integrity, which they cited least frequently. The state with the single kiosk measure used it for program access B. Key Performance Measures and Gaps, by Initiative State and local offices are using a large number of measures to oversee their modernization initiatives. Many of these measures exist within most states implementing an initiative. Some state and local offices supplement the common measures with others less common to enhance the set of measures used. States also described gaps in their measurement activities and their desire to calculate additional performance measures and aggregate data. Using measures is practical for states\u2014they need the information the measures provide to make operational decisions. For example, they may ask these questions: How many staff do they need for a given shift? Are potential applicants using the available options to learn about and apply to the program? They also may ask the following questions but not necessarily be in a position to use the measures to address them: How is this initiative impacting the broader picture of program access? How does the work under one initiative impact the operations of another for improving application processing? In this section, we describe the types and numbers of measures that states use for each initiative, their purposes in using them, and gaps in measurement activities. 1. Call Center The call center is a point of access whereby individuals call to obtain information or some other service by listening to a recording, talking to a person, and\/or interacting with a computer. There are three functionalities that states might choose to implement: call center, computer phone system, or change reporting. The general call center has agents available to answer caller questions. 26 Table III.2. Number of Measures States Are Collecting to Assess Progress Toward Modernization Goals Initiative Number of States with Initiative Number of Different Measures of Initiative Reported Use of Measure Number of Measures Collected for Each Goal Program Access Accuracy and Integrity Efficiency Customer Service Call Center 28 34 Identify staffing needs Identify need to train staff Assess efficiencies and inefficiencies of operations Assess customer service 19 12 18 20 Online System 33 27 Demonstrate program access and increased participation Track utilization of online application feature, including making comparisons with paper applications 20 17 16 19 Document Imaging 24 8 Track system operations Manage staff workload 5 3 7 4 Kiosks 14 1 None reported 1 0 0 0 Partnering 35 3 Track approvals Track partners providing assistance Track partner assistance provided\u2014from monitoring to evaluating effectiveness and reimbursing them for assistance provided Enhance program access 3 2 2 3 Waiver of Face-to-Face Interview 40 12 Show effectiveness of interview option Determine accuracy, including making comparison with face-to-face interviews Comply with federal waiver requirement for performance measure calculations 8 8 8 9 Shortened Interviews 12 2 None reported 0 0 0 0 Expedited Applications 20 2 Increase program access Track approvals 2 2 2 2 Note: Totals are based on responses from the 45 states that participated in the study. Performance Measures Summary Mathematica Policy Research 27 Throughout this report, we describe the performance measures associated with the overall call center as applying to incoming calls. The computer phone system facilitates clients interacting with a computer through an integrated voice recognition system or similar technology to get answers to questions, access account information, or report account changes. It also may route callers to a call center agent. States can choose to implement a computer phone system with the abilities to complete all transactions electronically (and not transfer the caller to an agent), transfer the caller to an agent, and\/or accept change reporting, allowing the caller to report changes to his\/her case. Last, the call center might have the ability to accept change reporting from the SNAP client. In Table III.3, we list the most frequently calculated performance measures for the call center initiative. States report that they use these measures to identify staffing and training needs and assess the efficiency of operations. For example, call center supervisors rely on the information these measures provide to identify when they need to add staff to answer calls due to high call volume. If high call volume is not an issue, supervisors might want to assess whether their call center agents have adequate skills to handle the calls and provide training where appropriate. Because states tend to emphasize their immediate staffing needs at daily and hourly frequencies, they did not report looking at the measures over the longer term. Also, the results of these measures fluctuate at peak times in the week and month, so averaging them across longer time periods hides the fluctuation. Table III.3. Top Performance Measures: Call Center Number of States Implementing Initiative Top Measures Number of States with Measure 28 Percent Calls Abandoned Average Calls Handled Average Hold Time Average Call Duration Percent of Total Calls Handled Average Waiting (to Speak to Agent) Average Answer Speed Average Queue Time Average Staff on Phones Average Time Prior to Abandonment 22 22 21 21 18 17 17 16 15 15 Note: Some states reported collecting multiple related measures that they defined in the same way (such as Average Hold Time, Average Queue Time, and Average Answer Speed). Using their definitions and details about the functionality of the call center, we retained only one measure out of those defined the same way, choosing the measure name most consistent with the way we and other states were defining it. Counts are based on responses from the 45 states that participated in the study. 2. Online System An online system is a website an individual may use to submit information. Like the call center, the online system has three basic functionalities from which states can choose to implement. These are the screening tool, the online application, and online accounts. The state either creates the screening tool (internally or under contract) or provides a link through which a client can enter household information and assess eligibility for SNAP benefits. Through the online application functionality, the client can access, complete, and submit the application online, either with or without an electronic signature, depending on the state-selected method. Finally, online accounts Performance Measures Summary Mathematica Policy Research 28 provide individuals with the option to check account history or benefit status or submit change reporting. In Table III.4, we show the most frequently calculated performance measures for the online system initiative. Table III.4. Top Performance Measures: Online System Number of States Implementing Initiative Top Measures Number of States with Measure 33 Percentage of Applications Received Average Number Submitted per Month Average Number of Screenings Completed: Online Average Number of Screenings Started: Online Average Number Started per Month 15 12 9 8 8 Note: Counts are based on responses from the 45 states that participated in the study. State and local offices report using these measures to monitor system utilization. Also, these measures allow them to determine whether individuals are electing to use the online venue to access the program, with an increase in this number demonstrating the online system’s usefulness for increasing program access. However, the measures focus on general utilization; most states do not have additional measures to identify the characteristics of individuals electing to use the system features. For example, the measures do not track the use of individuals over 65 in comparison to those under 65 and cannot demonstrate whether the initiative is increasing program access for a particular demographic. State and local offices also reported a gap in their measurement activities related to tracking the proportion of applications received online versus in paper format. Some states would like to compare the percentage of applications submitted online and on paper, including approval and denial rates, but the previously mentioned system limitations do not make this possible for many of them. 3. Document Imaging A state implementing document imaging is operating a system or a process in which it receives documents in either paper or electronic form, scans the paper documents, and stores all documents electronically for future access. State might store these images in an independent directory not linked to another system or as part of an electronic case record. In Table III.5, we list the most frequently calculated performance measures and aggregate data for the document imaging initiative. The most frequently calculated performance measures are the percentage of documents scanned and the average processing time. Nine states collect these measures, which indicates that only a third of states with this initiative have performance measures (as opposed to aggregate data). As a result, we also chose to include in the table the aggregate data calculated for this initiative. States reported that they track the numbers of documents scanned, received, and processed, and the completion of tasks associated with processing the document. Even though some states track these activities, they do not have a performance measure to assess how using the information stored in the document imaging system affects the accuracy of the application decision or processing time. Performance Measures Summary Mathematica Policy Research 29 Table III.5. Top Performance Measures: Document Imaging Number of States Implementing Initiative Top Measures Number of States with Measure 24 Performance Measures Percentage of Documents Scanned Average Processing Time Aggregate Data Number of Documents Scanned Number of Documents Received Number of Documents Processed Number of Tasks Complete 9 9 18 12 11 11 Note: Counts are based on responses from the 45 states that participated in the study. 4. Kiosks Kiosks are computer terminals in a community location or SNAP office at which an individual may access a SNAP application or change reporting function. Kiosks are public or semi-public terminals, so clients’ personal computers are not included in this definition. We list in Table III.6 the most frequently calculated kiosk performance measures. About a quarter of states are implementing this initiative, and one state calculates the applications submitted after the applicant completes a screening. A few states are tracking the measures listed in this table for changes, account access, and screenings completed. Table III.6. Top Performance Measures: Kiosks Number of States Implementing Initiative Top Measures Number of States with Measure 14 Performance Measure Percentage of Screenings Resulting in Application Submission: Kiosk Aggregate Data Number of Uses to Submit Changes Number of Uses to Access Online Account Number of Screenings Completed: Kiosk 1 3 2 1 Note: Counts are based on responses from the 45 states that participated in the study. 5. Partnering (for Application Assistance) Partnering for application assistance occurs when organizations or agencies other than the SNAP office provide access or services to SNAP clients under a formal or informal agreement with that office. The agreement may be in the form of a contract, memorandum of understanding, or another type of request. In Table III.7, we list the most frequently calculated performance measures and aggregate data for the partnering initiative. States did not report tracking performance measures for their partnering activities other than the percentage of partners providing application assistance. They use this measure to assess their outreach activities with target populations. The aggregate data that states most often track include their number of partners, the number providing application assistance, the number accepting applications, and the number of applications received per partner. They generally use this Performance Measures Summary Mathematica Policy Research 30 information to reimburse partners as a part of their statewide outreach plan. In some states, the application approval status also affects the amount of payments to the partner. Because so few states closely monitor partner activities, they cannot use measures to demonstrate broader impact on program access. States also do not use the aggregate data and application determinations to track program access for underserved or specific populations. Table III.7. Top Performance Measures: Partnering Number of States Implementing Initiative Top Measures Number of States with Measure 35 Performance Measure Percentage of Partners Providing Application Aggregate Data Number of Partners Number Providing Application Assistance Number Accepting Applications Number of Applications Received per Partner 10 20 18 17 15 Note: Counts are based on responses from the 45 states that participated in the study. 6. Waiver of Face-to-Face Interview The waiver of the face-to-face interview is a formal approval received from the FNS national office to perform interviews by telephone. We list the most frequently calculated performance measures and aggregate data for the waiver of this interview in Table III.8. Those state and local offices reporting that they calculate this set of measures said they do so to assess the accuracy of the eligibility determination as part of FNS’s quality control process. Calculating the error rate of the face-to-face interview often is challenging for them. The information for the interview type (either telephone or face-to-face) typically is stored in the case file notes and so does not lend itself to creating automated reports for calculating these measures efficiently. Table III.8. Top Performance Measures: Waiver of Face-to-Face Interview Number of States Implementing Initiative Top Measures Number of States with Measure 40 Performance Measures Error Rate for Cases with Telephone Interviews Error Rate for Cases with Face to Face Interviews Percentage of Interviews Conducted by Telephone Percentage of Interviews Conducted Face-to-Face Aggregate Data Total Number of Interviews Number of Redetermination Interviews 16 14 13 13 20 19 Note: Counts are based on responses from the 45 states that participated in the study. 7. Shortened Interviews Some clients are eligible to participate in SNAP eligibility interviews, consisting of a shorter list of questions than the regular interview. In Table III.9, we list the most frequently calculated Performance Measures Summary Mathematica Policy Research 31 Table III.9. Top Performance Measures: Shortened Interviews Number of States Implementing Initiative Top Measures Number of States with Measure 12 Performance Measures Percentage of Clients Electing Shortened Interview Percentage of Total Interviews that Are Shortened Aggregate Data Number of Shortened Interviews Number of Clients with Shortened Interviews 2 2 3 1 Note: Counts are based on responses from the 45 states that participated in the study. performance measures for the shortened interviews. The measures for this initiative are not used broadly across states. 8. Expedited Applications\u2014Submitted Online Some states provide the capacity for clients to complete expedited applications online. In Table III.10, we list the most frequently calculated performance measures for this initiative. Table III.10. Top Performance Measures: Expedited Applications\u2014Submitted Online Number of States Implementing Initiative Top Measures Number of States with Measure 20 Performance Measures Percentage of Applications Approved: Online All Percentage of Applications Approved: Online Expedited Aggregate Data Number of Expedited Applications Received Online Number of Applications Approved: Online Expedited Number of Applications Approved: Online All 10 9 9 9 5 Note: Counts are based on responses from the 45 states that participated in the study. C. General Gaps in Performance Measurement Within each initiative above, we address very specific gaps in performance measurement\u2014areas in which states do not use specific measures broadly or lack measures for specific activities occurring in an initiative. In addition, some gaps we have identified are not as tied to individual initiatives. Some states expressed a desire for more aggregate data. Also, states provided feedback on the utility and feasibility of additional national-level performance measures for reporting on SNAP activities. Apart from desiring additional performance data, states reported that they would like to enhance their ability to analyze the data within their state systems and the time, financial, and technological barriers to developing additional performance measures. Aggregate Data Gaps. Some state and local offices expressed a desire for more aggregate data so they can learn more about client experience with SNAP and better understand the client decision making process around completing the application and recertification processes. For example, some states and partners expressed the desire to better understand SNAP churning (when households exit Performance Measures Summary Mathematica Policy Research 32 SNAP for a short time, such as one to four months, and then reapply for benefits). Identifying patterns in disenrollment and re-enrollment would facilitate program efforts to reduce the occurrence of churning; some aggregate data the states identified for this purpose include information on the source of the application (mail, fax, online, partner), interview completion mode (phone or face-to-face), reasons for failing to complete the interview, and benefit denial reasons. Respondents also discussed tracking such data as the number of telephone interview attempts, customer experience with interviews, and the time lapse between interview completion and the date the application was processed. Partner Organizations Desire Application Determination Information. Partner organizations would like to have information on the application determination for individuals they assist with applying. To assist them in targeting their outreach efforts, partners would like to know the amount and approval date for approved applications and the denial reason and denial date for denied applications. Some states are unable to provide partners with this information, while those that are able to do so vary in the information they provide. Some SNAP offices do not track partner-assisted applications. This means that to determine the impact of outreach efforts on increasing SNAP enrollment, partners must call applicants to ask if they received approval notification and their monthly benefit amount. This process is time consuming, and it is challenging to reach all of those they assisted. When an applicant grants the partner written permission to discuss his or her case with the SNAP office, partners can call the office to obtain this information. Some SNAP offices have been able to develop a formal process through which they track all partner-assisted applications. This process most frequently occurs when the partner receives reimbursement from the state for each application with which the partner provides assistance. In this process, the SNAP office provides the determination information to the partner at a specified interval so that the latter can assess its outreach efforts. However, SNAP offices vary in the specific information they provide. Some states only indicate the final determination of either approval or denial. Others provide the amount of benefits approved; few states provide details on the reason for denial. National-Level Performance Measures. Some states would like to have additional national- level performance measures to gauge their activities against other states (as opposed to establishing new national standards for these activities), while others feel the current national-level measures are sufficient and that they do not have the resources to generate additional reports. Those states that would like to gauge their activities against others expressed a desire to see the proportion of applications processed online and on paper as well as the average number of days to process applications (rather than the percentage processed within 30 days, as required by FNS). Initiative-specific national measures they requested also include the percentage of applications received online versus in paper to assess utilization patterns. For the call center, respondents were most interested in two measures: average wait time and average call duration. Seeing these results nationally would allow states to look at their operations compared to the states they identify as their peers\u2014taking into consideration the characteristics of the state\u2014to assess their own performance and the reasonableness of their results. Those states that did not see the need for additional nationally reported performance measures said that the current FNS national measures are sufficient. Also, the current reporting effort consists of many measures, and increasing the number would overwhelm them with reporting activities. Performance Measures Summary Mathematica Policy Research 33 Moreover, states reported that it would be challenging to use additional measures to make comparisons across states because of variations in initiative scope, capacity, operations, and the nature of state demographics. Analytical Capabilities Desired. Apart from more performance measures and aggregate data, state and local offices reported a desire for increased analytical capabilities for the measures they already are calculating. Some aggregate reports are available only at the state level, so staff do not have the ability to drill down to the local or individual levels. Increasing such analytical capabilities would help with training and monitoring staff. For example, with such data, a supervisor could try to identify performance challenges according to application disposition and processing timeframe; alternatively, a program manager could view the state’s overall application processing time to ascertain how this number varies by county and local office to identify variations in processing time, which in turn could help identify outliers. Also, program managers could look at application denials according to the applicants’ demographics to assess in more detail subpopulations’ access to the program. To a lesser extent, some state and local officials said that reports, once produced, often are not available or distributed in a timely way across the state. For existing performance measures, states would like their analytical abilities enhanced in two primary areas. The first includes automating manual reports. Some measures require time-consuming manual calculations at the local office; such data are more susceptible to error than automated data because of this manual process. Also, the reporting burden resulting from aggregating and submitting these data to the state or other reporting entity increases because the paper documents must be totaled manually. Barriers to Measure Development. Despite the desire to have more measures, track more aggregate data, and increase the number of measures reported nationally, states encounter barriers\u2014 time, financial, and technological constraints\u2014to identifying, specifying, and developing new measures from data they currently collect. Respondents reported that it is challenging to devote time to assessing critically their performance measurement gaps so as to identify how calculating an additional measure could help them operate their modernization initiatives. Respondents also reported that technical challenges pose a barrier to increasing their measure development, specifically when tracking applications received and disposition by application source. In Table III.11, we describe the current state capabilities. More states have the capacity to track applications that were submitted either online or directly to the SNAP office (mailed, faxed, or hand-delivered) than applications submitted through a kiosk, call center, or community partner. However, the ability to track approval and denial determinations is more limited than the ability to track the application source. More states are able to track the source because mode- or initiative-specific tracking efforts exist for this information. However, source information is less often stored in the eligibility system, where it could be used to assess the mode in relationship to the final determination. A local office director reported an interest in seeing improvements in the technology and computer system that they currently have to generate better data and better use the data that is already captured in the system. For example, for the waiver of the face to face interview, this is only recorded in documentation. Reports cannot be run on this initiative, even though the data is recorded. Performance Measures Summary Mathematica Policy Research 34 Table III.11. Number of States Able to Track Application Status, by Application Source Submitted Application Complete Application Approved Denied All Applicants 34 34 33 Source of Application Delivered to office (mail, fax, in person) 26 20 17 Submitted through kiosk 3 1 1 Submitted online 25 14 10 Submitted to call center 4 4 3 Submitted by community partner 11 11 9 Note: This table includes both state-reported performance measures and aggregate data to generate the total numbers in these categories. Counts are based on responses from the 45 states that participated in the study. Each respondent indicated its own purpose for collecting the measure, and those can differ across the state and local offices within a state. The measures in this table include those reported by both offices, so a measure may be captured more than once. Specifically, respondents said that their eligibility systems do not store this information in a discrete field(s). Although it is possible to access individual records within the eligibility system to see the final disposition, without storing the application source these calculations cannot be made in the same system. For example, many states reported that their online systems operate separately from their eligibility systems, and these cannot be linked easily or at all for reporting purposes. Even though one system records the online applications received, it does not store the application disposition data as well. Each system retains a portion of the information to perform this calculation, but these are not connected for reporting purposes. Likewise, for partner-submitted applications, few states have created a new eligibility system field or other discrete method to identify them. Some states with document imaging systems reported that it is possible to look at the scanned image of an online application or partner-assisted application to determine from which of these sources the application was processed. D. Use of Standards, Benchmarks, and Incentives Performance measurement typically includes the use of standards, benchmarks, and incentives for the selected measures. Standards and benchmarks establish the criteria against which performance is measured. A standard is the desired outcome of an activity\u2014for example, answering a call within three minutes\u2014while a benchmark is the desired rate of success\u2014for example, answering at least 80 percent of calls within three minutes. An incentive is a motivator for performance and can encompass rewards and public recognition of the highest performers and\/or some combination of sanctions and technical assistance to the lowest performers. If no standards exist, there is no way of knowing if an activity is meeting its goals. States’ use of standards, benchmarks, and incentives varies across their modernization initiatives, measurement purposes, and initiative operations. We found five initiatives with standards, benchmarks, and incentives; one initiative with standards and benchmarks, but no incentives; and two initiatives without any of these. Call centers and online systems each have three functionalities; states vary in their application of standards, benchmarks, and incentives to these initiatives. In Performance Measures Summary Mathematica Policy Research 35 Table III.12, we present the number of measures with standards or benchmarks, by initiative, and summarize how states use them. Table III.12. Use of Standards, Benchmarks, and Incentives Initiative (Number of Measures Using Standards or Benchmarks) Use of Standards, Benchmarks, and Incentives Call Center (20) Standards: Evaluate performance of individual call center agent or a group of agents Incoming Calls Benchmarks: Assess volume of calls through specifying waiting and call duration times Incentives: Determine high or low staff performance review; penalize vendor poor performance (when contracting with a vendor) Standards: Track incoming call handling Phone System Benchmarks: Resolve maximum number of calls within phone system or via agent Incentives: Not reporteda Standards: Track processing of all changes and their processing time Change Reporting Benchmarks: Not provided Incentive: Determine high staff performance review rating Online System (3) Standards: Track portion of applications received online Online Application Benchmarks: Increase usage of application option Incentives: Not used Not used Online Screening and Online Account Access Document Imaging (6) Standards: Assess quantity of documents scanned, by document type, within specified timeframe Benchmarks: Scan all documents within a certain timeframe Incentives: Satisfaction of achieving mandate; penalize vendor poor performance (when contracting with a vendor) Kiosks (0) Not used Partnering\u2014Application Assistance (2) Standards: Monitor partners providing application assistance Benchmarks: Ensure that all partners provide assistance Incentives: Pay partners for approved applications for which they provided assistance Waiver of Face-to-Face Interviews (6) Standards: Monitor the proportion of interviews conducted face-to- face and telephonically to track utilization Benchmarks: Ensure that the percentage of telephone interviews does not exceed waiver requirement Incentives: Not reporteda Shortened Interviews (0) Not used Expedited Applications (2) Standards: Process expedited applications Benchmarks: Process expedited applications within 7-day timeframe required by FNS Incentives: Not reporteda a Not reported indicates that the respondent identified the use of an incentive but did not describe the specific incentive utilized. Performance Measures Summary Mathematica Policy Research 36 Use of Standards. States vary in their use of performance standards. We found more performance standards for many of the unique performance measures associated with the call center, document imaging, waiver of the face-to-face interview, expedited applications, and partnering initiatives. States do not use standards for the performance measures associated with certain online system functionalities (online screening and account access), kiosks, and shortened interviews. Use of Benchmarks and Incentives. States use benchmarks to monitor SNAP office operations, such as supervising staff activities and initiative operations and tracking partner activities, including vendors contracted to operate an initiative and community-based organizations providing application assistance. For the online application, some states reported wanting to increase the use of this application mode and identify the proportion of the total applications received by each mode. Call center supervisors evaluate agent activities and track incoming call handling and processing changes, as well as the computer phone system’s handling of calls. For the waiver of face-to-face interviews, staff monitor the proportion of interviews they conduct on the phone to limit them to the 50 percent balance required for the FNS waiver of face-to-face interviews. As to online expedited applications, standards focus on state staff approving applications within the seven-day timeframe mandated by FNS. States use both positive and negative incentives. For the call center functionality incoming calls and document imaging, vendors contracted to operate the call center on the state’s behalf are held accountable to benchmarks established in the contracts and penalized for poor performance, which is a negative incentive. Similarly, call center agents working for a state-operated initiative have benchmarks to ensure that the calls they handle meet the intended goal. This incentive is both positive and negative because the achievement of or failure to meet the benchmark affects their performance review rating. For some state staff working in document imaging centers, the incentive may not be quantified in a performance review but rather by the general satisfaction of achieving the state mandate. Partner organizations receive positive incentives for achieving their benchmarks in the form of payment for the assistance they provide. E. Relationship of Measures to SNAP Lifecycle In attempting to meet FNS’s standards or additional state requirements, such as lowering the budget, states may pursue one or more changes to their program’s administration. Generally speaking, these initiatives are tied to one or more steps in the application, maintenance, and recertification process\u2014that is, the SNAP lifecycle. This lifecycle has seven general stages: (1) learn about SNAP, (2) acquire application, (3) application, (4) interview, (5) eligibility determination, (6) case maintenance, and (7) recertification. We mapped the modernization initiative and application tracking measures provided on the survey onto one or two of the stages. We then analyzed how respondents reported their measurement activity and purpose for measurement in each stage. We list the number of measures collected for each stage of the lifecycle in Table III.13. Within each stage, we categorize the purpose that states cited for collecting these measures and present the number of different measures calculated for the particular stage and purpose, the number of states with the measures, and the number of measures with a standard. States calculate most of the measures in each stage of the lifecycle, with most measures found in the application stage. They also calculate many measures within the eligibility determination, case Performance Measures Summary Mathematica Policy Research 37 Table III.13. Measures and Reported Purpose by Stages of the Lifecycle Stage in Lifecycle Purpose of Measurea Number of Respondents Stating a Purpose for Collection Number of Different Measures Calculated Number of States with Measures Number Having Standards Learn about SNAP 6 4 6 0 Monitor use 4 3 4 0 Monitor customer service 1 3 1 0 Maximize program access 4 2 4 0 Determine applicant interest 1 1 1 0 Acquire Application 0 0 0 0 Application 61 52 32 26 Monitor system use 34 44 24 14 Minimize client time required 26 37 22 12 Maximize program access 21 32 18 10 Distribute staff load 29 24 19 9 Monitor accuracy and integrity 11 13 9 4 Monitor partners 13 6 8 2 Interview 38 8 22 5 Increase efficiency 5 3 5 0 Monitor accuracy and integrity 24 6 17 4 Maximize program access 22 8 11 2 Monitor system use 6 6 6 0 Decrease staff workload 4 5 4 0 Monitor customer service 5 2 5 0 Eligibility Determination 66 24 40 18 Monitor accuracy and integrity 48 21 37 17 Monitor system use 42 21 28 13 Maximize program access 14 18 13 12 Monitor customer service 16 17 15 15 Monitor staff load and training 17 14 15 8 Monitor partners 6 2 6 1 Case Maintenance 56 22 33 15 Monitor program management 24 21 18 15 Minimize client time required and create efficient process 18 18 16 10 Monitor accuracy and integrity 35 15 30 3 Monitor staff load and training 24 14 15 7 Recertification 33 18 27 8 Maximize program access 12 12 12 6 Monitor accuracy and integrity 13 12 13 5 Monitor system load 9 10 9 1 Minimize client time required and create efficient process 13 8 13 5 Monitor program management 16 7 14 5 Monitor staff load and training 12 7 9 3 Note: Each respondent indicated its own purpose for collecting the measure, and these can differ across the state and local offices within a state. For example, of the 94 measures collected on the survey (excluding measures respondents wrote-in), we identified 56 (not shown) as being associated with the application stage (individual measures were categorized into up to two stages). For those 56 measures, 61 respondents provided a purpose for collecting at least one of the measures. After placing the various purposes reported into the categories listed, we found that, for 34 of the measures, one or more respondents collected the measure to monitor the use of the application system; for 26, one or more respondents collected the measure to help work toward minimizing client time required to apply. Totals are based on responses from the 45 states that participated in the study. Performance Measures Summary Mathematica Policy Research 38 maintenance, and recertification stages. The high concentration of measures in these four stages demonstrates that states focus their measurement on program administration as opposed to how individuals learn about SNAP or acquire the application; no state currently tracks the latter. Although states calculate many measures within the lifecycle stages, a smaller percentage of measures for application and recertification have associated standards than for some of the other measures (excluding the acquire application stage, for which no state calculates a measure). Across the lifecycle stages, states focus most frequently on system use and staff load. They also calculate many measures in these stages to maximize program access and monitor accuracy and integrity. In the application and eligibility determination stages, states calculate measures related to monitoring partners the least frequently; this is important to note, because FNS does not calculate national measures or require the reporting of information related to these partnering activities. However, FNS does work with states on an individual basis to monitor some partner reimbursement for application assistance activities. 39 IV. LIMITATIONS TO USE OF MEASURES AND STANDARDS FOR CROSS-STATE COMPARISONS While many states are collecting performance measures for initiatives that have the same name, these measures cannot be compared across states. Several factors related to modernization and performance measures affect cross-state comparisons. In this chapter, we focus on two aspects: contextual limitations (related to the context and environment in which the initiatives operate and the functions of the initiatives) and data limitations (related to the ability to calculate accurate, valid, and reliable measures). Because this is an exploratory study, we face limitations in assessing how well the particular performance measures identified by the states can be compared. We asked states for basic information about all performance measures they collect related to their initiatives, but to avoid placing an excessive burden on them, we limited the number of measures for which we requested clarifying information and further details. This means that we often received incomplete information about performance measures. Also, six states did not have time available to participate, so this study does not include an analysis of their performance measurement activities. A. Contextual Limitations State and local offices operate and implement modernization initiatives in different contexts, which influence their choice of performance measures and standards and the observable outcomes, as well as complicating cross-state comparisons. Common contextual limitations include: Initiative Scope and Features. Initiatives differ in their size and breadth, especially whether they serve the entire state or just a region, and what services are provided through them. System Limitations. Eligibility and reporting systems are limited in their ability to capture and report measures of interest to each state. Terminology Differences. Variations in measure wording and definitions across states mean that states are measuring different performance aspects. Varying Measurement Motivations. Motivations for using measures and standards also vary across states, with some choosing measures to manage staff workloads and others setting standards to comply with legislative or other mandates. 1. Initiative Scope and Features An initiative’s scope and its features or functionalities all play a role in determining performance measures and their associated outcomes, and likewise affect the performance standards guiding an initiative. Factors that can influence the measures, and examples of them, include the following: Geographic Reach. It may be misleading to compare two states’ initiatives if one serves an entire state and the other serves only a local area. For instance, states that run statewide call centers may set different standards for average number of calls handled than those running local centers. Limitations to Use of Measures and Standards Mathematica Policy Research 40 Complexity of Initiative Design. Some call centers share a phone line that rings at multiple desks, while others use software to queue and assign calls to agents. These features impact how many people can be served, regardless of the number of staff available. Outcomes for measures such as average answer speed also can differ. Complexity of Technology Supporting Initiative. For document imaging, some states put barcodes on the documents they provide to households so that when the documents are returned, they can be scanned and linked automatically to a case file. Documents clearly will be processed more quickly in these states than in those in which staff manually attach documents to cases. Also, both types of states will achieve much faster imaging speeds than those in which workers send the documents through scanners manually. Presence of Complementary Initiatives. Whether a state or locality has a waiver of face-to-face interviews and a call center to handle those interviews impacts data that feed into call center performance measures. The average wait time, answer speed, and call duration measures may achieve different levels in states that conduct phone interviews than in areas with a more limited number of services available by phone. Functions of the Initiative. States with online screening tools may have fewer applications submitted per month online if, before submitting their applications, people realize they are not eligible or would receive only a small benefit amount. Similarly, the ability to submit applications online and to print applications and mail them in could affect average number of applications submitted through the mail and online, or could increase the average number of duplicate applications if people choose to apply via both modes. Community Characteristics. States with a waiver of the face-to-face interview may encourage telephone interviews for all applicants, but individuals for whom transportation to the SNAP office is difficult may take advantage of this option more often than those with easy access. States with few urban centers may have higher take-up rates. 2. System Limitations The efficacy of comparing performance measures across states also is affected by the eligibility and reporting systems the states use. Respondents reported three main limitations to these systems: reporting capacity, system reliability, and system capability. Each affects states differently and may present more problems for some states than others. Some states’ systems do not have the capacity to report aggregate data or generate reports that can be manipulated by users. They often combine measures across programs, such as Medicaid, SNAP, and cash welfare. Alternatively, data may be aggregated to the state level and not be available by county or local office. Other measures may be available at a point in time but not be aggregated systematically to show trends or longer-term performance; for example, hourly or real-time reports captured throughout the day may need to be aggregated manually to days and then weeks or months. Reports also may capture only one piece of a broader measure. For instance, in some cases, a state can track only clients approved for SNAP benefits through their food bank partners, not those that applied and were rejected. Limitations to Use of Measures and Standards Mathematica Policy Research 41 State systems may be unreliable or face periodic outages. States reported times when they have had to enter data manually that should have been entered automatically, introducing the possibility of errors in the data. The systems also may be limited in what they can collect or measure. Some states rely on systems that have been in place for 30 years and are limited as to the amount of data they can record or process; these may not be updated easily. Other states with newer systems might find that their use of multiple assistance programs makes it more complicated to make system modifications. Some call centers have queues with limited capacity, which may have been set at what seemed an appropriate level when the initiative was launched but may no longer be adequate, given record- breaking participation levels. Callers trying to access such a system while the queue is full will not be counted in measures monitoring call center usage. 3. Terminology Differences A lack of standardized terminology limits cross-state performance comparisons across all initiatives. For example, many states measure call duration as the total number of minutes a worker spends on the phone with the caller, but another state includes the time a worker spends working on the case after the caller hangs up. Another state, which has multiple call centers working independently, calculates the measure of average calls handled differently across call centers. In yet another variation, one state excludes hang-ups and wrong numbers from its calculation altogether, while other states count them as handled calls. 4. Varying Measurement Motivations Although some states may calculate the same performance measures and standards, their motivations for doing so may vary. A performance measure designed to meet a legislative mandate in one state could be calculated differently from one used primarily for staffing purposes, and any performance standards associated with those measures likely will differ as well. For example, a state interested in ensuring that all calls are being answered may calculate the percentage of total calls handled by counting all calls that come into the call center, while a state interested in monitoring individual staff performance may include only calls that are not abandoned. This is not problematic in and of itself but can complicate comparisons of multiple states because motivations can affect how a state office prioritizes and uses its measures. B. Data Limitations While contextual limitations impact cross-state analysis of measures, the data used to calculate the measures also pose limits to performance evaluations and can jeopardize interpretations of individual measures. Performance measures are subject to three main types of data limitations: (1) accuracy, meaning whether data are entered and measures are calculated correctly; (2) validity, or whether states actually measure what they intend to measure; and (3) reliability, defined as the consistency in the calculation of performance measures over time and across sites. Limitations to the accuracy, validity, or reliability of measures hinder cross-state comparability and performance analyses. Limitations to Use of Measures and Standards Mathematica Policy Research 42 1. Accuracy When states use data to calculate performance measures, how they enter the data impacts the quality of the measure. Procedures to ensure that data are entered correctly also influence the measures’ dependability. Data entered automatically likely have fewer errors and less subjectivity (assuming the systems automatically generating those data were programmed correctly). Overall, state and local offices often reported using automatic data entry performance measures, leading to a higher degree of accuracy. However, this more often tends to be true at the state than the local level, where staff typically have less control over generating reports. 2. Validity Validity refers to whether or not states actually are measuring what they intend to measure, or if their measure includes data they do not intend to capture. Numerous respondents indicated in the survey that they had validity concerns about their measures, although few specified the reason for their concern, and it would have been too burdensome to request this information from each respondent. 3. Reliability Data reliability addresses the consistency of how measures are calculated over time and across localities or states. To be reliable, a measure must not have changed recently\u2014either in how it is defined or calculated. Such changes are not inherently bad but may indicate that a measure was not calculated optimally prior to the change, which thus impairs comparability over time. Changes to software or computer systems or purchases of new systems can bring about new calculations. Other changes occur because of alterations in business practices\u2014for example, staff reductions have led to some states reducing their call center queue capacity. In addition, a measure’s definition must reflect the same concept, regardless of where it is being collected. For example, a call center in one state or locality might assume that people hanging up within 15 seconds have dialed the wrong number and so not count them as having abandoned the call. Another center in a different state using a different definition would not count a call unless it stays in the system for at least 60 seconds. For this reason, the measure must allow for a concise definition and calculation that can be compared across sites. 43 V. OPPORTUNITIES AND CHALLENGES FOR UNIFORMLY MEASURING INITIATIVES FNS currently focuses its assessment of states’ SNAP activities through measures of timeliness, accuracy, and participation. It requires states to meet a set percentage of applications processed within a 30-day period and maintain high levels of accuracy in eligibility and benefit determination in relation to other states. It also measures the percentage of low-income individuals participating in SNAP in each state. In 2003, FNS began to provide financial incentives to states that achieved high performance in these areas relative to other states. For those not achieving such high performance, FNS has imposed penalties and\/or facilitated communication across the states so that those performing at a high level can share best practices and lessons learned with those performing at a lower level. The timeliness, accuracy, and participation measures assess total program outcomes\u2014they do not focus directly on how well individual initiatives are working in aid of or against the outcomes. The mix of initiatives, as well as how a particular state uses that mix, allows for different levels of performance on individual initiatives and similar results on overall program outcomes. For example, a state could achieve high levels of accuracy and also process nearly all its applications within 30 days, yet not image documents until a week after receipt or keep clients on hold for 30 minutes or more. In this situation, individual initiatives are working at subpar levels, but the efficiencies and services offered through other initiatives and processes compensate. Poor performance in some initiatives may not show up in the overarching measures unless the performance becomes so deficient that other activities no longer are able to offset them. On the other hand, a state’s modernization initiatives all could be working at a desirable level of performance yet not result in desirable program outcome measures. In other words, a state could have online applications that could be seamlessly entered into the eligibility system, all client documents imaged and attached to the case within a day, and all calls to the call center answered within a few minutes, yet not meet the 30-day timeliness requirement. In this chapter, we explore defining a compact set of measures for each initiative that states or FNS could use to assess their initiatives individually. Such a set of measures could be useful under a variety of circumstances: States not meeting timeliness, accuracy, and participation measures. If states are having difficulty in meeting the overarching outcome measures, they or FNS may want to assess their performance at the initiative level. States implementing a major change to their SNAP administration. When states first launch one or more initiatives, having a set of measures with which to assess the performance could help them identify and resolve problems quickly. In fact, FNS recently proposed a rule related to major changes in SNAP procedures that requires states to provide descriptive information regarding the major change together with an analysis of its projected impacts on program operations (Federal Register, v76, no 85, Tuesday, May 3, 2011, p. 24823). As part of the rule change, FNS may require the states to measure and report on specific aspects of a change. Once an initiative’s success is demonstrated, measurement could cease. Uniformly Measuring Initiatives Mathematica Policy Research 44 General requirements. FNS could require that all states implementing an initiative meet a basic set of standards, so that performance across states could be compared more easily. However, imposing a set of standards for each initiative has both limitations and dangers. If FNS were to set requirements for specific initiatives, states would do their best to comply so as not to face penalties. If financial incentives were included for high performance, states would work hard to receive the incentive. Also, they might improve performance on the individual initiatives in general, yet not improve access, accuracy, efficiency, or customer service. Examples of limitations and dangers inherent in setting standards for each initiative include the following: State administrative procedures are very different, resulting in substantially diverse performance expectations for individual initiatives across states. For example, the functionality of some initiatives varies across states: call centers simply may direct a person to the appropriate agent or may be able to immediately address a variety of customer questions. Even though a set of measures could be applied differently based on functionality, a state able to address a wide range of questions through automated responses will achieve a different level of performance on measures such as the average call duration or average waiting time to speak to an agent than an automated response unit providing limited information, such as office hours or assistance in locating the nearest office. States have different combinations of initiatives, which can affect performance on individual initiatives. A state that has both a call center and an online system to handle account inquiries, for example, will have different call center and online account usage than a state that has only one of these. This can affect wait time for the call center and lower the likelihood of abandoned calls or callers receiving a busy signal. States have limited additional resources for modernization. If states are required to build measures into their systems retroactively, they may need to expend resources to improve existing procedures or develop new initiatives. Building measurement into a new initiative could be expensive enough to prevent a state from implementing it. For example, some states conduct much of their document imaging manually and across numerous offices. For these states, tracking the number of documents scanned and the timeframe in which they were imaged could require a process that takes more time than the imaging itself. Requiring a certain level of performance could reduce creative approaches and access. In partnering with community-based organizations, for example, requiring a certain level of performance from partners could be counterproductive. If partners must meet a certain quota for application submissions, they may submit applications for individuals they really do not expect to be eligible. This would help them to reach their quota but create unnecessary work for the state’s eligibility workers. If partners must maintain a certain approval percentage for their submitted applications, they may limit the applications they submit only to those individuals of whose eligibility they are confident, denying assistance to those on the border of eligibility. Similarly, a state may try to increase the percentage of calls handled by the automated response unit and reduce the percentage needing to be transferred to agents. Navigating through the menus needed to achieve these results could become so difficult that calls stop coming into the system. Uniformly Measuring Initiatives Mathematica Policy Research 45 Several initiatives are not limited to SNAP. Call centers and online systems, for example, usually are set up for all programs administered by an agency; these generally include both medical assistance and cash welfare, but vary by state. The use of call centers and online systems, expected duration of calls, and number of available staff all will vary for these different programs. Based on our discussions with FNS and state, local, and community partner staff, we make the following recommendations for FNS and states when launching new initiatives: Performance of initiatives should be measured\u2014states acknowledge that where measurement occurs, performance improves. Performance measurement should be built into the initiative during the design stage. States should have an expected level of performance (standard) for each measure. The performance measures and standards should be tied to improving overall program outcomes. However, because of the limitations described above and in Chapter IV, and because the exploratory nature of this study did not allow us to develop a detailed understanding of every measure in every state, we are not recommending a set of performance measures for each initiative for use across all states in all situations. Instead, we present below a selected set of measures that appears useful within a broad range of state operational contexts. States implementing new initiatives could review this list for suggested measures. Although we hope most of these measures will be appropriate for the states, some may not fit within the context of a state, while others may be a close fit, but need slight modification. States will need to make these decisions based on their specific goals for implementation, the functionality of their initiative, other initiatives already in place, the unique features the initiative may entail, and the integration of SNAP with other assistance programs. Before presenting the measures for each initiative, we describe the criteria we used in their selection. After discussing the measures, we discuss several advantages and limitations. A. Criteria for Inclusion in the Proposed Set of Measures We considered a variety of criteria in selecting measures. We sought measures that are both in line with FNS’ four goals for modernization and can be measured reliably and without placing excessive burdens on states. In general, we preferred to choose measures that at least some states, and ideally many, already have in place. In some cases, however, we suggest measures that states are not collecting but appear feasible based on what we know about their existing measures. 1. Addressing Goals for Modernization As discussed in Chapter I, FNS has delineated four goals for modernization: accuracy and integrity, efficiency, customer service, and program access. Many modernization initiatives achieve several of the goals. All measures in our set, either by themselves or in combination with others, also address at least one of these goals. Uniformly Measuring Initiatives Mathematica Policy Research 46 Ideally, the set of measures for each initiative will fully address the goals achieved by the modernization initiative. In other words, if an initiative relates to three of FNS’s goals, then the full set of measures calculated also should cover each of these three goals. 2. Feasibility as a Reporting Requirement The measures must be applicable across a range of functionalities and not impose an unnecessary burden on states. To ensure this, we assessed each measure based on the following dimensions: Validity and Reliability: Is a measure an accurate reflection of performance toward a goal? Are the required data available? Can it be measured similarly across states? Can it be measured consistently over time? Efficiency: Is it reasonable to ask the state to collect the information? How much effort will it take? Is it an automated process or something the state would track manually? To identify our suggested set of measures, for each initiative we reviewed the full set of measures the states used, their frequency, the purposes states had for collecting them, states’ sense of their accuracy, and the ability of all states to collect them. B. Proposed Set of Measures For each initiative, we provide the name of the measure, along with a very specific definition, a rationale, and its limitations. At times, we suggest measures not currently collected by states, although these usually are minor modifications to measures they already do collect. For this reason, where possible, we indicate the number of states currently collecting the measure. The definition we provide for the measure may not match exactly what some states are reporting\u2014because of slight differences in how states define measures, we chose the measure we thought most appropriate for a broad range of states. 1. Application Tracking In Chapter III, we talked about measures in relation to the SNAP life cycle. One key aspect is the application process. Without making it through the process, applicants will not become participants. It thus is important to have some sense of the stage at which applicants may fall out of the process. Because this process crosses many of the initiatives, we look separately at broad measures for tracking applications. For application tracking, we sought measures to address the following questions: By what mode do applicants submit applications (mail, in-person, online, through a partner)? Do approval and denial rates vary by the application mode? How do the application modes and approval and denial rates vary by characteristics of the applicants? In Table V.1, we present our suggested measures for application tracking. States that collect these measures often do so on a monthly basis. Uniformly Measuring Initiatives Mathematica Policy Research 47 The set of ten measures simply addresses the first two questions above\u2014helping states understand the usage of the various application portals and identify any consistent differences in approval and denial rates. We recognize that the third question, related to understanding whether certain individuals have a more difficult time with the process than others, is vital to improving the application process. However, even some of the states with the most advanced computer systems and reporting ability cannot track applications at this level of detail. At this time, to suggest that states collect such measures would impose a substantial burden on them. Table V.1. Recommended Application Tracking Measures Measure Name Definition Rationale Comments Number of States Collecting Measurea Percent of Applications Submitted by Source Number of applications submitted through each mode (mail\/fax, in-person, online\/kiosk, call center, through a partner), divided by total number of applications. (Maximum of five measures, depending on activities in a state.) Helps the state to track the means by which applicants are applying. Even states that can track whether an application was submitted online often cannot track when it was delivered by mail separately from those dropped off at the office. These two may need to be combined. Mail: 10 Fax: 7 Paper: 14 Online: 13 Kiosk: 2 Call center: 4 Community Partner: 8 Percent of Applications from Each Source that Are Approved Number of applications submitted through each mode and subsequently approved, divided by number of applications submitted by mode. (Maximum of five measures, depending on activities in a state.) Helps the state to understand whether one mode of application tends to be used by more people who are not eligible or do not complete the process. Excludes from both the numerator and denominator the number of applications in process during the period data are collected. Mail: 9 Fax: 0 Paper: 14 Online: 8 Kiosk: 1 Call Center: 3 Community Partner: 10 a Counts are based on responses from the 45 states that participated in the study. 2. Call Centers Because the functionality of the 28 states’ call centers in the 45 responding states varies substantially, the data available and the appropriateness of each measure also vary. Some call centers are as basic as a shared phone line that rings at multiple desks. Other initiatives use software that queues calls and assigns them to agents not currently on a call. Some are solely voice response systems, while others pair automated voice responses with a function that routes harder questions to available agents. The technology supporting the phone system, purpose of the call center, and division of functions between call agents and an automated system all influence decisions about what to measure and, when measures are the same, what outcomes the state can strive to achieve. For the proposed set of measures, we sought to answer the following three questions: How busy is the call center? Is the call center able to handle the volume? How long do clients wait for service? Uniformly Measuring Initiatives Mathematica Policy Research 48 Table V.2 provides our suggested set of measures. States may prefer to collect these at least on a daily basis. Table V.2. Recommended Call Center Measures Measure Name Definition Rationale Comments Number of States Collecting Measurea Calls Received per Active Case Number of calls, divided by number of active cases. Demonstrates volume of calls in relation to caseload size, making the measure more comparable across states and regions. Include in the numerator and denominator only the cases in the area covered by the call center. States with online account access may have much lower call volume; states with more functionality may have higher call volume. High numbers of calls from nonparticipants would not be reflected in the denominator. Average Number of Calls per Case: 2 Number of calls received: 19 Average Number of Calls Handled per Agent Number of calls handled by agents, divided by number of agents. Demonstrates load on individual agent Need to set standards based on functionality of systems\u2014those with automated systems likely will see lengthier calls and thus have a lower expectation for number of calls possible per agent. Number of Calls Answered\/ Handled: 21 Percent of Calls Abandoned Number of calls abandoned before reaching automated system or agent. Potentially demonstrates dissatisfaction of callers with amount of time required to wait for service. If recorded information is played during hold time, it may answer the caller’s question and not truly be an abandoned call, even though it would be recorded as such. 22 Average Time to Answer by Automated Response Unit Total time callers wait to access automated system, divided by total number of calls into system. Potentially demonstrates dissatisfaction of callers with amount of time required to wait for service. Will not apply to call centers with no automated response unit. Average Answer Speed (not restricted to calls being answered by automated unit): 17 Average Waiting Time to Speak to Agent Total time callers wait before talking to an agent, divided by total number of calls into system. Potentially demonstrates dissatisfaction of callers with amount of time required to wait for service. Should not include time spent being directed by automated unit; clock should not stop if an agent answers only to place caller on hold. 17 a Counts are based on responses from the 45 states that participated in the study. Uniformly Measuring Initiatives Mathematica Policy Research 49 Most states with call centers collect either these measures or some closely resembling them. For example, many states collect information about the number of calls they receive. By dividing the number of calls received by the size of the caseload covered by the call center, states will have a sense of whether the call volume is larger than expected based on its functionality. A call center that routes calls to agents would not expect to see more calls per case than caseworkers expected to receive for their cases before the call center was implemented. A higher average load could indicate that inefficiencies in the design of the call center exist. States also often collect the number of calls queued. By relating this figure to the number of agents taking calls, the state can learn whether the number of calls queued is too high and possibly make more staff available to take calls. Most states with call centers also collect the remaining three measures defined here. These provide an indirect measure of client satisfaction\u2014whether clients hang up often and whether client wait times are seen as appropriate for the types of calls the call center generally handles. Wait times are especially important to callers using cell phones, who must pay for each minute they remain on the phone even if they are receiving no services. Not all states’ call centers should be subject to these measures. For example, a call center used solely to make outgoing calls for interviews would not experience callers abandoning their calls or waiting in queues. Also, these measures do not address the accuracy of the data provided by the call center, nor do they directly address customer satisfaction. The automated responses may provide wrong or outdated information (related to office hours or office closings, for example). Agents may act rudely or not answer the callers’ questions properly. Measuring this level of customer service generally requires developing a survey for callers, which can be cost prohibitive for the states. In effect, the states would have to develop a new initiative to measure the success of the call center. 3. Online Systems Online systems are common among states, with 33 of the 45 responding states providing the opportunity for applicants at least to fill out an application online, and some allowing them to submit the application over the Internet. Additional capabilities include online accounts that individuals can use to check the status of their application or access benefit information and systems that can accept changes to the income, expenses, or household composition reported by participants. For the proposed set of measures, we sought to answer the following questions: How often is the online system used for submitting applications? For accessing account information? For submitting changes? Are online application submissions of comparable quality to paper submissions? Are customers using the online system satisfied with access and ease of use? Table V.3 provides our suggested set of measures. It may be sufficient to calculate these measures on a monthly basis, but some may prefer a shorter time period. About half of the states with online applications track the percentage of all applications submitted online. Few states track measures associated with their online accounts. For online Uniformly Measuring Initiatives Mathematica Policy Research 50 Table V.3. Recommended Online System Measures Measure Name Definition Rationale Comments Number of States Collecting Measurea Percent of Applications Received Online Number of applications submitted online, divided by the total number of applications received. Demonstrates use of online system in relation to other modes of application. States may differ in how they count unsigned applications that must be printed and sent to the applicant for approval because an unsigned application does not constitute a completed application. 15 Percent of SNAP Caseload with an Online Account Number of active SNAP accounts, divided by size of SNAP caseload. Demonstrates use of system by entire caseload. Defining active as accounts associated with ongoing cases. 0 Average Number of Logins per Account Number of logins, divided by number of active accounts. Demonstrates use of system by those who have accounts. Defining active as accounts associated with ongoing cases. Number of log-ins: 4 Average Number of Changes Submitted per Account Number of changes submitted, divided by number of active accounts. Potentially demonstrates that system is difficult to use or not meeting clients’ needs. Defining active as accounts associated with ongoing cases. Number of changes submitted: 8 a Counts are based on responses from the 45 states that participated in the study. accounts, we suggest tracking the usage by looking at the percentage of participating households that have online accounts and the average number of logins the account holders make each month. We also suggest tracking the use of change reporting when that is available through the online system. If only a small percentage of its caseload is establishing accounts, a state may want to review the procedures for creating accounts. If usage remains low once accounts are established, it may be that the system is too difficult to use or does not have capabilities important to the users. As with the call center measures, the suggested measures for online systems do not address accuracy or customer service. Online applications may be more prone to error simply because the ease of submission may allow people to work faster and forget to report some income or expenses. Online systems also may process changes incorrectly or users may be more prone to enter information incorrectly if they do not understand how the system works. Customer service can be tracked through these measures only indirectly. Given the wide general use of online technologies, lack of client use likely indicates dissatisfaction with the system for one or more reasons. Identifying those reasons could require the development of a survey. 4. Document Imaging Document imaging currently is in place in about half of the states (24 of the 45 we interviewed). Nineteen of them did at least some measurement of their system. Several states have large document imaging centers, to which all hard copy documents are sent. Workers load stacks of these papers through large scanners that can scan thousands of documents within seconds. Clerical staff then process the scanned images by identifying the type of document and attaching it to the electronic Uniformly Measuring Initiatives Mathematica Policy Research 51 case file. In other states, clerical staff or case workers scan those documents related to their cases. Some states are moving in the direction of allowing clients to scan their own documents and attach them to their case record. Other systems can automatically process faxes of documents that include a barcode to identify the case to which the document belongs. For the proposed set of measures, we sought to answer the following questions: How quickly are documents imaged? How accurate is the imaging and association with a case file? In Table V.4, we present the proposed set of measures. These measures likely will be useful only for states with document imaging systems or dedicated staff, not for states whose workers image only the documents for their own cases. States with less automated systems would need to track this data manually\u2014a time-consuming effort. Table V.4. Recommended Document Imaging Measures Measure Name Definition Rationale Comments Number of States Collecting Measurea Percent of Documents Scanned within 24 Hours Number of documents successfully scanned within 24 hours of receipt, divided by total number of documents to be scanned received in the same 24-hour period. Demonstrates ability of system to process documents efficiently. All documents should be included\u2014even those that are damaged\u2014to track percent requiring manual scanning. Some states are not attempting to achieve scanning within 24 hours, so a longer timeframe may be preferable for them. 9 Percent of Documents Attached to Case within One Business Day of Being Scanned Number of documents attached to case record by close of business on the business day following imaging of the document, divided by number of documents imaged the previous day. Documents need to be accessible to case workers quickly for the system to work well. They are not accessible until they are attached to the file. Not all documents may have enough information in them to attach to a case immediately. Low attachment rate could reflect errors by the person submitting the document, as well as lack of ability of staff to attach documents to cases. Does not reflect how many documents never get attached to a case. 0 Percent of Orphaned Documents Number of documents scanned exactly four weeks prior but that are still not attached to a case, divided by daily average of scanned documents. Tracks extent to which scanned documents could not be tied to a specific case and thus are unusable for purposes of eligibility determination and\/or recertification. Difficult to set the time period over which to measure\u2014a worker may be able to attach the document days or weeks later if the sender provides information that can assist the worker in identifying the case. 0 a Counts are based on responses from the 45 states that participated in the study. Uniformly Measuring Initiatives Mathematica Policy Research 52 5. Partnering (Application Assistance) Partnering is a very common initiative. States make both formal and informal arrangements with partners and rely on them to conduct a variety of activities, including developing software systems, running call centers, and providing application assistance. In this section, we consider only partners providing application assistance, because the other types of partnerships are usually formal, entailing exhaustive sets of measures, and may be covered by the measures discussed previously for online systems and call centers. Of the 45 responding states, 34 have partners conducting application assistance. In some states, partners are reimbursed for their expenditures or paid by the number of applications submitted, the number of approved applications submitted, or a flat fee. In Table V.5, we present our suggested measures for states partnering with organizations without paid arrangements, under the assumption that those states with paid arrangements already are tracking a sufficient amount of information. For those partners not being paid, we suggest just two measures: the number of partners providing application assistance and the number of applications submitted through the partner. The two are, in fact, examples of aggregate data that can be used to gauge and track partnering activities, rather than performance measures that imply the existence of a desired level of performance. Some states track the number of submissions of online applications that originate at a partner site. This measure is limited because organizations can assist an applicant without using the agency’s dedicated online account number. Table V.5. Recommended Partnering Measures Measure Name Definition Rationale Comments Number of States Collecting Measurea Number of Partners Providing Application Assistance Number of partners who have formally declared they are working with the state, even though there may be no formal agreement on how they conduct this activity or how many people they are trying to serve. Helps the state to track the organizations, eases communication of changes in procedures. Definitions of application assistance may differ across states, making this more of a gauge for internal purposes than for comparison across states. 18 Number of Clients Assisted by Partners Report from partners of number of clients they assisted with the application. Helps the state understand the use of the partners by their residents and how much need there is for providing continued training to partner staff. Not all those receiving assistance ultimately will submit their applications. 9 Note: The two items listed here are both examples of aggregate data, rather than true performance measures. We continue to use the term measure in reference to them for consistency with other initiatives. a Counts are based on responses from the 45 states that participated in the study. Uniformly Measuring Initiatives Mathematica Policy Research 53 6. Waiver of Face-to-Face Interview Almost all states (40 of the 45 we interviewed) provided for a waiver of face-to-face interviews or conducted many interviews over the phone because of hardships faced by the client. Almost two- thirds of them were tracking some information about the waiver’s success, often related to accuracy as stipulated by a given waiver. For the proposed set of measures, we sought to answer the following questions: How prevalent is the use of telephone interviews? At application? At recertification? How accurate is the application or recertification when the interview is conducted by phone, compared to a face-to-face interview? We present our suggested measures in Table V.6, anticipating that most states would collect them monthly. Many states may find it difficult to calculate these measures. Although all case workers indicate whether the interview was conducted by phone or in person in the narrative associated with the case or in a field in an online system, the information is not always retained in the eligibility systems in an easily reportable way. Thus, at present, tracking the percentage of applications and recertifications for which interviews are conducted by phone could require changes to computer systems. Some states began tracking error rates based on the interview type but quickly became satisfied with the accuracy of the phone interview and stopped reviewing the data. Table V.6. Recommended Measures for Waiver of the Face-to-Face Interview Measure Name Definition Rationale Comments Number of States Collecting Measurea Percent of Application Interviews Conducted by Telephone Number of application interviews conducted by telephone, divided by number of application interviews. Demonstrates the use of the waiver at application. Percent of (all) Interviews Conducted by Phone:b 13 Percent of Recertification Interviews Conducted by Telephone Number of recertification interviews conducted by telephone, divided by number of recertification interviews. Demonstrates the use of the waiver at recertification. Percent of (all) Interviews Conducted by Phone:b 13 Error Rate for Cases with Telephone Interviews Number of cases in error that included a telephone interview, divided by number of cases in error. Demonstrates the ability of the eligibility worker and applicant to communicate all necessary information over the phone rather than through an in-person meeting. Error rates generally are calculated through formal quality control reviews, and some states can separate the error rate based on the interview type. For this measure to be accurate and unbiased, it likely would require a change in the QC review process\u2014the sampling would need to account for both types of interviews. 16 a Counts are based on responses from the 45 states that participated in the study. b Measure combines both application and recertification interviews. 55 VI. CONCLUSION A. Summary of Study Findings The primary goal of this study was to identify and describe the range of measures states and local offices are using to measure the performance of their SNAP modernization initiatives. We gathered information from 44 states and the District of Columbia, more than 100 local offices, and more than 50 community and business partners. Because it was the first study of its kind, we aimed to collect basic information about all measures the respondents were collecting and detailed information about select measures. The key findings are as follows: States and local offices are collecting a large number of measures. Across the eight modernization initiatives we examined, states and local offices reported using 89 different performance measures to assess their activities. Standards are less common and benchmarks and incentives are rare. Of these 89 measures, states reported developing performance standards for 39 measures; of those, only a few standards are assigned benchmarks and incentives to gauge their success. So states are setting a desired level of staff achievement for about half of their measures. In a few cases, they also identify a rate or percentage of time within which they can reasonably hope to reach that level; in a few cases, they provide some type of reward if the benchmark is met or a penalty if it is not. Measures are most commonly used for technology-related initiatives. The most common measures are those for call centers and online systems. The technology built into the systems make it easier to collect and report performance measures. Thus, states with more technology-driven initiatives have many more performance measures than states without such initiatives. In most states, staff are driven by federal accuracy and timeliness requirements. States are very focused on meeting FNS requirements for accuracy and timeliness. When necessary, they will allow performance on individual initiatives to fall below ideal levels (such as call center wait time) to focus staff resources on meeting these federal benchmarks. If FNS requires a measure, states will most likely comply. States will divert resources to meet federal requirements. When incentives are available, such as for high accuracy and timeliness rates, they are motivated to outperform other states to receive the incentive. When an initiative is launched, performance measurement can assess whether it is working as designed. In the early stages of implementation, reports can indicate that each piece of the initiative is performing its particular function as planned. Along with providing proof of success to stakeholders, this can help secure funding for continued operations. As initiatives mature, performance measurement allows for monitoring. Once staff are confident that the initiative is working as designed, they can analyze trends and identify potential problems through periodic reviews of reports. However, excessive monitoring can create unnecessary work for staff if they need to perform additional tasks purely for tracking purposes. Conclusion Mathematica Policy Research 56 As a measurement matures, real-time and daily reports help allocate limited staff resources. When initiatives are operating as planned, managers desire improvements in measures that allow them to track operations daily or in real time. Measures calculated at such short intervals can help managers assign staff to the highest priority tasks at each point in time. B. Setting Measurement Requirements: Recommendations and Limitations Measuring the performance of initiatives plays a key role in ensuring their success. When launching an initiative, states already should have considered and built in measures to assess its performance. They should have set some level of expected performance, which should be tied to meeting the overarching goals of providing accurate benefits in a timely fashion. With many states developing and launching broadly similar initiatives, sharing their lessons learned and discussing expectations for performance can help later-adopting states avoid the pitfalls faced by the early adopters. However, each state must operate SNAP within its own unique environment. They differ in the computer capabilities supporting their eligibility systems, assistance programs included in those systems, combinations of modernization initiatives, functionalities of their modernization initiatives, and abilities to calculate measures. Given each state’s unique operational environment, FNS and state staff must be cautious in setting expectations for a modernization initiative based on the performance achieved by other states. In this report, we provide a set of measures that each state could consider in light of their goals, technology, and circumstances for assessing a new initiative’s performance. This set of measures is intended to provide a comprehensive view of the initiative’s performance without being overly burdensome to collect. C. Next Steps In collecting the data for this study, we sought to identify the broad range of measures that states are using in implementing a fairly large set of initiatives. We then sought detailed information on a limited set of measures within each state and local agency. Collecting this level of detail for each measure would have placed an extraordinary burden on most of these organizations. In identifying the measures on which to follow up for additional details, we focused on those measures that states indicated were most important to them. Thus, the measures for which we sought more detail varied accordingly across states and local agencies. Although multiple states and local agencies may have collected a measure with the same name, we do not have the same level of detail on the measures from each of the states collecting it. This approach makes it difficult to identify the precise differences in the definitions and accurately compare results for the measures across states. Differences in results, even in states with the same set of initiatives and functionalities, may be due to the difference in the definition, rather than their performance. Given this basic understanding of the measures that states are collecting for each initiative, follow-up studies could focus on a key set of measures for each. The details could be gathered from each state collecting the measure to identify the differences more precisely and allow for better cross-state comparison (although it still will be limited by the variation in functionalities and accompanying initiatives). 57 REFERENCES Associated Press. U.S. reps to Vilsack: Review Indiana food stamp OK, http:\/\/www.wthr.com\/story\/10972768\/us-reps-to-vilsack-review-indiana-food-stamp- ok?redirected=true (downloaded September 21, 2011). Cody, Scott D., Ren\u00e9e Nogales, and Emily Sama Martin. Modernization of the Food Stamp Program in Florida. Washington, DC: Mathematica Policy Research, February 2008. Johnson, Nicholas, Phil Oliff, and Erica Williams. An Update on State Budget Cuts: At Least 46 States Have Imposed Cuts That Hurt Vulnerable Residents and the Economy. Center on Budget and Policy Priorities, http:\/\/www.cbpp.org\/files\/3-13-08sfp.pdf (downloaded on 5\/13\/11). Keefe, Kathleen, Emily Sama-Miller, Maura Bardos, Laura Castner, Elizabeth Clary, Sarah Wissel, and Larry Vittoriano. Performance Measurement for Supplemental Nutrition Assistance Program Modernization Initiatives: Individual State Findings. Washington, DC: Mathematica Policy Research, July 2012. Rowe, Gretchen, Sam Hall, Carolyn O’Brien, Nancy Pindus, Robin Koralek. Enhancing Supplemental Nutrition Assistance Program (SNAP) Certification: SNAP Modernization Efforts Interim Report Volume 1. The Urban Institute: Washington, DC, April 2010. 59 GLOSSARY Modernization Initiatives Call Centers: A point of access whereby individuals dial a telephone number to obtain information or some other service, by listening to a recording, talking to a person, and\/or interacting with a computer. Within a state, call centers may operate statewide or regionally, and may place calls, receive calls, or both. Each call center may have a different combination of functions, including: answering basic questions about a program or a specific case, change reporting, screening for potential eligibility, and taking applications. Caseworkers at call centers may also process information they receive including faxes or reported changes. Online Systems: An internet site that an individual may access and interact with in order to submit or receive information about their potential eligibility or existing SNAP case (not a website that only provides general information about a program). Online systems may allow users to screen for potential eligibility, submit applications online, or check the status of\/report changes to an existing case. Some online applications integrate directly with state eligibility determination software, while others are separate tools. In some states, partners (see below) have special levels of access to the online system to enable them to serve the applicants they assist. Document Imaging: A system or a process whereby paper documents are scanned and an electronic version is created and stored for future access. Kiosks: A computer terminal located in the community or an office (but not in a home) where an individual may access one or more functions of an online system. Partnering: Formal relationships or services that the SNAP agency solicits from non- governmental groups. The solicitation may be in the form of a contract, memorandum of understanding, or another type of request. Waiver of Face-to-Face Interview: Formal approval received from FNS to perform interviews by telephone rather than in person and application, recertification, or both. Shortened Interview: Use of an abbreviated interview protocol for certain types of households based on their characteristics. Measurement Terminology Data Element: An instance of an activity or characteristic that is the basis of aggregate data and performance measure calculations. Aggregate Data: Counts of data elements, such as the number of observations. Performance Measure: Calculation used to assess an activity and is typically displayed as percentages and averages. Performance Standards: Include both standards and benchmarks. A standard is the desired outcome of an activity\u2014for example answering a call within three minutes, while a benchmark is the desired rate of success\u2014for example answering at least 80 percent of calls within three minutes. APPENDIX A PERFORMANCE MEASURES AND AGGREGATE DATA Performance Measures and Aggregate Data Mathematica Policy Research A.3 Table A.1. Measures and Aggregate Data for Call Centers Number of States with Call Center: 28 Name of Performance Measure or Aggregate Data Number of States with Measure or Aggregate Data Performance Measures Percent Calls Abandoned 22 Average Calls Handled 22 Average Hold Time 21 Average Call Duration 21 Percent of Total Calls Handled 18 Average Waiting Time (to Speak to Agent) 17 Average Answer Speed 17 Average Queue Time 16 Average Staff on Phones 15 Average Time Prior to Abandonment 15 Average After-Call Work Time by Staff 13 Percent of Queued Calls Handled 13 Percent of Calls Completed in the Computer Phone System 12 Percent of Changes Processed 10 Average Time to Process Change 10 Average Time to De-Queue 7 Percent of Calls Requesting Agent 4 Average Number of Calls per Case 2 Percent of Screened Individuals Potentially Eligible: Call Center 1 Percent of Screenings Resulting in Applicant Receiving Benefits: Call Center 1 Percent of Screenings Resulting in Application Submission: Call Center 1 Average Calls by Reason for the Call 1 Average Completion Length 1 Average Customer Satisfaction Rating 1 Average Time to Process Changes and Documents 1 Call Response 1 Percent of Agents Assigned to Phones Available to Take Calls 1 Percent of Calls Busy 1 Percent of Documents Processed within 30 Days 1 Percent of Needs Met through Service Referral or Actions Taken 1 Percent of Voicemail Calls 1 Percentage Answered in Less than Three Minutes 1 Percentage of Calls Within Their Categories of Time Intervals 1 Standard of Promptness for Applications and Recertifications 1 Aggregate Data Number of Calls Abandoned 23 Number of Calls Answered\/Handled 21 Number of Calls (In Flow) 20 Number of Calls Queued 20 Number of Calls Received 19 Number of Agents Assigned to Phones 17 Number of Changes Received 14 Number of Changes Processed 13 Performance Measures and Aggregate Data Mathematica Policy Research Table A.1 (continued) A.4 Number of Calls Receiving Busy Signal\/Unable to Connect with Phone System 12 Number of Hang-ups 10 Number of Calls Transferred to Agent 10 Number of Calls Completed in Computer Phone System 9 Number of Clients Accessing Computer Phone System Data 8 Number of Calls Not Completed in Computer Phone System 6 Number of Applications Processed: Call Center 6 Number of Cases Pending Benefit Decision: Call Center 5 Number of Client Calls Returned 4 Maximum Time to Abandon 1 Maximum Time to Answer 1 Number of Calls Answered in English (In Service Area) 1 Number of Calls Answered in English (Outside Service Area) 1 Number of Calls Answered in English (Total) 1 Number of Calls Offered 1 Number of Clients Reporting Case Change 1 Number of Clients Reporting Verifications Turned In 1 Number pf Clients Requesting Case Status 1 Number of LEP (Language) Calls 1 Number of Referrals Made 1 Number of Transactions 1 Number of Trips Scheduled 1 Total Calls Receiving a Queue Full Message 1 Total Number of Calls Unanswered 1 Note: Counts are based on responses from the 45 states that participated in the study. Performance Measures and Aggregate Data Mathematica Policy Research A.5 Table A.2. Measures and Aggregate Data for Online Systems Number of States with Online System: 33 Name of Performance Measure or Aggregate Data Number of States with Measure or Aggregate Data Performance Measures Percent of Applications Received 15 Average Number Submitted per Month 12 Average Number of Screenings Completed: Online 9 Average Number of Screenings Started: Online 8 Average Number Started per Month 8 Average Number of Changes Submitted per Month 7 Percent of Users Selecting Spanish Language 6 Percent of Screened Individuals Potentially Eligible: Online 6 Average Level of Benefits 6 Average Number of Accounts Created per Month 6 Percent of Applicants Who Received Help to Apply Online 5 Percent of Screenings Resulting in Application Submission: Online 4 Percent of Applicants with Hardship Reason for Requesting Phone Interviews 4 Average Number of Log-ins per Month 3 Percent Reporting Easy Experience to Complete Application 2 Percent Who Would Use Again 2 Percent of Multiple Applications 2 Average Time to Complete Screening 1 Percent of Screenings Resulting in Applicant Receiving Benefits: Online 1 Average Number of Online Applications 1 Average Time to Complete Online Application 1 Average Time to Complete Online Screening 1 Percent of applications received outside business hours 1 Percent of applications started online that were submitted 1 Percent of First-Time Applications Approved 1 Percent of First-Time Applications Denied 1 Percent of Multiple Applications: Online 1 Aggregate Data Number of Online Screenings Completed: Online 13 Number of Applications Started 12 Number of Online Screenings Started: Online 9 Number of Changes Submitted 8 Number of Abandoned Applications 7 Number of Online Accounts Created 7 Number of Applicants who Received Help to Apply Online 6 Number of Log-ins 4 Number of Applications Downloaded 3 Number of One-page\/Requests for Assistance Submitted 3 Number of First-Time Applications Approved 3 Number of Clients Requesting Help to Apply Online 3 Number of Applications Submitted With All Application Questions Answered 3 Performance Measures and Aggregate Data Mathematica Policy Research Table A.2 (continued) A.6 Number of Applicants who did not Receive Help to Apply Online 3 Number of Changes Started 3 Number of Online Applications Submitted 3 Total Number of Applications Submitted per Month 2 Number of First-Time Applications Denied 2 Number of Applications Submitted 2 How Long Users Spent on Each Page of the Online Application 1 Number of Accounts Created 1 Number of Applicants Screened as Potentially Eligible 1 Number of Applications in Progress 1 Number of Applications Received 1 Number of Applications Submitted and Transferred 1 Number of Applications Submitted for Each Type of Program 1 Number of Multiple Applications 1 Number of One-Page Requests for Assistance Submitted 1 Number of Online Applications Submitted per Month 1 Number of Online Instant Messaging\/Chats 1 Number of Referrals From 311 Online 1 Number of Users Accessing a Specific Page 1 Number Submitted per Day 1 Percent Completed After Hours 1 Total Number of Website Hits per Month 1 Note: Counts are based on responses from the 45 states that participated in the study. Performance Measures and Aggregate Data Mathematica Policy Research A.7 Table A.3. Measures and Aggregate Data for Document Imaging Number of States with Document Imaging: 24 Name of Performance Measure or Aggregate Data Number of States with Measure or Aggregate Data Performance Measures Percent of Documents Scanned 9 Average Processing Time 9 Percent of Documents Received by Fax 5 Percent of Documents Received by Mail 4 Percent of Documents Imaged by Next Business Day 1 Percent of Documents Imaged Same-Day 1 Percent of Documents Scanned within Standard of Promptness 1 Ratio of Staff to Documents Scanned 1 Aggregate Data Number of Documents Scanned 18 Number of Documents Received 12 Number of Documents Processed 11 Number of Tasks Completed 11 Number of Documents Received: Mail 7 Number of Documents Received: Other Electronic 6 Number of Alerts Processed 6 Number of Documents Received: Fax 5 Amount of days in queue to be indexed 1 Number Indexed for a Document Type 1 number of barcode errors 1 Number of documents attached 1 Number of documents in Error Queue 1 Number of documents pending indexing 1 Number of Documents Per Case 1 Number of documents scanned but not attached 1 Number of Documents Tagged to a Case 1 Number of Queues with Over 1,000 Documents 1 Number of staff assigned to scanning 1 Percent of documents attached in a timely manner 1 Time Between Scanning and Indexing 1 Time Between Scanning and Tagging 1 Note: Counts are based on responses from the 45 states that participated in the study. Performance Measures and Aggregate Data Mathematica Policy Research A.8 Table A.4. Measures and Aggregate Data for Kiosks Number of States with Kiosks: 14 Name of Performance Measure or Aggregate Data Number of States with Measure or Aggregate Data Performance Measures Percent of Screenings Resulting in Application Submission: Kiosk 1 Aggregate Data Number of Uses to Submit Changes 3 Number of Uses to Access Online Account 2 Number of Screenings Completed: Kiosk 1 Note: Counts are based on responses from the 45 states that participated in the study. Performance Measures and Aggregate Data Mathematica Policy Research A.9 Table A.5. Measures and Aggregate Data for Partnering Number of States with Partners: 34 Name of Performance Measure or Aggregate Data Number of States with Measure or Aggregate Data Performance Measures Percent of Partners Providing Application Assistance 10 Average Cost of Partner Arrangement 5 Percent of Partners Providing Access to Terminals\/Kiosks 2 Aggregate Data Number of Partners 20 Number Providing Application Assistance 18 Number Accepting Applications 17 Number of Applications Received per Partner 15 Number Clients Assisted per Partner 9 Number with Terminals\/Kiosks to Submit Application 7 Number of Clients Assisted 2 Number of Applications Completed 1 Number of Applications Submitted per Month 1 Number of Clients Educated 1 Number of Clients served 1 Number of Days Applications are Pending 1 Number of Outreach Events 1 Number of Screenings per Partner 1 Note: Counts are based on responses from the 45 states that participated in the study. Performance Measures and Aggregate Data Mathematica Policy Research A.10 Table A.6. Measures and Aggregate Data for Waiver of Face-to-Face Interview Number of States with Waiver of Face-to-Face Interview: 40 Name of Performance Measure or Aggregate Data Number of States with Measure or Aggregate Data Performance Measures Error Rate for Cases with Telephone Interviews 16 Error Rate for Cases with Face to Face Interviews 14 Percent of Interviews Conducted by Telephone 13 Percent of Interviews Conducted Face to Face 13 Percent that are Missed and have Notice Mailed 11 Percent of Telephone Interview Requests Honored 5 Average Length of Interview 2 Average Total Interviews Per Worker 1 Percent of Closures Related to Telephone Interview 1 Percent of Interviews Conducted by Telephone for Redeterminations 1 Percent of Interviews that are automated 1 Percentage of Interviews Conducted by Phone without Documented Hardship Reason 1 Aggregate Data Total Number of Interviews 20 Number of Redetermination Interviews 19 Number of Face To Face Interviews 13 Number of Telephone Interviews 13 Number of Applicants Requesting a Phone Interview 12 Number of Missed Interviews 11 Notice of Missed Interview Mailed 11 Number of Applicants Requesting an In-Person Interview 8 Number of Home Visit Interviews 6 Number of Applicants who Did Not Answer to Type of Interview Requested 4 Number of Times Unable to Schedule Interview with Applicant 3 Note: Counts are based on responses from the 45 states that participated in the study. Performance Measures and Aggregate Data Mathematica Policy Research A.11 Table A.7. Measures and Aggregate Data for Shortened Interview Number of States with Waiver of Shortened Interview: 12 Name of Performance Measure or Aggregate Data Number of States with Measure or Aggregate Data Performance Measures Percent of Clients Electing Shortened Interview 2 Percent of Total Interviews that Are Shortened 2 Aggregate Data Number of Shortened Interviews 3 Number of Clients with Shortened Interviews 1 Note: Counts are based on responses from the 45 states that participated in the study. Performance Measures and Aggregate Data Mathematica Policy Research A.12 Table A.8. Measures and Aggregate Data for Complete Expedited Applications Online Number of States with Ability to Complete Expedited Applications Online: 20 Name of Performance Measure or Aggregate Data Number of States with Measure or Aggregate Data Performance Measures Percent of Applications Approved: Online All 10 Percent of Applications Approved: Online Expedited 9 Aggregate Data Number of Expedited Applications Received Online 9 Number of Applications Approved: Online Expedited 9 Number of Applications Approved: Online All 5 Note: Counts are based on responses from the 45 states that participated in the study. Performance Measures and Aggregate Data Mathematica Policy Research A.13 Table A.9. Measures and Aggregate Data for Application Tracking Number of States with Application Tracking: 45 Name of Performance Measure or Aggregate Data Number of States with Measure or Aggregate Data Performance Measures Percent Agency Caused Errors 41 Percent Client Caused Errors 41 Percent Errors with Reported Information Disregarded 40 Percent Errors where Information was Incorrect 40 Percent Errors where Policy was Incorrectly Applied 39 Percent Errors where Staff Failed to Follow-up on Inconsistent Data 39 Percent Errors where Information was Not Reported 39 Percent All Other Errors 38 Error Rate by Source 36 Percent of Total Applications Approved 33 Average Benefit Amount (for those eligible) 29 Percent of Complete Applications Denied 28 Percent of Total Recertifications Approved 28 Average Processing Time of Application Decision 27 Percent of Applicants that Appear Eligible for Expedited Benefits 21 Percent of Complete Recertifications Denied 20 Percent of Initial Applications Denied 19 Average Processing Time of Recertification Decision 19 Percent of Recertifications Denied 18 Percent of Applications Approved After Paper Submission to Local Office 14 Percent of Applications Not Completed 14 Percent of Applications: Paper Submission to Local Office 14 Percent of Recertifications Approved After Paper Submission to Local Office 13 Percent of Applications: Online 13 Percent of Recertifications: Paper Submission to Local Office 12 Percent of Community Partner Applications Approved 10 Percent of Mailed Recertifications Approved 10 Percent of Applications: Mailed 10 Percent of Mailed Applications Approved 9 Percent of Online Applications Approved 8 Percent of Applications: Community Partner 8 Percent of Applications with Incomplete Information 8 Percent of Applications: Faxed 7 Percent of Recertifications: Mailed 7 Percent of Online Recertifications Denied for Failure to Submit Documentation 6 Percent of One-page\/Requests for Assistance Denied 6 Percent of Recertifications with Incomplete Information 6 Percent of Community Partner Recertifications Approved 5 Percent of Recertifications: Online 5 Percent of Recertifications: Faxed 5 Percent of Online Applications Denied for Failure to Submit Documentation 5 Performance Measures and Aggregate Data Mathematica Policy Research Table A.9 (continued) A.14 Name of Performance Measure or Aggregate Data Number of States with Measure or Aggregate Data Percent of Applications: Call Center 4 Percent of Online Applications Denied for Failure to Complete Interview 4 Percent of Online Recertifications Approved 3 Percent of Call Center Applications Approved 3 Percent of Call Center Recertifications Approved 3 Percent of Recertifications: Community Partner 3 Percent of Recertifications: Call Center 3 Timeliness Rate 3 Percent of Applications: Kiosk 2 Average Timeliness Rate: Approvals 2 Average Timeliness Rate: Recertifications 2 Payment Accuracy Rate 2 Percent of Applications Approved Timely 2 Percent of Kiosk Applications Approved 1 Percent of Online Recertifications Denied for Failure to Complete Interview 1 Case Error Rate 1 Negative Error Rates by County 1 Percent of Accurate Cases 1 Percent of Cases Reviewed that Are Accurate 1 Percent of Negative Errors 1 QA Requirements Associated to Denial Decisions 1 QC \/ Case Review Timeliness Measure 1 QC Error Rate 1 QC Timeliness Measures 1 Timeliness Portion Of Case Review and QC Process 1 Percent of Applications Denied 1 Percent of Applications Pending 1 Percent Change in Applications Received by County 1 Percent of Applications Received Online 1 Percent of Applications: Expedited 1 Percent of Applications: In-Office 1 Percent of Total Recertifications Completed 1 Application Currency 1 Application Processing Time for Approvals for Initial Applications 1 Average Days to Process Approved Applications 1 Average Processing Time of Approvals (Application And Recertifications Combined) 1 Claims Processing Currency 1 Expedited Currency 1 Expedited Standard of Promptness Rate (County\/Worker Only) 1 Percent of Applications Processed Timely 1 Percent of Approved Applications Processed within 7, 15, 30, And 45 Days 1 Percent of Documents Processed Within 30 Days 1 Percent of Recertifications Approved Timely 1 Percent of Unexcused Applications Overdue at End of Month 1 Performance Measures and Aggregate Data Mathematica Policy Research Table A.9 (continued) A.15 Name of Performance Measure or Aggregate Data Number of States with Measure or Aggregate Data Unexpedited Standard of Promptness Rate (County\/Worker Only) 1 Aggregate Data Number of Agency Caused Errors 40 Number of Errors where Policy was Incorrectly Applied 39 Number of Errors with Reported Information Disregarded 39 Number of Client Caused Errors 37 Number of Errors where Information was Incorrect 36 Number of Errors where Information was Not Reported 36 Number of Errors where Staff Failed to Follow-up on Inconsistent Data 35 Number of All Other Errors 35 Total Number of Applications Completed 30 Total Number of Recertifications Completed 27 Number of Applications: Paper Submission to Local Office 21 Total Number of Applications Started 19 Number of Applications Approved: Paper Submission to Local Office 17 Number of Recertifications Approved: Paper Submission to Local Office 17 Number of Applications: Online 17 Number of Applications Denied: Paper Submissions to Local Office 17 Number of Recertifications Denied: Paper Submissions to Local Office 16 Total Number of Recertifications Started 15 Number of Cases with Complete Verification Submitted 15 Number of Applications Approved: Face to Face Interviewed 14 Number of Recertifications Approved: Face to Face Interviewed 14 Number of Applications Approved: Phone Interviewed 14 Number of Recertifications: Paper Submission to Local Office 14 Number of Recertifications Approved: Phone Interviewed 13 Number of Applications: Mailed 13 Number of Recertifications: Mailed 11 Number of Recertifications Approved: Mailed 10 Number of Applications Approved: Community Partner 9 Number of Applications Approved: Mailed 9 Number of Applications: Community Partner 9 Number of Requests for Assistance\/One-page Applications 9 Number of Applications Denied: Community Partner 9 Number of Applications Approved: Online 7 Number of Recertifications: Online 7 Number of Applications: Faxed 7 Number of Applications Denied: Mailed 7 Number of Recertifications Denied: Mailed 7 Number of Recertifications: Community Partner 6 Number of Applications Denied: Online 6 Number of Recertifications Approved: Community Partner 5 Number of Applications Approved: Faxed 5 Number of Recertifications: Faxed 5 Number of Recertifications Denied: Online 5 Performance Measures and Aggregate Data Mathematica Policy Research Table A.9 (continued) A.16 Name of Performance Measure or Aggregate Data Number of States with Measure or Aggregate Data Number of Applications Denied: Faxed 5 Number of Recertifications Denied: Faxed 5 Number of Recertifications Approved: Faxed 4 Number of Recertifications Denied: Community Partner 4 Number of Applications Approved: Call Center 3 Number of Recertifications Approved: Call Center 3 Number of Applications: Kiosk 3 Number of Applications: Call Center 3 Number of Recertifications: Call Center 3 Number of Applications Denied: Call Center 3 Total Number of Applications Approved 3 Total Number of Recertifications Approved 3 Number of Applications Denied 3 Number of Recertifications Approved: Online 2 Number of Recertifications Denied: Call Center 2 Number of Applications Approved 2 Number of Applications Approved: Kiosk 1 Number of Applications Denied: Kiosk 1 Contributing Causes of Errors 1 Number of Applications Approved: Total 1 Number of Expedited Applications Approved 1 Number of Initial Applications Approved Over 30 Days 1 Number of Second Party Reviews 1 Number of Total Applications and Recertifications Approved 1 Denials for Expedited Applications by Reason 1 Number of Applications Denied by Denial Reason 1 Number of Applications Denied: Total 1 Number of Expedited Applications Denied 1 Number of Recertifications Denied 1 Number of SNAP Cases Closed by Closure Reason 1 Total Number of Applications Denied 1 Total Number of Recertifications Denied 1 Number of Applicants that Appear Eligible for Expedited Benefits 1 Number of Applications Not Submitted 1 Number of Applications Submitted Outside Business Hours 1 Number of Applications that Appear Eligible for Expedited Benefits 1 Number of Applications with Incomplete Information 1 Number of Applications Withdrawn by Applicant 1 Number of Applications Withdrawn by System 1 Number of Cases Pending 1 Number of Cases Withdrawn 1 Number of Registered Applications 1 Number of Applications Online 1 Number of Applications Pending Decision 1 Number of Applications Processed: Paper Submission to Local Office 1 Performance Measures and Aggregate Data Mathematica Policy Research Table A.9 (continued) A.17 Name of Performance Measure or Aggregate Data Number of States with Measure or Aggregate Data Number of Applications Received 1 Number of Expedited Applications Pending 1 Number of Pending Applications 1 Number of Recertifications Approved Over 30 Days 1 Number of Recertifications Processed: Paper Submission to Local Office 1 Number of Resurrected Applications 1 Total Calls Busy 1 Total Number of Applications 1 Total Number of Applications To Be Processed 1 Total Number of Recertifications 1 Applications in Intake Mode over 30 days 1 Applications Not Timely 1 Applications Process Time 0-7 Days 1 Applications Process Time 8-29 Days 1 Number of Applicants Pre-Screened, by Worker 1 Number of Applications that Are Not Processed Timely 1 Number of Cases Ready To Go in <25 Days, by Worker 1 Number of Cases Ready To Go in >24 Days\/Resurrected, by Worker 1 Number of Cases Ready To Go, by Worker 1 Number of Days Application Pending 1 Number of Expedited Applications Approved in 5 Days or Less 1 Number of Expedited Applications Approved or Rejected in More than 5 Days 1 Number of Expedited Applications Pended in 5 Days or More 1 Number of Overdue Applications 1 Pending Applications by Days in Process 1 Pending Cases >30 Days, by Worker 1 Processing Time Report 1 Total Number of Pending Cases, by Worker 1 Note: Counts are based on responses from the 45 states that participated in the study. Although all states collect measures for the purposes of quality control, we did not require states to report these measures when they did not do so for purposes outside of the federal requirements. The totals of the quality control-based measures reflect a combination of those who collect them for federal reviews and those who collect them for their own purposes. www.mathematica-mpr.com Improving public well-being by conducting high-quality, objective research and surveys Princeton, NJ \u25a0 Ann Arbor, MI \u25a0 Cambridge, MA \u25a0 Chicago, IL \u25a0 Oakland, CA \u25a0 Washington, DC Mathematica\u00ae is a registered trademark of Mathematica Policy Research SNAPModernization Profiles .pdf Performance Measurement for Supplemental Nutrition Assistance Program Modernization Initiatives: State Profiles Report Final Report Performance Measurement for Supplemental Nutrition Assistance Program Modernization Initiatives: State Profiles Report Final Report Acknowledgments Executive Summary Diversity in State Measures Challenges of Comparing Measures Data Collection I. Study Background, objectives, and Analytical Approach A. Overview of Data Collection 1. Preliminary Assessment and Instrument Development 2. Respondent Selection and Data Collection 3. Results of Data Collection Efforts B. Data Analysis Approach 1. Constructing State Profiles 2. Identifying Challenges and Performance Across States II. How are States Performing? A. Observed Results on Measures Across States 1. Call Centers 2. Online Systems 3. Document Imaging 4. Kiosks 5. Partnering 6. Interview Initiatives 7. Application Tracking B. Setting Standards and Benchmarks 1. Preventing Corrective Action 2. Encouraging Staff Productivity 3. Monitoring and Sustaining Performance 4. Responding to Legal Action C. Conclusion III. Limitations to use of measures and standards A. Contextual Limitations 1. Initiative Scope and Features 2. System Limitations 3. Terminology Differences 4. Varying Measurement Motivations B. Data Limitations 1. Accuracy 2. Validity 3. Reliability C. Conclusion References Glossary Modernization Initiatives Measurement Terminology SNAP Modernization .pdf Performance Measurement for Supplemental Nutrition Assistance Program Modernization Initiatives: Integrated Report Final Report Performance Measurement for Supplemental Nutrition Assistance Program Modernization Initiatives: Integrated Report Final Report ACKNOWLEDGMENT CONTENTS TABLES executive summary Methods Background Key Study Findings Limitations Setting Measurement Requirements: Recommendations and Limitations I. Introduction A. What Is Modernization? B. Why Modernize? C. What Modernization Initiatives Have States Put in Place? D. Why Is Performance Measurement Important? E. How Are Measures Currently Used for Federal Oversight? F. How Have States Used Performance Measures? 1. Early Stages of Implementation 2. Maintenance Stage of Implementation 3. Real Time \/ Daily Measures \/ 4. Focus on Quality and Timeliness G. How Are Measures Assessed? H. Study Objectives and Research Questions a. Describe Each State’s SNAP Performance Measures and Standards in Detail: b. Provide a Detailed Description of How Performance Measures Are Implemented, What Results Are Observed, and How the Results Are Used in Each State c. Assess the Pros and Cons of Applying Alternative Performance Measures and Standards Across States II. methods 1. Introduction 2. Who We Interviewed 3. What We Learned 4. A Limited Set of Initiatives to Examine 5. Data Collection Instruments 1. Selecting Respondents 2. Responses Received 3. Level of Response III. summary of Performance Measures Across Initiatives A. Purpose of Measures by Modernization Goal B. Key Performance Measures and Gaps, by Initiative 1. Call Center 2. Online System 3. Document Imaging 4. Kiosks 5. Partnering (for Application Assistance) 6. Waiver of Face-to-Face Interview 7. Shortened Interviews 8. Expedited Applications\u2014Submitted Online C. General Gaps in Performance Measurement D. Use of Standards, Benchmarks, and Incentives E. Relationship of Measures to SNAP Lifecycle IV. Limitations to use of measures and standards for cross-state comparisons A. Contextual Limitations 1. Initiative Scope and Features 2. System Limitations 3. Terminology Differences 4. Varying Measurement Motivations B. Data Limitations 1. Accuracy 2. Validity 3. Reliability V. Opportunities and Challenges for Uniformly Measuring initiatives A. Criteria for Inclusion in the Proposed Set of Measures 1. Addressing Goals for Modernization 2. Feasibility as a Reporting Requirement B. Proposed Set of Measures 1. Application Tracking 2. Call Centers 3. Online Systems 4. Document Imaging 5. Partnering (Application Assistance) 6. Waiver of Face-to-Face Interview VI. Conclusion A. Summary of Study Findings B. Setting Measurement Requirements: Recommendations and Limitations C. Next Steps References Glossary Modernization Initiatives Measurement Terminology appendix a Performance Measures and Aggregate Data ”