John M. Hughes,
Assistant Director
Office of Probation and Pretrial Services
The Administrative Office of the U.S. Courts

THE FEDERAL JUDICIARY is committed to developing a results-based management and decision-making framework for its probation and pretrial services program. This article will describe progress in the largest area of that program: post-conviction supervision by probation officers.1 The goal is to collect, analyze, and use data from probation officers’ electronic case files and a variety of other sources to inform decision-making and drive performance improvements aimed at reducing recidivism and fostering long-term positive changes in individuals supervised. On an ongoing basis, we want the ability to test underlying assumptions about the relationship between supervision practices and supervision outcomes.

The process has necessarily been slow and methodical. After all, we are attempting to transform a large system that spans all 50 states and United States territories,2 but also one that places considerable decision-making autonomy in each of the 94 district courts.

The governance structure of the federal judiciary is largely decentralized. The judiciary governs itself primarily through local mechanisms, with the Judicial Conference and the circuit councils setting policy and providing guidance where necessary. Judges at the local level enjoy considerable latitude to structure their court operations to suit local conditions.3

Day-to-day operations of probation offices are managed locally, including the hiring and managing of staff and the use of resources.4 While chief probation officers may only be removed for “cause,” they—and chief pretrial services officers—understandably conform their offices to the culture and practices of the court. This ensures adaptability to local conditions and responsiveness to the needs of the courts. It also can make implementation of national policies and initiatives challenging.

We have made progress toward becoming results-based in our decentralized system though consensus building, collaboration, and communication, which will be discussed below. Essential to making progress, however, has been the strong leadership of the Criminal Law Committee of the Judicial Conference of the United States.5 The Committee, which has broad jurisdiction over the federal probation and pretrial services system, is firmly committed to the overall strategic direction to become results-based6 and also has expressed support for the use of evidence-based practices (EBP) in developing probation and pretrial services policies. In that regard, the Committee was particularly influenced by a study in Washington State (see Aos, Miller, & Drake, 2006). Further, the Committee seeks to use empirical cost-benefit analyses in developing and justifying the annual budget request for the probation and pretrial services program.

Eventually, we will have the ability to measure performance for all parts of probation and pretrial services work: 1) pretrial services investigations; 2) pretrial services supervision; 3) presentence investigations; and 4) post-conviction supervision. Some efforts have been made in each area, but our initial focus has been on post-conviction supervision because it is the single largest part of the system’s workload and budget.

Figure 1

Evidence-Based Practices

Our strategic goal to become a results-based program fits hand-in-glove with adopting evidence-based practices (EBP). In fact, while we build the outcomes-based system, we have been promoting EBP on our website, in our newsletter, and in other communications.7 We have made presentations about our strategic goal and also promoted EBP at dozens of conferences, training events, and new chief orientations since 2006.8 The Federal Judicial Center (FJC)9 is complementing this effort in 2008 with programs for U.S. circuit, district, and magistrate judges that address offender re-entry and EBP. The AO has added staff with EBP expertise and has made them available to probation and pretrial services offices seeking training or guidance. We established the Research-to-Results (R2R) pilot in 18 sites10 in 2006 to introduce various EBPs and learn more about how to implement them on a broader scale.

How We Got Here

We have followed three tracks over the past several years to arrive at the simple goal of becoming a performance-based system. Each track is described in the first section of this paper:

• two “futures planning” conferences;
• a broad strategic assessment by a team of independent consultants; and
• a collaborative effort by court subject matter experts, FJC and AO staff to improve post-conviction supervision policies, including substance abuse and mental health issues.

The rest of the paper will describe how we developed a methodology to become a performance-based system and how we are developing the technology with which to make it a reality.

I. The Three Tracks

A. Futures Planning

The End of “Nothing Works”. Two futures-planning conferences of chief probation and pretrial services officers sponsored by the Federal Judicial Center (FJC)11 were invaluable in developing a strategic vision unique to federal probation and pretrial services and reaching consensus on the system’s goals and values. Nearly all chief probation and pretrial services officers participated in the first one in San Antonio in 2000, the theme of which was “Working Together to Shape the Future.” Structured brainstorming sessions were particularly successful at generating lively discussions not about what the system does, but about what the system should be trying to accomplish. Many participants noted that it was refreshing after so many years under the cloud of “nothing works” that they were once again agreeing that people can change, and that proactive interventions by officers can facilitate long-term, positive changes in offenders. A common observation was “this is why I got into this business in the first place.”12

The enthusiasm generated at the conference was accompanied by wide agreement that Congress and the public will hold the system increasingly accountable in the future and that we must establish desired outcomes that are clear, measured, and communicated. While other goals were identified,13 the importance of clarifying and agreeing upon the core mission and creating a system by which to measure desired outcomes emerged as the top priorities.

The clearest direction that emerged from the futures exercises was agreement on the need for a shared understanding of our core mission plus increased accountability to one another, the public, and our funding sources for accomplishing the objectives that comprise the mission.14

Following the 2000 conference, the FJC established working groups of chiefs and FJC and AO staff to maintain the momentum. The efforts continued at another chiefs conference in Salt Lake City in 2002, when the chiefs reached agreement on a Charter for Excellence, a document that spells out a shared understanding about the work of probation and pretrial services officers, the goals that matter most, and the values that the system stands by.

Figure 2
Charter for Excellence

Conceived at the Federal Judicial Center’s 2000 and 2002 National Chiefs’ Conference

At the 2002 conference, chiefs agreed that the Charter would be a useful tool to help shape a common system culture and influence how everyone in the system does his or her work on a day-to-day basis. They felt that Charter principles could influence how we implement policy, procedures, and other operational processes. They suggested that events at the district, circuit, and regional levels could be centered around promotions of the Charter. Toward this end, the FJC formed a working group made up of chiefs and AO and FJC staff to keep emphasis on the Charter and ensure its lasting relevance.15 The group, whose work continues, has tracked how probation and pretrial services offices have implemented the Charter and continues to look for ways to maximize its value to the probation and pretrial services.16

B. The Strategic Assessment

The AO contracted with a team of independent consultants, led by IBM Business Consulting Services,17 in 2000 to conduct a comprehensive strategic assessment of the federal probation and pretrial services system. The purpose of the study was to make recommendations for ensuring the future quality and success of the probation and pretrial services system through analysis, comparative research on other systems, and broad consultation with outside experts, judges, probation and pretrial services staff, and other stakeholders.

The IBM “Study Team” collected and analyzed quantitative and qualitative data to identify what the system does well, what could be improved, and what factors facilitate or impede the quality of service delivery in each of the system’s functional areas, including pretrial services investigations and supervision, presentence investigations, and post-conviction supervision. More specifically, the Study Team:

• reviewed all relevant statutory and regulatory directives, policy and program guidance, planning documents, and research studies;
• analyzed historical and projected workload, budget and staffing data;
• conducted interviews with more than 300 individuals, including past and present system leaders on the Criminal Law Committee and the Chiefs Advisory Group, and the AO, and other stakeholders in the Department of Justice, the Federal Judicial Center, the United States Sentencing Commission, the General Accountability Office, prosecutors, defense bar, and staff of the Senate Judiciary Committee;
• conducted focus groups totaling 170 individuals from 20 district courts;18
• surveyed a random sample of 110 federal district judges and 115 federal magistrate judges19;
• surveyed all chief probation and pretrial services officers;20
• researched how other probation and pretrial services operations are generally organized.21

The result of the Study Team’s work was published in a September 2004 report (see Strategic Assessment, 2004) that noted “staff are highly regarded by internal and external stakeholders” and that “stakeholders generally reported satisfaction with system functions.” The report also noted: “Although considerable resources are devoted to reviewing the extent to which district procedures and activities comport with national standards, the system currently conducts only limited assessment of the outcomes of its work.”

The Study Team explained that it was not possible to assess the effectiveness of policies and procedures because the system did not have a performance-based management system that links mission and goals to strategies and anticipated outcomes. The IBM report contained three sets of recommendations, each with several steps. The main recommendation was to “become a results-driven organization with a comprehensive outcome measurement system”. The consultant recommended that we seek expert assistance to deal with the complexities of creating such a system. The complete list of recommendations is shown below.

Figure 3
Recommendations Overview

C. Fine Tuning Our Post-Conviction Supervision Practices

In January 2000, at around the same time that the AO and chiefs were working with the FJC on our “desired future,” and while the team of consultants was getting started on the strategic assessment, the AO formed a working group to review and update the supervision policies contained in The Supervision of Federal Offenders (Monograph 109).22 One objective was to ensure that recent crime legislation and court decisions were reflected in our policies. Another was to incorporate practices from the literature that had been shown by research to reduce recidivism.

The Supervision of Federal Offenders helped lay the groundwork for transforming post-conviction supervision into an outcomes-based program. First of all, it clearly defined the desired outcomes of post-conviction supervision as “the execution of the sentence and the protection of the community by reducing the risk and recurrence of crime and maximizing offender success during the period of supervision and beyond.”23

The goal of supervision in all cases is the successful completion of supervision during which the offender commits no new crimes, is held accountable for victim, family, community and other court-imposed responsibilities, and prepares for continued success through improvements in conduct and condition. The emphasis on continued success after the period of supervision is in recognition that our ultimate goal is to change the offender’s behavior and thereby protect the community and reduce costly recidivism.

The new policy recognizes that successful reentry into the community is fostered by having the probation officer engage the offender as early as possible in planning, preferably while the offender is still in custody. Early involvement allows the officer to identify basic survival needs and interventions that will support the offender and make success more likely (see Taxman, Byrne, & Young, 2003). While good probation officers have always developed initial case plans that recognize the risks, needs, and strengths of each individual offender and apply a corresponding level of supervision, the policy now clearly emphasizes this practice in all cases: “The purpose of supervision planning is to create an evolving, individualized outcome-based plan of action to monitor compliance with the conditions of release and intervene as necessary to address any identified risks.”

Whether to control risk or provide treatment, the level of supervision should be proportionate to what is needed and not more. Officers use the Risk Prediction Index (RPI)24 to statistically estimate the likelihood that the offender will be arrested or have supervision revoked, but they also consider other types of risks presented by the individual, including the risk of committing sexual abuse or associating with criminals. Officers also identify needs, such as for stable residence, meaningful employment, and substance abuse and/or mental health treatment. They also identify strengths on which the offender may build to increase the likelihood of success, such as a particular talent, strong motivation, and a supportive prosocial community network of family and/or friends. We recognize the need for a standard risk/needs assessment tool for use in the federal system and hope to have one soon, but in the meantime it is the policy that probation officers assess each offender on a case-by-case basis using the RPI and all information available and tailor the case plan accordingly.

Community supervision is now more clearly viewed as a dynamic, issues-driven process in which the probation officer must stay informed and be responsive, intervening as necessary with strategies tailored to changing risks, needs, and strengths of each individual under supervision. Planning must include specific goal-directed objectives to be accomplished by the offender and monitored by the officer. The offender must be actively engaged in planning his own future. The officer is expected to develop different or additional approaches in response to emerging risk issues or instances of noncompliance, and to discontinue planned strategies that no longer have a purpose or are not working .

To achieve the desired outcomes of supervision, probation officers are expected to apply the principles of good supervision in every case. Their application will help to ensure that supervision resources are applied where they are needed the most and are not wasted on cases where little or no intervention is necessary. Good supervision is:

• individualized—tailored to the offender’s risks, needs, and strengths
• proportional—involving the least intrusive means necessary
• purposeful—directly related to the case objectives
• multidimensional—using concurrent strategies from a variety of disciplines
• proactive—-actively monitoring changes in behavior
• responsive to changes— adjusting as needed on an ongoing basis

Before issuing the revised Supervision of Federal Offenders (Monograph 109), the AO and the FJC had partnered to develop a training and implementation plan. Among other reasons, we wanted to ensure that the new focus on outcomes and the officer’s proactive role was understood.25

The AO established a separate working group in 2005 to focus specifically on the supervision of offenders in need of substance abuse and mental health treatment.26 The group met often over the next two years to develop proposed revisions that would bring the Supervision of Federal Offenders in line with current knowledge in the field of substance abuse treatment and probation supervision. Substance abuse treatment concepts were clarified.

• A Model for Positive Change: This section outlines a conceptual model that explains how to assist offenders in making long-term changes to their behavior.
• Role of the General and Treatment Specialist Officer: This section outlines the responsibility of the officer and responsibility of the treatment specialists.
• Confidentiality and Disclosure: This section addresses the complexities of what information must be kept confidential and under what circumstances this information can be released.
• Correctional Strategies: This section provides detail about strategies that will encourage offender behavior change. Topics covered in this section include case planning, treatment referrals, treatment matching, program fidelity, and the use of incentives.
• Controlling Strategies: Despite all of the probation officer’s efforts to the contrary, some offenders fail to comply with the conditions of the court. This section provides guidance in monitoring and responding to non-compliance. Topics covered in this section include drug and alcohol testing and responding to relapse.

The section dealing with mental health treatment was also updated to reflect current knowledge in the field of mental health treatment and probation supervision, including treatment for both mental health and substance abuse disorders, as the working group reasoned that it is often difficult to disentangle the two disorders. Examples of guidance that are new to the monograph include the following:

• Correctional Strategies: This section addresses how to intervene in order to stabilize an offender’s mental illness while identifying and addressing the root cause(s) of criminal behavior. Topics in this section include case planning, screening and assessment, treatment selection, treatment fidelity, and the use of incentives.
• Controlling Strategies: Mental illness symptoms can cause non-compliance in probation supervision. This section provides guidance to understand and treat mental illness while keeping both the offender and the community safe. Topics addressed in this section include managing non-compliant behavior, managing decompensation, responding to threats of violence and emergency mental health commitments.
• Concurrent Planning: Offenders with mental illness may need lifelong formal medical care to manage their symptoms. This section provides guidance for officers to assist offenders in establishing community-based services that will exist beyond the term of supervision.

Chiefs, deputy chiefs, supervising officers and the AO are again working with the FJC to introduce the new changes to the Supervision of Federal Offenders and provide training.27

II. Building the Infrastructure

A. Developing the Methodology

In March 2004, the AO invited experts on outcome measurement methodology to serve on an ad hoc panel to help refine operational definitions and associated measures for each “desired outcome” and suggest statistical approaches for analyzing the information that will assure “apples-to-apples” comparisons and allow benchmarking with other programs. The panel was made up of the directors of research for the Federal Judicial Center and the Federal Bureau of Prisons, and academics from Temple University and the University of Maryland.28

The panel recommended that the AO use the same measures of recidivism used at the Federal Bureau of Prisons because of the significant population overlap and because the research could be efficiently coupled. The AO thus adopted two measures of recidivism: 1) arrests for new criminal offenses, and 2) charges for new criminal offenses with revocations resulting in a return to federal prison. The panel also recommended following cohorts of offenders received for supervision for uniform follow-up periods, including periods beyond the conclusion of supervision.

Regarding substance abuse, the panel recommended that the AO use definitions in common use by the substance abuse treatment and prevention community, including SAMHSA and NIDA, regarding the characteristics of the abuse, interventions, and treatment, and both program outcome and treatment outcome. The AO adopted the following definition of substance abuse: the use of illegal drugs or maladaptive patterns of use of alcoholic beverages and other legal substances, including prescription drugs. Because legal and illegal substances are treated differently under this definition, the two should be treated differently in the results-based framework. Drug abuse is the use of illegal drugs, including occasional use, maladaptive patterns of use and dependence. Alcohol abuse is a maladaptive pattern of alcohol use or dependence on alcohol.

The AO created a Data Analysis Branch to evaluate, analyze, and recommend data quality improvements that address data accuracy, timeliness, and accessibility. We also formed a “data quality improvement working group” chaired by a chief probation officer; developed and distributed a data quality manual; created a data quality web site; and held regional conferences in Philadelphia, Memphis, and San Diego for a total of 287 data quality analysts and managers from probation and pretrial services offices from every district. The essential message we conveyed repeatedly was that data will become more and more important in the future and that each probation and pretrial services office needs to develop local data quality improvement programs.

In April 2005, the AO contracted for expert help in identifying the steps that must be taken to develop the technical aspects of the results-based framework. As a start, the contractor conducted a comprehensive search of official sources29 and created a glossary of key terms identified by the ad hoc expert panel and others that would be used in the development and documentation of our outcome-based decision-making framework,30. We hope this will avoid confusion about terms and definitions as we proceed.

With input from system stakeholders, the contractor also developed a “logic model” that depicts how supervision and programming support for persons under supervision are thought to be linked to desired outputs, interim outcomes, and ultimate outcomes. The purpose was to help visualize how we can test underlying assumptions about the relationship between what the system does and what it is trying to accomplish, and what other factors are thought to influence this relationship. The contractor recommended seeking expertise in advanced statistical techniques that can be applied to test the relationships in the logic model between discretionary processes while controlling for inputs that are primarily static and outside the control of the probation officer31.

Figure 4

In September 2006, the AO entered into a contract to obtain that expertise.32 The contractor first refined the logic model in order to organize its analyses and begin to quantify the paths of the model. Refinements will continue over time, but the evolving logic model reasonably represents the commonly-accepted understanding of how the system works.

The contractor created an analysis file made up of data from more than 100,000 supervision terms commenced in fiscal years 2005 and 2006. This cohort will be followed over time, with additional data being added from other judiciary sources and external sources to the analysis file as it becomes available. When this is complete, we will have a sound baseline on which to conduct more sophisticated analyses to see relationships among variables in ways not previously possible and move us closer to having the capacity to conduct causal analysis.

B. Upgrading the Technology

A basic challenge that we face is transforming a data-collection system that had evolved over the years for administrative purposes into one that could also be used for research. For example, for accounting purposes we tracked spending on substance abuse treatment, but not treatment or relapse prevention services for which there were no expenditures. Likewise, we knew what we paid for drug testing and the results when officers used contract laboratories, but not when they used other testing methods paid for locally. While we had data concerning actual revocations, we lacked data concerning strategies applied to non-compliance with conditions short of revocation.

We had already begun in October 2001 to roll out a second major generation of the Probation and Pretrial Services Automated Case Management System (PACTS) well before the Charter for Excellence (2002), the new Supervision of Federal Offenders (2004) and the Strategic Assessment (2004) were complete. We have since tried to address limitations by adding data elements to PACTS that will support our analyses, but have had mixed results. For example, a July 2004 release added data elements to capture data for non-contract treatment services, including in-house, “free” community-based programs, and private providers paid by the offender. Unfortunately, the use of the new data elements is spotty across the system as many officers fail to enter the information. This has been true with other data elements as well.

We recognize that officers are not normally keen on collecting data and doing paperwork more relevant to research than to day-to-day operations. Plus, we would rather they spend their time supervising offenders and serving the court. For both these reasons, our long-term goal is to extract outcome measures and other data directly from the automated case management tool used by officers in their day-to-day duties. We are developing the Electronic Reporting System (ERS), for example, to allow treatment vendors, drug labs, and even offenders to communicate with officers via the Internet. This will make case management more efficient for officers while also allowing us to capture that information for research purposes. In this way, data will simply be a by-product of officers using the system and its collection would be mostly invisible. Further, we assume that data actually used by the officer in the course of managing cases will be more accurate than data entered solely for statistical gathering purposes.

“PACTS Generation 3” is currently under development as a replacement for the current generation, and is expected to be rolled out in fiscal year 2011. PACTS Gen3 will be more useful to officers as a day-to-day tool and will be more intuitive . It is being designed from the ground up by officers33 for officers and will likely carry a significantly different user interface from the current version of PACTS. There will be one record per offender regardless of how many districts may have investigated or supervised the individual. Further, it will for the first time create a single national database containing all cases opened by probation officers in all 94 districts. PACTS Gen3 will result in a superior case-management tool providing managers and policy makers with a greatly enhanced outcomes-based framework. It will better enable users to “plug-in” locally developed applications. With one record per client, users will simply click on the screen to transfer information and will no longer have to send electronic and paper documents

While PACTS will remain our principal data source, we are working with other parts of the judiciary34 and other government agencies to add a variety of rich sources to the analyses and move us closer to cause-effect analyses of relationships among variables. So far, we have agreements with the Bureau of Prisons,35 the FBI,36and the Bureau of the Census to obtain data, and are in discussions with others.37

Simultaneously, we are developing the Probation and Pretrial Services Decision Support System (DSS)38 as the means to integrate those mountains of data in a single “data warehouse” that we call the National PACTS Reporting (NPR) System. In combination with business intelligence (BI) tools, we will be able to combine the data in any order at different levels of summarization over various time periods.39 In short, it will turn data into useful information.

Figure 5

III. Collaboration and Communication

Over the past eight years we have pursued the goal of creating a results-based system with a comprehensive outcome-measurement system. Despite our decentralized structure, system leaders reached consensus in developing the Charter that we are a national system with shared goals and values and will work collaboratively to achieve those goals. Further, we have shaken off the “nothing works” era and reinforced a system-wide belief that proactive interventions by officers can facilitate long-term, positive changes in offenders.

Our efforts have been collaborative: working groups of chiefs, deputy chiefs, supervising officers, senior officers, and line probation and pretrial services officers from all over the system have been involved in different parts of the effort.40 We have worked closely with the FJC and have consulted criminal justice experts from academia41 and entered into contracts for technical expertise and business intelligence software. Slowly B but surely B a results-based decision-making framework is taking shape.

Ongoing communication about our strategic direction would be important in any case, but is especially so in the federal probation and pretrial services system due to mandatory retirement at age 57. Of the 119 chiefs on board at the time of the 2000 National Chiefs Conference, when the call to become outcomes-based was first uttered, a total of 92 (77 percent) retired over the next eight years. Given that high rate of turnover, the AO has paid particular attention to maintaining continuity. We have made presentations about our strategic direction and tied it to a promotion of EBP at dozens of conferences, training events, and new chief orientations. In 2006, we held the first of many “Chiefs and Deputy Chiefs Administrative Meetings” as a means to discuss the strategic direction in which we are going as well as EBP and “nuts and bolts” administrative issues. These meetings have been particularly helpful in discussing our strategic direction42.

Conclusion

As we get closer to completing the technical infrastructure, we get more excited about the possibilities of testing assumptions about the relationship between what officers do with actual accomplishments. Most of us got into this business to make a difference and will welcome the possibility to learn how to do that more effectively. We inherited an excellent system from those who came before us, and want to leave it not just in good shape, but with the means for continuous improvement for many years to come.

back to top

References | Endnotes