Volume 70 Number 2
Federal Probation
 
     
     
 
Assessing the Role of Clinical and Actuarial Risk Assessment in an Evidence-Based Community Corrections System: Issues to Consider
 

James M. Byrne
Department of Criminal Justice and Criminology, University of Massachusetts, Lowel
April Pattavina
Department of Criminal Justice and Criminology,University of Massachusetts, Lowell

Issue 1: Evidence-based Practice and the (Missing) Link Between Risk Assessment and Risk Reduction
Issue 2: The Implications of Actuarial and Clinical Assessment for Line Staff and Management
Issue 3: The Need to Combine Individual and Community Risk Assessment
Concluding Comments

THE RISK ASSESSMENT process is undergoing major change in federal, state and local community corrections agencies across the country. New assessment instruments are being introduced, case management systems are being redesigned, and the roles and responsibilities of line staff and management in community corrections agencies are being redefined, in large part due to the application of new, “soft” computer technology in community corrections agencies (Pattavina and Taxman, 2006). As Gottfredson and Tonry (1987) predicted in the late 1980s, “both the literature and practical application of science-based prediction and classification will continue to expand as institutions evolve to become more rational, more efficient, and more just” (vii). While rationality, efficiency, and justice are laudable goals for any criminal justice organization, we suspect that ultimately, it is the effectiveness of the community corrections system—both in terms of short-term offender control and long-term offender change—that really matters to the public, and by extension, to policymakers and practitioners. In the following article, we examine three key issues related to assessing the effectiveness of risk assessment procedures that need to be addressed: 1) evidence-based practice and link between risk assessment and risk reduction, 2) the implications of both actuarial and clinical assessment for line staff and management, and 3) the need to combine individual risk assessment and community risk assessment in the next generation of risk-driven community corrections strategies. We conclude by offering three simple recommendations designed to improve the effectiveness of the risk assessment process in federal, state, and local community corrections agencies.

back to top

Issue 1: Evidence-based Practice and the (Missing) Link Between Risk Assessment and Risk Reduction

When the term “best practices” is used, it typically refers to the results of an evidence-based review of the research on a topic of interest (e.g. scared straight programs, prison and community-based treatment programs, etc.). Essentially, there are three different types of evidence-based reviews: 1) the “gold standard” evidence-based review focuses only on randomized, controlled experiments; 2) the “bronze standard” evidence-based review includes both experimental and well-designed quasi-experimental research, while using nonexperimental research studies to confirm findings from higher quality research; and 3) the unscientific (or nonsense) review, which does not identify specific study review criteria, relying instead on a selected subset of all studies available for review on the topic of interest. Not surprisingly, the last category of unscientific reviews is usually written by advocates of a particular program or strategy. In their most extreme form, the authors of the review simply allude to an evidence-based review or to “best practices,” with no supporting documentation. Unfortunately, much of what is currently available in the corrections field—both institutional and community corrections—falls into this last category.

Applying the “gold standard” for evidence-based reviews to the “risk assessment” process in community corrections mandates that at least two randomized field experiments must have been conducted in this area before we can offer an assessment of “what works” (see, for example, the reviews conducted for the Campbell/Cochrane collaboration at www.campbellcollaboration.org). Unfortunately, no experimental research has been conducted on this topic in community corrections, leading us to conclude that we don’t know whether there is a link between risk assessment (i.e., classification of an offender into high-risk, medium-risk and low-risk classification categories) and risk reduction (i.e., a lower rate of recidivism for offenders than anticipated, given their risk level) due to the types of supervision and services we make available to offenders at each level of risk.

Much of what we currently do in community corrections is based on assumptions about the risk-reduction effects of placing offenders into different supervision levels that have not been tested empirically, using randomized field experiments. What would happen, for example, if we placed high-risk offenders under “medium” or “low” supervision? Alternatively, what would be the impact of placing a low- or medium-risk offender under “maximum” supervision? Until we have the results of quality, experimental research to review, we will continue to make assumptions about “what works and “best practices” in terms of both risk assessment and risk reduction that are simply not supported by a careful, “gold-standard” evidence- based review.

back to top

Issue 2: The Implications of Actuarial and Clinical Assessment for Line Staff and Management

One argument that can be made concerning the use of clinical vs. actuarial risk assessment is that the line staff currently hired in community corrections do not have the background and qualifications necessary to conduct “clinical” assessments of offender risk, particularly for special category (e.g. mental health, substance abuse, sex offender) and multiple problem offenders. Assuming for the sake of argument that you want to introduce clinical assessments into your federal, state, and local community corrections agency, you have two choices: 1) recruit/hire line staff with the necessary qualifications to conduct clinical risk assessments (perhaps with minimal additional training); or 2) privatize the assessment process, using the network of current mental health treatment providers as your “target” potential provider. Indeed, it could be argued that by moving away from clinical and toward actuarially-based risk assessments, we are attempting to simplify classification/decision-making in an effort to reduce the need for higher-skilled line staff (i.e., the “dummying down” of community corrections). In this scenario, it is possible to envision a probation or parole agency where line staffs are responsible for case planning and supervision, but other functions (assessments, treatment, and services) are subcontracted to agencies in the private sector.

While improving staff quality and/or privatization are options to consider even if you are not using clinical assessments, the evidence available from major reviews of the available research certainly suggests that you will not improve the risk assessment system using clinical assessments, because actuarial risk assessments consistently “outperform” clinical risk assessment procedures (see Harris, this issue; and Gottfredson and Moriarty, this issue). However, we should point out that the new generation of “actuarial” risk assessment instruments currently being used in community corrections agencies–including the popular LSI-R instrument discussed in several articles in this issue–actually requires both objective and subjective (or clinical) assessments by line staff. In fact, the distinction between actuarial and clinical assessment is becoming blurred, with consequences for line community corrections personnel (and management) that are important to consider. It is our contention that if we continue further in this direction, then changes in either staff quality or in the privatization of the assessment function may be needed.

According to a recent review by Brumbaugh and Steffey (2005), three of every four probation and parole agencies in this country employ “objective” risk/needs instruments to classify offenders, using either the shorter Wisconsin risk/needs assessment instrument or the longer, 54-item LSI-R mentioned earlier. Both instruments require line staff to make both objective assessments (such as prior convictions, current employment) and subjective assessments (such as extent of drug problems, attitude, mental health). Not surprisingly, the results of a number of inter-rater reliability studies reveal that line community corrections staff are much more consistent in their scoring of objective than subjective items (see, e.g. Austin, this issue; Byrne and Robinson, 1991; and Harris, this issue).

The use of a large number of items in a risk instrument is likely to exacerbate the inter-rater reliability problem. Austin (this issue) for example, pointed out that (in an inter-rater reliability study he conducted) of the 54 items included in the LSI-R (37 yes/no items and 17 likert scale items), only 16 items had an agreement rate of 80 percent or higher, with 38 other items scoring in the 60-70 percent range. Overall, Austin found that these scoring differences on individual risk items resulted in disagreement on the scoring of the offender’s risk level in 29 percent of the 120 cases reviewed by the two staff members included in the inter-rater reliability test.

We can only speculate about how such differences in the scoring of individual risk items would affect risk assessment (and the classification of offenders into high-, medium-, and low-risk categories) across an entire department. However, Austin’s research certainly suggests that new strategies need to be developed to improve the level of inter-rater reliability before the agency embarks on the time-consuming risk assessment construction/validation process. Our recommendation is to collect data on fewer items, focusing primarily on objective items that are relatively easy to code. Austin (this issue) found that he could improve both the reliability and validity of the LSI-R by focusing on a subset of only 8 of the 54 original LSI-R items. According to Austin, “not only do these items have better predictive ability, but also they reduce the “high-risk” category” (this issue). Since most observers (see, e.g., Lowenkamp and Latessa, 2005) recommend providing the highest level of supervision and services to high-risk offenders, the cost effectiveness of a more precisely defined—and smaller—high-risk classification category should be obvious.

In addition to Austin’s research, the findings from other inter-rater reliability studies indicate that line staff characteristics (such as age, gender, race, location, experience) will likely affect the scoring of risk assessment instrument items in ways that are important to consider. For example, Byrne and Robinson (1992) identified gender bias as a potential problem affecting inter-rater reliability. In their study of inter-rater reliability among 130 probation officers, they distributed two different versions of a “case study”: in version A, the juvenile ( Sandy) was described as female; in version B, the juvenile was described as male. There were no other differences between the two “case studies.” Significant differences in overall risk scoring were identified, with the female version of the case receiving higher scores than the male version of the same case, resulting in a greater proportion of the female cases being classified as high risk (40.4 percent) than their “male” counterparts (33.1 percent).

We suspect that in addition to variations in scoring and consistency due to offender characteristics (such as gender, race, class), there will be variation in scoring and consistency due to the characteristics of the line probation/parole officers completing the assessment (see, e.g. Byrne and Robinson, 1990).The findings from both the Austin study and Byrne and Robinson study underscore the importance of conducting an inter-rater reliability study, not only to support initial risk instrument development, but also to examine the very real possibility that bias (related to both the characteristics of offenders and the characteristics of line staff) is having a detrimental effect on the risk assessment process.

One other area where clinical (subjective) judgment enters into the classification process is in the agency’s risk scoring “override” policy. While there will undoubtedly be circumstances where an offender will be either over-classified or under-classified by line staff members, and/or by management decisions to ignore risk altogether (such as offense exclusions for sex offenders), it is critical that the level, type, and circumstances of over-ride usage be monitored on an ongoing basis. A simple rule of thumb for this type of review is to apply a 10 percent rule: if more than 10 percent of the agency’s risk scoring decisions are being changed, then the agency has a problem in this area that needs to be resolved.

Finally, our discussion of the utilization of clinical and actuarial risk instruments in community corrections would be incomplete without mention of the “validation” problem. According to a recent study by Hubbard, Travis, and Latessa (2001), “only 30 percent of the agencies that use an assessment instrument reported that the instrument was validated for their local population” (as summarized by Brumbaugh and Steffey, 2005:59). Without the completion of the necessary validation research, there is no way of knowing whether the risk instrument being used by a particular agency results in an accurate classification of offenders into the low-, medium-, and high-risk categories used to allocate scarce probation and parole resources, both in terms of supervision and in terms of services.

While there certainly has been much discussion of the need to apply the “risk principle” (risk, needs, responsivity) to offenders supervised in the community corrections system (see, e.g. Andrews, et al., 1990, Latessa and Lowenkamp, 2005), it appears that the determination of “risk level” may be inaccurate for a significant number of these offenders. Recent attempts to improve the risk assessment process using actuarial instruments may have made matters worse, because of the inter-rater reliability problems associated with the more complex risk assessment instruments currently being used, such as the 54-item LSI-R. The implications of our brief review of the use of actuarial vs. clinical assessment are straightforward: in order to improve both reliability and validity, risk assessment instruments need to be designed using a small number of objective risk items and tested (for reliability and validity) on an ongoing basis.

We anticipate that the continued development of LSI-R type assessment instruments, along with the use of offender-specific assessment devices (for categories such as sex offender, mentally ill offender, substance abusing offender, multiple-problem offender) will require more qualified line staff and/or the use of private sector assessment centers. However, current line staffs in community corrections agencies are certainly qualified to classify the risk level of offenders using the simplified versions of the actuarial risk instruments advocated by Austin (this issue). The challenge for community corrections is to allocate resources to offenders placed in various risk classification levels in a manner that maximizes the system’s overall effectiveness. Getting the assessment “right” is the critical initial step, but it must be followed by improvements in treatment classification, and subsequent case planning strategies (see Taxman, this issue).

back to top

Issue 3: The Need to Combine Individual and Community Risk Assessment

A number of recent evidence-based reviews of the research in community corrections have identified statistically significant, but modest (10 percent) recidivism reduction effects associated with a variety of community treatment strategies (see Welsh and Farrington, 2006). We suspect that the recidivism reduction effects identified in these studies would be even more pronounced if individual-level assessments of risk were combined with community-level risk assessments (Byrne, 2006; Pattavina, Byrne, and Garcia, 2006), based on the premise that community-level risk assessment is a necessary “first step” in the community change process.

We offer this assessment based on two related factors: first, there is a large body of research supporting the notion that an individual’s risk of re-offending is affected —both positively and negatively—by the community in which he/she resides while under community supervision (Sampson and Bean, 2005; Sampson and Raudenbush, 2004; Pattavina, Byrne, and Garcia, 2006). Second, the treatment resources available to offenders will also likely vary by the “risk level” of the neighborhood, with higher-risk neighborhoods offering fewer (and lower quality) treatment options to offenders living in these areas (Jacobson, 2006). Accuracy of the individually-based risk classification system will likely improve with the inclusion of overall community risk level (high vs. low/medium risk, for example, based on offender density and/or the area’s crime rate), along with selected community “risk” characteristics (such as unemployment rate, proportion of residents living in poverty, size/characteristics of first generation immigrant population). Similarly, the accuracy of the individually-based treatment classification system (linking offenders at different risk levels to appropriate treatment) would also be improved by an assessment of community risk level, because this classification decision could be based on an assessment of the likely impact of community culture (such as attitudes toward substance use, criminal thinking, etc.) on the attitudes and behavior of offenders residing in “highrisk” and low/medium-risk neighborhoods (Sampson and Bean, 2005).

 back to top

Concluding comments

While there have been significant improvements in the individual offender assessment procedures used by community corrections agencies over the past two decades, our brief review suggests the following: 1) we need to conduct high quality experimental research on the effectiveness of both risk and treatment classification systems, using risk reduction as our primary outcome measure; 2) we need to consider simpler alternatives to both the general (e.g. LSI-R) and offender-specific (e.g. mentally ill, substance abuser, sex offender) risk assessment devices; and 3) we need to incorporate community-level risk factors into our current assessment system.