Susan Turner
University of California, Irvine
Terry Fain
RAND Corporation
Background
Methods
Results
Discussion and Conclusions
RISK AND NEEDS assessment has been central to correctional operations for decades. Assessment not only helps predict offender future behavior, it can also help organizations allocate staff workload and resources. Before the late 1970s, judgments about offender risk were often subjective, based on experience or the intuition of correctional practitioners (Solomon & Camp, 1993). Objective systems began to appear in the 1970s and offered the promise of more efficient and systematic means of classification for offender risk and management than clinical intuition alone. The National Institute of Correction’s model Risk Classification initiative, undertaken in the early 1980s, introduced many jurisdictions to objective case classification (Jones, Johnson, Latessa, & Travis, 1999). Today, risk and classification tools are used in a myriad of criminal justice decisions—from pretrial release to parole supervision for both juvenile and adult populations. More recent “third generation” instruments include criminogenic needs of the offender that should be addressed in order to reduce recidivism (Bonta, 1996).
One of the most critical issues for assessment instruments is their predictive validity. An instrument should be able to accurately predict which offenders will and will not recidivate. Whether an instrument is selected from a number of commercially available products (such as the Level of Service Inventory and Correctional Offender Management Profiling for Alternative Sanctions) or developed by a jurisdiction, it should be validated on the local population. The current article discusses the validation of the San Diego Risk and Resiliency Checkup on a sample of juvenile offenders in Los Angeles County.
Background
Although the Los Angeles County Probation Department routinely gathered background information on youths entering its juvenile system, no validated risk assessment was being used through the early 2000s. As part of a court settlement regarding services provided to minority youth in the county, the department was required to allocate resources for the administration of a validated risk and needs instrument to its juvenile probationers. Of particular importance was that the instrument work well for youths of all ethnicities.
Working with a committee representing the parties of the court settlement, researchers assisted in identifying and eventually validating a risk assessment instrument to be used in the county. After surveying instruments currently in use in the United States, we determined that items used in risk and needs instruments generally fell into one of nine conceptual categories: prior and current offenses/dispositions, family circumstances/parenting, education, employment, peer relations, substance abuse, leisure/recreation, personality/behavior, and attitudes/orientation. However, many of the instruments that we found in use had not been validated on the populations to whom they were administered, so that we were unable to determine their effectiveness in distinguishing high-risk youths from low-risk youths.
We identified three instruments that had undergone validation: the Youth Level of Service Inventory (YLSI) (Multi-Health Systems Inc., 1998), the San Diego Risk and Resiliency Checkup (SDRRC) (Little, n.d.), and the Washington Association of Juvenile Court Administrators Risk Assessment (WSJCA-RA) (Washington State Institute for Public Policy, 2004). Each includes multiple items for the conceptual categories we identified, and each offered advantages and disadvantages when compared to the others. The Department favored the SDRRC, primarily because it could be administered during the intake process. It also preferred the SDRRC’s emphasis on positive (“protective”) factors, whereas most risk and needs assessment instruments primarily focus on risk factors. The remaining settlement parties agreed, and the SDRRC was selected as the instrument to be tested.
The San Diego Risk and Resiliency Checkup
The SDRRC consists of 60 items in six conceptual categories, half of which are risk factors and half protective factors. The conceptual categories are delinquency, education, family, peer relations, substance use, and individual factors. Each conceptual category includes five protective factors and five risk factors. Each item is scored as “yes,” “no,” or “somewhat.” Scores from the risk and protective subscales are combined into a single resiliency score. The SDRRC also includes additional protective factors and additional risk factors that are not included in the resiliency score, but which may be used to tailor an individual’s supervision. A copy of the SDRRC instrument is included in the Appendix.
One important difference between the SDRRC and most other risk and needs instruments is that a higher score on the SDRRC implies higher resiliency, i.e., a lower score corresponds to a higher risk of re-offending. Most risk and needs instruments, by contrast, associate high scores with high risk of recidivism. The SDRRC does not contain any preset cut-points for youth risk levels.
The one existing validation study of the SDRRC was performed by Little (n.d.). This study included 2,633 youths surveyed in San Diego between February 5, 1999, and March 28, 2001. The SDRRC was found to be effective in predicting future offenses (Little, n.d.). The total resiliency profile appeared superior to either of the total risk and total protective scales. The correlation between the total resiliency profile and occurrence of a subsequent offense was -0.146 (p<.001). Using a logistic regression model to predict follow-up offenses, Little also found age, gender, ethnicity, and prior criminal history, as well as resiliency score, to be significant predictors of re-offending (Little, n.d.).
Methods
Selecting the Sample
We wanted to assure adequate statistical power for detecting differences in recidivism rates between low-, moderate-, and high-risk youths, as well as differences between groups defined by race/ethnicity and gender. Because the SDRRC does not have any preset risk cut-points, the pilot study proposed to divide the sample into approximate thirds defining low-, moderate-, and high-risk groups. The probability of detecting a difference in recidivism rates between the three risk groups depends upon the number of groups (in this case, three); the sample size of each group, and the spread of the true rates of recidivism. Because we did not know the true rates of recidivism for the different risk groups, we proposed three plausible “true” scenarios for the probability of rearrest at 6 months for low-, moderate- and high-risk youths.
The three were:
| Low | Moderate | High | |
| Scenario 1: | 11% | 21% | 27% |
| Scenario 2: | 12% | 25% | 32% |
| Scenario 3: | 12% | 18% | 32% |
With these three scenarios, 1 we determined that at least 120 to 140 youths in each risk level would need to be included in order to be able to detect differences. However, we also wanted to be able to detect differences for key subgroups: boys as well as girls; and for blacks, Hispanics, and white/other youths. Each of the subgroups of interest needed between 100 and 120 youths within low, moderate, and high-risk groups for adequate power. Therefore, we needed approximately 300 to 400 of each gender and each race/ethnic group.
Our final sample size target was 1200 youths for the study. This included 800 males and 400 females, and 400 each of whites, blacks, Hispanics. Because probation officers assess youths in both court- and non-court venues, we designated approximately 800 court cases and 400 non-court cases. 2
Table 1 shows the full stratified target sample, with the size of the sample in each cell.
Four area offices were chosen for the assessment in order to provide county geographical representation. 3 These were Long Beach/Harbor (South), Pomona (East), Centinela (West), and Van Nuys (North). Each area office was to supply one-quarter of the target sample assessments.
Training
Probation officers volunteered for the assessment pilot. Originally 18 Deputy Probation Officers—14 field and 4 Camp Community Transition Program (CCTP)—were trained in the administration of the SDRRC. The three-day training consisted of an overview of the instrument; its application and practice; overview of evidence-based practice, including the overview of the six criminogenic needs and the eight guiding principles for risk/recidivism reduction; motivational interviewing techniques; and the actual administration of the tool. Training was conducted by staff from Justice System Assessment & Training, the firm that developed the SDRRC. Deputy Probation Officers (DPOs) were provided with an incentive of 30 minutes overtime payment for each assessment completed during the pilot.
Data Collection
Data were collected in three general areas: assessment scores, services received, and recidivism.
Assessment Scores. DPOs administered the assessments to youths. Information on each of the additional risk and protective factors that do not contribute to the overall resiliency score was also recorded. The assessment form also includes demographic variables (age, gender, ethnicity), information about proficiency in English, and criminal history.
Assessments were conducted from December 6, 2002, through October 30, 2003. A total of 1,165 youths were assessed by Los Angeles County probation officers. We also gathered information on whether the youth’s case proceeded to supervision or ended at investigation (no further probation supervision).
Recidivism. Using the Probation Department’s databases, we obtained information on arrests during the 12 months after assessment for each subject. These data include both juvenile and adult arrests. Date of each arrest, charges, and disposition were recorded. We also used records from juvenile halls and juvenile camps to determine how many days a given youth was incarcerated during the 12 months after assessment.
We were unable to determine whether a given youth was rearrested during the follow-up period for 129 (11.1 percent) of the 1165 youths originally assessed for the study. Our final sample is 1,036 youths. Missing data were primarily due to incomplete disposition records, so we were unable to determine whether some youths were in custody (and therefore incapable of being rearrested). We found no significant differences on gender or age between the deleted cases and those in the final sample. Significantly more Hispanics, and significantly fewer blacks, were in the final sample than among the deleted cases (p < .001).
Weighting the Final Sample to Reflect Population of Probation Youths in Los Angeles
Our final sample did not exactly match the target sample presented in Table 1. In particular, the final sample included somewhat more males, more Hispanics, and more court cases than we had originally targeted. By weighting our final analysis sample to represent the entire population of investigation and supervision cases for Los Angeles (as described below), we have adjusted for differences between the targeted sample and final sample, so that our analyses do accurately represent the gender and ethnic mix among all Los Angeles cases.
In order to weight the final sample, we obtained the frequency of all youth investigation and supervision cases for Los Angeles during the same time period as the pilot assessment, with information on youth gender, race/ethnicity, 4 and court vs. non-court case type (see Appendix [PDF, 3.33MB]). Within each combination of gender, race/ethnicity, and court vs. non-court case type, we defined a weight to be the ratio of youths in the probation population to the number of youths in the final sample. This allowed us to weight the data to reflect the entire population on these characteristics. 5 All analyses were conducted on the weighted final sample.
Results
Mean Differences in Resiliency by Demographic Characteristics
SDRRC resiliency scores differed by gender, age, and ethnicity; some differences were large enough to be statistically significant (see Table 3). The most pronounced differences were for different ethnic groups, with pair-wise comparisons of whites, blacks, Hispanics, and “other” race all producing significant differences. “Other” youths (primarily Asians) had the highest mean resiliency scores, followed by whites, blacks, and Hispanics, respectively. There were also marginally significant differences between males and females (t = 1.89, p < .06) and between youths aged 15 or 16 and those aged 17 or 18 (t = -1.93, p < .06).
Characteristics of the SDRRC
The SDRRC comprises two subscales: the protective subscale and the risk subscale. Each of these contains subscales for delinquency, education, family, peer, substance use, and individual. Table 4 shows the correlations between the overall resiliency score and the individual subscales. Table 5 gives the correlations among the subscales. It is important to note that the total SDRRC scale reflects “resiliency.” Resiliency is defined as the net sum of protective and risk factors. Protective and risk factors are scored differently. The higher the protective score, the more protective factors the youth has. Risk scores have negative values; the more negative the value, the higher the risk. Thus we would expect positive correlations between the total resiliency score and 1) total risk score, 2) total protective score, and 3) the subcomponents of both risk and protective scales. In fact, that is what we see in Table 4. At the same time, however, we see fairly high correlations between individual subscale items (see Table 5), suggesting that they may be redundant. Redundancy among the subscales of the resiliency score was also reported by Little (n.d.) in her analysis of the SDRRC.
Relationship Between Resiliency and Recidivism as Measured by Subsequent Arrest
For each of the youths assessed, we determined whether the youth was arrested within the 12 months following the administration of the assessment. The major question for the validation study is whether scores obtained on the SDRRC are related to subsequent recidivism.
One of the issues for recidivism studies is whether or not subjects are “at risk” to reoffend. Individuals may be removed from the sample before they have a chance to reoffend—they may be sentenced to terms of incarceration during the entire follow-up period. In some cases, these individuals may be excluded from analyses, or they may be treated as censored observations. In order to determine how large a problem this might pose for the current study, we calculated the number of days youths were “on the street” from the point of their assessment until 12 months later. The vast majority of youths (over 90 percent) had at least 10 months of street time. For the remaining youths, analyses revealed that even those with very minimal “street time” (less than 2 months) were arrested. For this reason, we did not exclude any youths from our analyses of recidivism.
Table 6 presents the recidivism results for the full sample. For this and other analyses, we divided the sample into approximate thirds and categorized the resulting groups as “low” (those with score 12 or less), 6 “medium” (those with scores between 13 and 33), and “high” (those with score of 34 or higher). Table 6 shows that the scale does validate for the overall sample. Only 8 percent of “high” resiliency youths were arrested, compared with almost 36 percent of those with “low” resiliency.
Subgroup Analyses
Figures 1, 2, and 3 present the results by age, gender, and ethnicity, respectively. Within each of the major racial groups, the resiliency score is significantly related to recidivism. Regardless of ethnicity, the higher the resiliency, the lower the likelihood of arrest for youths. The same holds true for males and females, and across all ages. The discriminatory power of the instrument appears to be greatest for the younger youths in the sample (age less than 15), most likely due to more variability in outcomes among younger juveniles.
Assessing Scale Properties and Recidivism
Prior analyses have examined the relationship between the total resiliency score and rearrest. In Table 7 below, we present the relationship of individual subscales to rearrest. Recall that the more negative the risk score, the higher the risk. Thus we would expect a negative correlation between risk subscales and rearrest. All subscales correlate significantly with rearrest. The absolute correlation between the total resiliency score and rearrest is 0.27—similar to the correlation observed by the Washington State Institute for Public Policy (2004) for misdemeanor and felony recidivism for the Washington Pre-Screen Assessment inventory. Interestingly, it is higher than the correlation reported by Little (n.d.). Resiliency scores have a higher correlation than do their respective protective and risk subscales with only one exception (family protective factors).
Controls for Additional Factors Related to Recidivism
Earlier analyses have examined the univariate relationship between SDRRC score and recidivism. In the following analyses, we examine the relationship controlling for additional factors that may impact how well SDRRC predicts recidivism. These factors include age, gender, and race/ethnicity, as well as whether the case is supervision (vs. non-supervision) and court (vs. non-court).
Table 8 presents the results from a logistic regression analysis of the total sample. We see that, even controlling for other factors that might be related to recidivism, SDRRC resiliency is still significantly related to rearrest. Other factors are also related to rearrest: age (not being in the youngest or oldest age group 7), being male (as opposed to being female), being black (as opposed to being white), and being under probation supervision during the 12-month follow-up period. The overall measure of the model yielded a Wald chi-square value of 102.1 (p < .0001).
The relatively lower correlations between SDRRC items and rearrest for Hispanic and “other” youths observed in Table 4 might suggest that the resiliency measure is not as strong a predictor for some groups as it is for others. In order to test this, we included interactions terms between race/ethnicity and resiliency in the model identified in Table 8 above. Results, shown in Table 9 below, confirm that resiliency is differentially related to recidivism for whites (compared with Hispanics), although not significantly for blacks or “other” youths.
One of the questions we want to answer is whether the provision of services influences the youth’s recidivism. We would expect those receiving services might have lower recidivism rates. In order to evaluate this possibility, we tested the multiple logistic regression model presented above, with the inclusion of the number of services received by youths. Results of this regression showed that the number of services was positively correlated with recidivism. In other words, the more services received, the more likely the youth was to have an arrest during the follow-up period. This is most likely due to the fact that higher-risk youths are provided more services. In fact, the correlation between SDRRC resiliency and the number of services was -0.19 (p < .0001). We conducted supplemental analyses in which we divided the sample into low-, moderate-, and high-risk groups and performed the regression runs within each risk group. Results showed no significant relationship between the number of services and recidivism once youth resiliency was controlled for.
Discussion and Conclusions
Our analyses showed that the SDRRC has both internal and predictive validity for youth in the Los Angeles County Probation system. Total resiliency scores are correlated with total risk and total protective scores; subscales within risk and protective scores are significantly correlated with their overall scales. Subscales are often highly correlated with each other, however, suggesting a degree of redundancy in the instrument. The instrument and its subscales were significantly related to arrest for youths 12 months after their assessment. The scale was also significantly related to recidivism for major subgroups of interest: youths of different ethnicities, as well as both males and females. In analyses which took into account other factors related to recidivism, the SDRRC remained a significant predictor of subsequent arrest. However, the scale does seem to work differently for some youths. In particular, the scale is not as strong a predictor for Hispanic youths as for other youths.
Limitations of Current Research
Research studies are subject to limitations, and this one is no exception. Our follow-up was limited to 12 months following youth assessment with the SDRRC. Although this provides a window of time over which to observe behavior, longer follow-up time periods are preferable. Initially, a longer follow-up period had been planned, but the assessment phase took longer than expected.
As with many recidivism studies, our study relies on official records for measurement of youth behavior. We did not have access to youth’s self-reported criminal behavior, which can provide a more direct measure of criminal behavior (only a fraction of offenses result in arrest). Future research may want to examine the extent to which the SDRRC also corresponds with self-reported criminal behavior. To our advantage, however, the pilot test was conducted before the SDRRC was implemented. In this way, the validity testing was not contaminated by any system policies or practices that were based on classifications by the SDRRC.
As indicated earlier, the SDRRC does not have any predetermined cut-points for resiliency. Without cut-points for classification, we could not conduct any meaningful analyses of false positives and false negatives—or the extent to which errors in prediction are made when using the SDRRC. Cut-points will be determined during the implementation phase of the instrument in Los Angeles. We recommend that sensitivity analyses be part of continued monitoring of the instrument once it has been integrated into Probation practices (as described below).
In addition, more thorough examination needs to be conducted on differences in the scales and subscales for different subgroups of youths. This should also be part of continued monitoring of the instrument.
Systemwide Implementation of LARRC
In summer of 2004, the Los Angeles County Probation Department started the process to institutionalize the SDRRC, now referred to as the LARRC. Training on LARRC began on August 4, 2004. In December 2004, staff began completing the LARRC assessment utilizing an automated system.
The Los Angeles County Probation Department has started a policy that requires all DPOs in the Juvenile Bureaus to assess and reassess minors assigned to their caseloads at defined intervals as part of a plan to enhance case management services. As investigators are trained in the administration of the LARRC, the assessment will be utilized at the investigation level (the point at which the pilot assessment was done) and will continue through the supervision stages in order to address protective/risk/resiliency factors, update case planning efforts, and link minors to appropriate services and interventions.