- The Need for Risk Assessment Validation
- The New Risk Assessment
- Better Risk Prediction
- More Accurate Risk Factors
- Redefining Risk
- Risk Suppression
- Independent Expert Review
- Next Steps
[The following article offers a descriptive overview of a new pretrial assessment instrument developed by the Pretrial Services Agency for the District of Columbia (PSA). As the implementation process moves forward and PSA compiles and analyzes data on the instrument, the authors plan to present more detailed information on the implementation process and data analysis.]back to top
The Need for Risk Assessment Validation
[The pretrial program’s] assessment and recommendations should be based on an explicit, objective, and consistent policy for evaluating risks and identifying appropriate release options. The information gathered in the pretrial services investigation should be demonstrably related to the purposes of the pretrial release decision and should include factors shown to be related to the risk of nonappearance or of threat to the safety of any person or the community and to selection of appropriate release conditions.
National Association of
Pretrial Services Agencies
Standards on Pretrial Release, Third Edition
TO MEET ITS mandate to promote pretrial justice and public safety, the Pretrial Services Agency for the District of Columbia (PSA) is committed to using a research-based risk assessment instrument to gauge each defendant’s potential risk of failure to appear (FTA) and rearrest while on pretrial release. Use of a research-based risk assessment tool helps the agency to ensure that its release and detention recommendations to the courts are most effective—but least restrictive—for the District of Columbia’s defendant population.
PSA has used some form of risk assessment since its inception in 1967—which represents the longest continuous use of risk instruments in the pretrial field. The Agency first used a “problem/solution” grid that matched factors believed to contribute to pretrial misconduct. For example, defendants eligible for pretrial release but with prior failures to appear could receive a recommendation for regular reporting to PSA and notification of upcoming court dates. In 2005, PSA adopted a point-based assessment instrument that combined existing research and literature in the pretrial and criminal justice fields with collective input from Agency management. This instrument identified 38 risk factors that were assumed to relate to likelihood of defendant failure to appear and rearrest (see Table 1 on page 29).
PSA’s vision of being a leader in the justice system1 fueled the Agency’s effort to develop and validate a new risk assessment instrument, strongly borrowing the best features of its previous risk assessment instrument. In 2009 PSA contracted with the Urban Institute (UI) and Maxarth Corporation to develop and validate its new risk assessment tool. Our goal was to create an instrument that improved our ability to 1) target supervision and treatment resources to defendants who, although released, present a greater probability of being rearrested while awaiting trial or missing a court appearance; 2) minimize resource investment on defendants that require less intervention based on risk; 3) account for the current and rapidly changing needs and issues facing its current defendant population; and 4) consider advances in high-risk defendant supervision such as electronic surveillance and targeted substance abuse treatment and mental health services. UI and Maxarth submitted the final risk assessment instrument and final report to PSA in April 2012. The result, we believe, is a risk assessment that greatly improves our ability to predict future misconduct, classify defendants into the appropriate levels of supervision, and target agency resources to best promote public safety and pretrial justice.back to top
The New Risk Assessment
The new risk assessment maintains the best features of the current tool—such as automatic calculation of separate failure to appear and rearrest risk levels, use of risk factor information routinely obtained during the PSA investigation, and internal quality control protocols—while also enhancing predictive ability. As with the current instrument, the new tool automatically calculates and scores risk factors as staff enter diagnostic information into PSA’s information management system. Many pretrial risk assessments require staff to calculate risk scores manually, which increases the potential for incorrect results. Automated computation also allows PSA to consider as many risk factors in the assessment as the research suggests. In fact, the new instrument examines nearly twice the number of risk factors as the current tool. Besides expanding the number of risk factors considered, the new instrument also assesses each defendant’s specific risk to commit new dangerous, violent, or domestic violence charges.
Another advantage of PSA’s new assessment tool is that it more accurately gauges a wider variety of pretrial misconduct. The benefit to PSA, its partner agencies, and the D.C. community is better matching of higher-risk defendants with appropriate levels of supervision, enhanced identification of defendants who could be released safely with no supervision or minimal monitoring, and better pretrial release and detention decision-making.
A final noteworthy feature of the new risk assessment instrument is that it will calculate risk models or different outcomes, including failure to appear, any rearrest, domestic violence and dangerous rearrest, and dangerous and/or violent rearrest. The outcome for each model will have a risk level and a risk score. The risk levels will correspond with the following categories: very low, low, medium, high, and very high and the scores will range from 0-100.back to top
Better Risk Prediction
Often, the key to an effective risk instrument is predictive validity—the degree to which the calculated risk score predicts whether or not the defendant will be involved in a future event or misconduct such as rearrest or fail to appear in court. Based on UI and Maxarth’s research, the proposed assessment has a 16 percent greater predictive accuracy than PSA’s current risk assessment in identifying defendants most likely to miss future court dates or to be rearrested. Although rearrests on dangerous or violent charges are rare within the local defendant population, the new assessment identified these events eight percent more accurately than the current assessment.back to top
More Accurate Risk Factors
The new assessment also improves content validity—how accurately risk factors used reflect defendants’ criminogenic risk. Agency staff and leadership were engaged throughout the developmental process. In its discussions with UI and Maxarth, PSA management identified variables to be considered in the research, based on staff’s clinical experience and recent risk assessment research in the pretrial field. In addition, the new assessment also nearly doubled the number of risk factors compared to our previous instrument (70 factors, up from 38 under the current assessment) and weighted them more accurately according to their empirical relationship to FTA and rearrest (See Table 2 below). Using 44,823 administrative records of defendant cases filed in the Superior Court for the District of Columbia and the U.S. District Court for the District of Columbia between October 2007 and August 2010, the team developed risk models based on five domains for risk factors: defendant characteristics (9 factors), prior criminal history (39 factors), instant offense types (14 factors), lockup drug tests (5 factors), and current criminal justice status (3 factors).
PSA also identified five pretrial misconduct outcomes that included failure to appear; any rearrest involving a new papered criminal charge or serious traffic offense; rearrest for a dangerous/violent/domestic violence offense charge; rearrest for a domestic violence charge; and persistent drug use. The combination of these outcomes and subgroups resulted in 11 risk prediction models and resulting scales that all predicted the probability of pretrial misconduct more accurately than the previous risk assessment instrument.
Consistent with findings from other pretrial risk assessment studies, the criminal history and current charge domains had the highest correlations to FTA and rearrest for any new criminal charge.2 The current charge domain also better predicted the risk of rearrest on a dangerous or violent charge. While significant, “dynamic” risk factors (those that might change during the course of the pretrial period), such as demographic and social information and current status with the justice system, were less predictive of pretrial misconduct. See Figure 1.back to top
The UI/Maxarth’s research design is the first in the pretrial field to include in the definition of “safety risk” (beyond rearrest on any new charge) new violent offenses, dangerous charges, or domestic-violence related crimes. As a result, the new risk assessment will help PSA to distinguish general and specific criminality risks and determine if certain defendants pose a greater risk of involvement in more serious crimes if released during the pretrial period.back to top
Many of the defendants studied under the risk assessment were on pretrial supervision. This previous supervision may have helped suppress the defendant’s risk of failure to appear and rearrest. However, supervision also may have minimized the true relationship of certain factors to pretrial risk. The UI/Maxarth research team developed a method that reduced the potential impact of “supervision suppression” across common supervision conditions. Specifically, they conducted analysis using the observed risk predictors and ensured that the covariance between these predictors and conditions of pretrial release or the extent to which one of these related factors may change and cause change in the other was eliminated. This ensured that pretrial misconduct would not be biased, whether or not risk suppression existed in the data, if the risk predictors were unrelated to the conditions of release. They also conducted modeling to determine the probability that each defendant would likely receive different supervision or release conditions, developed probability treatment weights, and then balanced the data while developing and validating risk assessment instruments. The result was a more accurate description of the relationship between risk factors and outcomes for pretrial defendants in the District of Columbia.back to top
Independent Expert Review
The risk assessment development and validation study included a thorough review of the design, methodology, analysis, and recommendations by an external, independent review panel composed of respected national experts in the field of pretrial and post-sentence risk assessment. The review panel critiqued UI/Maxarth’s research design and methodology and gauged whether the findings and recommendations were consistent with the data. After a comprehensive review, the panel confirmed the integrity of the research design and the final risk instrument. Individual panel members also made constructive and beneficial recommendations regarding implementation, staff training, risk assessment automation, and engaging stakeholders and partner agencies that will be included in our implementation phase.back to top
The new risk assessment continues PSA’s commitment to grounding its operations and practices in solid, evidence-based research. By more closely aligning release and detention recommendations with factors associated with failure to appear and rearrest, the new risk assessment will improve our ability to predict defendant misconduct and target supervision resources accordingly. The new assessment will also enable PSA to define and assess “risk” in different ways, further tailoring recommendations and supervision to specific types of potential misconduct.
Finally, the implementation phase will build in the capacity for PSA to test and retest the predictive accuracy of newly-identified variables for the assessment against failure to appear and rearrest. Specifically, the new risk assessment tests the predictive accuracy of risk factors by creating risk models and looking at the relative impact of the factors on different outcomes. It also looks at the impact of various predictive domains on the scores. The impact of the scores is computed separately for each predictor on each model (subgroup and outcome combinations).
The new risk assessment brings a greater degree of science and precision to PSA’s release and detention recommendations. The UI/Maxarth team employed a solid methodology in creating the assessment, incorporating the best of what we know from the criminal justice field and from previous risk assessment research. The team’s particular attention to risk suppression, weighting, and validation all enhanced the overall quality of the research and usefulness of the findings.back to top
In order to implement the new risk assessment, PSA convened a cross-functional project team consisting of representatives from the Office of Operations, the Court Services Program, the Supervision Program, the Treatment Program, and the Office of Strategic Development, along with the Office
of Information Technology. The implementation project team will facilitate the education of both internal and external stakeholders about the new instrument. It will also oversee the development of internal policies governing the use of the instrument throughout PSA operations and development of the necessary training for front-line staff.
In conjunction with the Agency’s Office of Information Technology, PSA’s risk assessment implementation team is developing functional requirements for the automation of the risk assessment instrument (RAI) to ensure that PSA’s information management system fully supports the new instrument. Risk factor calculation will continue to be automatic and transparent as staff perform routine investigative data entry. Continued automation will also allow PSA to consider additional risk factors in the assessment. Major milestones during the implementation phase will be:
- completion of required Pretrial Realtime Information System Manager (PRISM) updates and revisions to support the new instrument (PRISM is the agency’s web-based client and case management system);
- discussions with major stakeholders about the new assessment;
- completion of supporting operational procedures;
- staff training on the new instrument; and
- an impact review to gauge the new instrument’s effect on release and detention recommendations, assignments to supervision, supervision compliance rates, and rates of FTA and rearrest.
In addition, PSA will work with independent evaluators to determine the practicality of a separate risk screener to gauge risk throughout the supervision period and adjust case management levels accordingly.
For more information, please contact Spurgeon Kennedy at Spurgeon.Kennedy@psa.gov, 202-220-5654 or Michael Williams at Michael.Williams@psa.gov, 202-585-7035.back to top