Kimberly Gentry Sperber, Talbert House
Martha Henderson-Hurley, Ohio Department of Rehabilitation and Corrections
Dena Hanley, University of Akron
Guidelines for Academics
Guidelines for Practitioners
Conclusions
RECENTLY, GENDREAU, French, and Gionet (2004) lamented the state of correctional treatment, stating that the majority of treatment programs examined to date have been “sadly lacking in therapeutic integrity” (p.28). They are not alone in this criticism. Latessa, Cullen, and Gendreau (2002) have even gone so far as to accuse many treatment programs of practicing “correctional quackery.” That so many correctional treatment programs are still in this state after years of research into “what works” baffles many researchers. Clearly, there is a gap between correctional research and correctional practice.
Why this disconnect between research and practice continues to exist is a topic for debate. Much of the discussion in this area, however, has focused on what academics believe to be the responsibilities of treatment practitioners and administrators in moving programs toward better alignment with the “what works” findings. While there has been some modest acknowledgement that academics could do more to assist practitioners with implementing and maintaining evidence-based practices, recommendations directed at researchers seem to be aimed at simply doing more of what they have traditionally done—evaluate programs to better explicate the differences between programs that work and programs that do not work (e.g., Cohn, 2002; Latessa et al., 2002). There has been little (if any) exploration into what practitioners need from the academic community in order to operationalize evidence-based practices, yet the field will not continue to advance until researchers and practitioners form an alliance that fosters mutual goals, mutual accountability, and mutual respect.
To that end, this paper challenges academics and practitioners alike to step out of their comforzones and to honestly assess their roles and responsibilities for moving the field forward. This means that some academics will need to move from passive critic to active collaborator, while some practitioners will need to take responsibility for knowing “what works” and for dedicating existing resources to ensuring the delivery of evidence-based practices. To further explore the realities of collaboration, the authors draw on their collective experiences as both academics and practitioners to provide guidance to academics and practitioners alike. We begin with the responsibilities of academics.
Guidelines for Academics
Academics are called to do more to help bridge the gap between research and practice. To bridge this gap, academics must be able to think beyond traditional research methodology. In the traditional research practice, the academic comes into the agency and decides the research design, analysis, and impact. However, a more effective approach calls for the collaboration of researchers and practitioners to decide these issues together.
Strengthening Partnerships with Programs
Academics often charge that practitioners do not understand the research process and struggle to interpret research findings; yet, researchers have done little to help improve practitioner abilities. Similarly, academics often lack a foundational understanding of program operations and the implications for research projects. One suggestion for improving understanding on both sides is to approach collaborations as true partnerships. One way of strengthening partnerships with programs is to include program staff in the development of the evaluation plan. There are several advantages to this approach. First, this inclusion gives program staff the opportunity to educate the researchers about the program, including subtle nuances that might be important but not easily understood if the researchers simply rely on a general overview of the program. For example, most programs claim to use a cognitive-behavioral approach, yet these approaches often differ in fundamental ways. A second advantage is that program staff can help inform the researchers about the resources needed to collect the desired data and can help work out the logistics associated with data collection. Finally, allowing program staff to have input into the way the data will be collected helps to increase their understanding and buy-in, which may result in a smoother evaluation process.
Another way to strengthen partnerships with practitioners is to understand that practitioners do not view a program evaluation as merely an academic endeavor into applied research later to be translated into another line on a vita. Engaging in an evaluation needs to be a meaningful and practical experience. This means that the process and the results of the evaluation have to be useful to the program. Researchers can help achieve this goal by ensuring that implications from the study are operationalized into concrete suggestions for improvement. For example, instructing an adolescent girls’ program to “make sure the program celebrates women’s and girls’ ways of being in the world” is neither concrete nor helpful to a program attempting to enhance its implementation of evidence-based practices.
Providing Basic Research Guidelines
A second way that academics can help practitioners is to provide them with some basic research guidelines. In order for practitioners to participate directly in the research process, which is necessary for advancing evidence-based practices, practitioners must have a working knowledge of research methods and evaluation (Buysse, Sparkman, and Wesley, 2003). Academics can help by providing comprehensive research guidelines written in a manner that is understandable to those providing direct services within corrections. These guidelines should provide guidance on quantitative as well as qualitative methodology. At a minimum, any research guidelines developed for practitioners should include three sections.
The first section should provide a “how-to” guide on accessing relevant research literature. Practitioners tend to rely on trade magazines and newsletters for programmatic updates. Such sources are often limited in depth and breadth. A consequence of this is that practitioners who want to know more often do not know where to look to find the best research literature, nor are they versed in how to evaluate basic research designs.
The second section should provide coverage on how to evaluate quantitative research without being a statistician. This section should cover the basic logic of quantitative research methodology in laymen terms. At a minimum this would include: 1) basic coverage on how validation studies operate and why they are important; 2) a discussion of reliability and validity and why they are important to practitioners; 3) a basic discussion of statistical significance; and 4) coverage of the evaluation study process.
Section three should discuss qualitative research methodology. Qualitative research often provides the context for quantitative research and reveals nuances that numeric data simply cannot. This section should describe basic qualitative research methodology. The focus of this section is on the techniques used in qualitative research, such as how to identify stakeholders and how to conduct focus groups for qualitative research. The section should also include information on how to evaluate qualitative research findings. With these types of guidelines, practitioners could be more competent consumers of research studies and their findings. This is assuming that they have simple access to such findings, however.
Dissemination of Research and Evaluation
A third way that academics can strengthen partnerships with practitioners is to assess the utility of current methods for information dissemination to practitioners. Academics tend to view publication in tier-one peer-reviewed journals as the primary goal of research. A consequence of this mentality is that often what drives research in corrections is not whether changes in practice are necessary for improvement but rather what research is publishable in a top journal. Academics must begin to view changes or improvements in practice as a primary goal of research, with publication as one step toward that goal (Walshe and Rundall, 2001).
The push for publication primarily in tier-one peer-reviewed journals and textbooks by academics also means that few practitioners are likely to have access to research findings. Walshe and Rundall (2001) discuss two primary modes of access to research findings: the “pull” method” and the “push” method. The “pull” method, most often used by academics, is reactive. Researchers wait until a clinician contacts them seeking information or assume that simply providing access through libraries, journals, and databases will be enough for their message to get out. Given agency budgetary constraints, the “pull” method is likely to be unsuccessful in disseminating information. In contrast, the “push” method is a proactive process whereby researchers directly deliver research findings to practitioners. Examples of “push” methods include sending copies of research reports directly to agencies, providing training within agencies, or attending practitioner conferences and presenting research findings. An examination of who attends practitioner-oriented conferences would most likely reveal that only a small number of academics attend and present on current research and practice topics. Thus, an important opportunity to improve programming in corrections is missed by most researchers.
Guidelines for Practitioners
There is a clear continuum of quality (or lack thereof) in correctional treatment programs. On one end of the continuum are those programs that clearly are effective at reducing recidivism—those programs comprising the evidence for “what works”—and on the other end of the continuum are those programs that are clearly not effective and that willfully reject the “what works” literature—the correctional “quacks.” It is to those programs in between the two extremes that this paper is directed, for the authors are in agreement with Cohn (2004) and his assertion that many program administrators do see research as valuable; they are simply challenged with understanding and using it. This should not be a surprise, as this is typically not within their realm of expertise.
Role of Research and Evaluation within Programs and Agencies
Practitioners often view research language as foreign or at least difficult to understand. Equating steps in research methodology to similar steps in clinical practice could help to demystify the research process for staff (Raines, 2004). For example, practitioners are often quick to identify problems with a client’s functioning by assessing the client’s status and then comparing this to the client’s baseline data. This process is similar to the development of a problem statement for a research project (Raines, 2004). The comparison of these two processes places research methodology, at least partially, in terms that are understandable to practitioners.
Even more familiar to most clinical staff is the idea of treatment planning and selection of appropriate interventions. A client’s intervention plan can be compared to the quantitative or qualitative design of a research project. While a treatment plan outlines goals, objectives, and methods to be used for a specific client to achieve a desired outcome, research methodology outlines the goals, objectives, and methods to be used for a research project to achieve a desired outcome (Raines, 2004). Similarly, just as a treatment plan calls for specific interventions with a client (e.g., group counseling), so does an evaluation plan. The “intervention” phase of a research plan simply equates to the data collection activities involved in the evaluation (Raines, 2004). Finally, practitioners must evaluate and document whether the interventions that have been used with a client have been effective; this is done by assessing whether the client has shown progress on his or her treatment goals and is documented in the client’s chart. Evaluators, on the other hand, demonstrate effectiveness through data analysis and document such in research reports or journal articles (Raines, 2004). Continual exploration by practitioners into the overlap between the role of a practitioner and the role of an evaluator can help to consistently chip away at barriers to understanding research.
Role of Continuous Quality Improvement
While many agencies are not able to conduct regular evaluations of their programs, they can monitor the quality of their programs (Latessa et al., 2002). This is especially important in light of Lowenkamp and Latessa’s (2002) finding that programs that monitor quality tend to have lower recidivism rates. Through utilization of continuous quality improvement (CQI) principles and strategies, programs can systematically use data to assess processes and treatment outcomes (Mabry, Sperber, and Atkins, 2003). To do this, however, program staff must first have an understanding of how CQI differs from traditional quality assurance (QA).
Many program staff are familiar with traditional quality assurance activities. QA approaches tend to be retrospective and merely emphasize compliance with standards and regulations (JCAHO, 1994). Often these standards are not directly related to the efficacy of programming, especially in terms of reducing recidivism. In the field of corrections in particular, standards and regulations are most often centered on facility sanitation and security. Programs are rarely audited for their adherence to evidence-based practices and the extent to which they address the criminogenic needs of their clients.
CQI, on the other hand, is a prospective process that emphasizes the systematic collection and use of data aimed at continually improving program processes and client outcomes (Mabry et al., 2003). While this sounds conceptually straightforward, correctional agencies often have little experience operationalizing a strong CQI program. First, CQI represents a cultural shift for many correctional agencies. Second, agency staff often do not know where to begin when searching for appropriate measures of program or client performance. Finally, program staff often do not know how to use the data once they have gathered it.
Creating a Learning Culture. As previously mentioned, many correctional agencies are used to operating under a traditional QA approach. Because the focus is primarily on compliance, agencies often end up engaging in hide and seek behaviors to avoid putting problems out in the open. Under a CQI culture, staff are expected to identify problems and to share that information with all necessary parties. This requires a safe environment with no fear of punishment from administration.
In order to create an environment where all staff feel safe disclosing problems, some minimum requirements need to be met. The most basic requirement is that the board of trustees and the administration fully support this process. This support then needs to be communicated to the line staff. The focus on quality and continual learning has to infuse all levels of the organization (Mabry et al., 2003).
Choosing Appropriate Measures. When choosing measures of program or client performance, the agency should identify both process measures and outcome measures. Examples of process measures include such indicators as the number of case management contacts with clients, length of stay, and number of substance abuse groups attended. Examples of outcome indicators include such measures as changes in antisocial thinking, risk to re-offend, employment, and recidivism.
Tracking outcome indicators often involves the use of pre- and post-tests with clients. Practitioners typically have little training in choosing these instruments, however. Consequently, there are at least three criteria that should be met when choosing an instrument (Mabry et al., 2003). The first criterion is that staff should not create their own scales. They should instead seek out existing tools that have already been shown to be reliable and valid. The second criterion is that staff should seek out low-cost or no-cost instruments, given the limited resources agencies have to dedicate to data collection. The third criterion is that staff should attempt to locate short, self-report instruments. Again, this should help minimize the costs involved by limiting staff time involved in data collection efforts.
While these guidelines represent a good start for practitioners, we must acknowledge that they often lack the expertise to select instruments that best fit their services as well as their budgets (Mabry et al., 2003). One potential solution for this is to develop a resource manual for staff that explores various assessment tools. The manual should include a list of available instruments with corresponding reliability and validity data as well as cost data. The manual could also include a directory of online resources for assessment instruments and full-text research articles.
Using Data for Action Planning. With knowledge comes responsibility. This means that agencies should not collect data if they are not prepared to act upon the findings. Under a CQI model, programs are required to create a plan of corrective action (or an action plan) when they are not performing at the expected thresholds on certain measures (JCAHO, 1994). While the phrase “action plan” may be new for correctional staff, the task of creating action plans is not. To illustrate, action planning is simply a process for aligning goals with actions to achieve specified results. There are several examples of documents in agencies that meet this definition; these include client service plans, staff development plans, and agency strategic plans.
Regardless of the type of action plan to be created, the plan should meet five criteria. First, staff need to describe the actions to be taken in concrete terms. Second, they need to choose action steps that are feasible within the program structure and resources. Third, they need to establish a deadline for completion of the action plan. Fourth, the action steps need to be measurable. There needs to be a clear, objective method for determining whether the action steps were in fact accomplished. Finally, the staff with the most knowledge about the problem should be the ones guiding the action plan rather than choosing people based solely on title or role (e.g., the program manager). The result of this is an approach that empowers line staff to engage in problem-solving in a way that uses data to inform operational decision-making (Mabry et al., 2003).
Role of Staff Supervision
The role of staff supervision is often neglected in discussions of monitoring adherence to the principles of effective interventions. To illustrate, the first response to challenges in implementation of a new or enhanced practice is to hire an external consultant (often an academic) to come in and train the staff. If implementation continues to falter, agencies will often opt for more training rather than examine whether the staff are being supervised appropriately. Given Sexton’s (2003) finding that staff competence is directly correlated with reductions in recidivism, correctional agencies have a responsibility to ensure the competence of their staff. This can be accomplished through use of a systematic approach to supervision.
When discussing the appropriate structure for staff supervision, the first issue to be addressed is to whom the supervision is directed. It is our experience that conversations about staff supervision often center on clinical staff only rather than on all direct ser vice staff. This not only ignores a large sector of the staff—primarily security staff—it also demonstrates limited understanding of applying evidence-based practices. These practices are not only relevant to the group and individual sessions conducted by the clinical staff; they are also relevant to general milieu management. The reality is that non-clinical staff spend significantly more time with clients than do the clinical staff. If the treatment approach does not infiltrate the milieu, the program has not fully implemented the treatment approach. It is for this reason that any discussions of staff supervision need to incorporate all direct service staff.
The second issue to be addressed is the type of data to be used in assessing staff competence. Staff evaluations have traditionally relied heavily on whether staff have completed the necessary tasks of their jobs. Examples of these tasks include documentation, head counts, monitoring medications, and checking facility safety. It is less common to assess staff on their ability to effectively utilize evidence-based techniques or practices. Examples of such behavioral skills might include treating clients with respect, avoiding power struggles with clients or other staff, modeling pro-social behavior, and helping clients to identify thinking errors and appropriate replacements.
The third issue to be addressed is the use of the evaluation data. For example, a common practice across agencies is to formally evaluate staff only on an annual basis. In between these formal evaluations, they may or may not receive feedback on their skills. A better approach is to continuously collect data on staff performance that can then be used in two ways. First, the data can be shared with individual staff members during formal supervision meetings to highlight their strengths and weaknesses. Staff who are not performing to expectations can then collaborate with their supervisors to create a staff development plan on how to improve. Once this plan has been implemented, data can be collected again to assess for improvement. If done within a culture of learning, this can be empowering for staff. Second, the data can be examined at the aggregate level to assess trends in skill strengths and deficiencies. The data can then be used to inform decisions about further training (rather than relying on training as a reflex).
Conclusions
Criticism from researchers aside, many correctional programs now exist in a political and fiscal environment that requires that programs demonstrate effectiveness to sustain their funding. This is creating challenges for program administrators who have limited fiscal and human resources to gather and interpret this proof. While this paradigm shift is often attributed to—or sometimes blamed on —the “what works” literature, it is this same literature that can help programs to survive the change and to improve the lives of their clients as well. This is only true, however, to the extent that academics and practitioners can come together in a mutually respectful and accountable partnership.
Many practitioners have an open mind regarding research, but they also require assistance. Academics must respond to these needs by assisting staff in understanding methodologies, selecting reliable and valid tools, and supporting the implementation of results even after the final report is written. Similarly, program administrators must assist in the implementation of evidence-based practice by not only using words such as “evidence-based” and “research,” but by incorporating these efforts into the culture of the programs. This includes research and evaluation, CQI, and effective staff supervision. The bottom line is that practitioners and researchers need to adopt a shared vision where both parties are responsible for expanding the knowledge base of “what works” and for transforming the field into one of evidence-based practice. It is time for action.