Judging Risk

Risk assessment plays an increasingly pervasive role in criminal justice in the United States at all stages of the process—from policing to pretrial detention, sentencing, corrections, and parole. As efforts to reduce mass incarceration have led to the adoption of risk-assessment tools, critics have begun to ask whether various instruments in use are valid, and whether they might reinforce rather than reduce bias in the criminal justice system. Such questions, however, have largely neglected how decision-makers use risk assessment in practice. In this Article, we explore the judging of risk assessment and why decision-makers so often fail to consistently use quantitative risk assessment tools.

We present the results of a novel set of studies of both judicial decision-making and attitudes towards risk assessment. In our first study, we find that even in Virginia, whose risk assessment instrument has been hailed as a national model for the use of risk assessment, sentencing data indicates that judicial use of risk assessment is highly variable. In our second study, the first comprehensive survey of its kind, we also find quite divided judicial attitudes towards risk assessment in sentencing practice. Even if, in theory, an instrument can better sort offenders in less need of jail or prison, in practice, decision-makers may not use it as intended.

Still more fundamentally, in criminal justice, unlike in other areas of the law, one does not have detailed regulations concerning the use of risk assessment that specify the content of assessment criteria, the peer review process, and standards for judicial review. We make recommendations for how to better convey risk assessment information to judges and other decision-makers, and how to structure that decision-making based on common assumptions and goals. We argue that judges and lawmakers must revisit the use of risk assessment in practice. We conclude by setting out a roadmap for the use of risk information in criminal justice. Unless judges and lawmakers regulate the judging of risk assessment, the risk-assessment revolution in criminal justice will not succeed in addressing mass incarceration.

Table of Contents Show

Introduction

America is experiencing a risk-assessment revolution in criminal justice. The role of risk assessment is increasingly prominent at all stages of the criminal justice system, including policing, pretrial detention, sentencing, corrections, and parole.[3] The assessment of offender risk was a central feature of criminal sentencing prior to the mid-1970s, when jurisdictions throughout the United States began rejecting forward-looking risk assessment in favor of backward-looking retributive concerns. As jurisdictions have reconsidered the use of incarceration, risk assessment has returned in new and more sophisticated ways.[4] Efforts to reduce mass incarceration have led to the adoption of risk assessment tools as an alternative to money bail, as a way to determine the length of incarceration and the intensity of probation or parole supervision, and as a tool to decide which mentally ill or substance-abusing offenders might safely be diverted to treatment in the community. For example, in its 2017 revision, the Model Penal Code prominently endorsed consideration of risk.[5] Similarly, the First Step Act, perhaps the most far-reaching federal sentencing reform in a generation, mentions risk or risk assessment no less than one hundred times, and relies on risk assessments to allocate prison programming and determine prisoner release.[6]

As risk-based approaches have multiplied, a growing number of critics have asked whether certain risk instruments are predictively valid and whether they might reinforce rather than reduce bias, including racial bias, in criminal justice outcomes.[7] We argue here that the discussion about the role of risk assessment has neglected a separate and fundamental question: how do judges and others use risk assessment in practice?

That question now has a constitutional dimension. The Fifth Circuit affirmed a federal judge’s order that the cash-bail system in Harris County, Texas, violated the Due Process Clause because it adopted a “flawed procedural framework” in which bail decisions by individual judges were arbitrary in practice.[8] The federal judge found that Harris County judges departed from release recommendations made by the pretrial services department as much as 66 percent of the time.[9] The federal judge has now entered a detailed final consent decree in the litigation.[10] Additional litigation is pending in other circuits, and the Harris County litigation may provide both a constitutional roadmap and a model for remedies.[11] In California, lawmakers enacted noteworthy legislation barring cash bail, but permitting local courts and judges to exercise discretion regarding what policies to adopt.[12]

In light of litigation and innovation across a wide range of jurisdictions, a model for the regulation of criminal justice risk assessment is urgently needed. Because the federal First Step Act mandates periodic risk assessment for virtually all federal prisoners, much attention will be paid to how federal prison staff implement risk assessments pursuant to the Act and exercise their discretion.[13] However, in criminal justice, unlike in other areas of the law, one fundamentally does not have regulations concerning the use of risk assessment, such as rules specifying assessment criteria, the peer review process, and standards of judicial review.[14] A model for the regulation of criminal justice risk assessment is urgently needed.

In this Article, we present empirical evidence on how judges actually use (and fail to use) risk assessment instruments. States have made the use of these instruments advisory rather than presumptive or mandatory, and as a result, the discretion of the decision-maker plays an important role.[15] Nonetheless, very little information is available about how judges actually use these risk assessments.[16] To address this question, we conducted a set of studies of judicial decision-making. We focused on Virginia because, in the words of the Model Penal Code, “[o]n risk assessment as a prison-diversion tool, Virginia has been the leading innovator among American states.”[17] Virginia was the first state to incorporate risk assessment into its sentencing guidelines to permit alternative sentences for “lowest risk” drug and property offenders.[18] We found that in 2016, of the entire population of offenders who qualified for the use of the Nonviolent Risk Assessment (NVRA), 46 percent scored in the category of low risk offenders, and were therefore eligible for an alternative sentence.[19] Of those, 42 percent did receive an alternative sentence and 58 percent did not.[20] We observed extremely wide variation between both individual judges and entire judicial circuits in their use of the NVRA.

Second, we surveyed judges in Virginia and found highly divergent attitudes towards risk assessment. We found that a strong majority of judges endorsed the principle that sentencing eligible offenders should include consideration of recidivism risk. However, a strong majority also reported that in their jurisdictions, they lack adequate alternatives to imprisonment. A sizable minority of judges had great discomfort with the goals and the use of risk assessment at sentencing. Some described risk assessment as just “another tool that aids but does not supplant judicial judgment.”[21] Others expressed extreme distaste for risk assessment. For example, one judge stated, “Frankly, I pay very little attention to the [risk assessment] worksheets . . . I also don’t go to psychics.”[22] That some judges were not fully cognizant of the availability of risk assessment in sentencing was also unsurprising, given the almost complete lack of judicial training on the subject.[23]

These studies of judicial practice and opinion concerning risk assessment produced several important insights into how to better institutionalize use of the risk assessment. To change behavior, it is not enough to adopt a technical tool—attitudes towards the use of risk assessment in decision-making need to be addressed if the tool is to be used well.[24] A new approach is needed that takes account of the interface between general quantitative risk information and the officials, such as judges, prosecutors, and probation officers, who take that information into account in decision-making. That interface must be evidence-informed and based on common goals. In this Article, we provide a roadmap of three such goals: (1) presenting risk information in a more comprehensible way to decision-makers; (2) structuring decision-making to better make use of that information; and (3) accompanying these reforms with ongoing monitoring, through judicial review and independent scientific peer review of data to assess on-the-ground use of risk assessment.

In Part I of this Article, we present an overview describing the growing use of risk assessment instruments in criminal justice, including use encouraged by the development of empirically validated quantitative tools to assess risk. We explain what risk assessment is, how it can be used and in what criminal contexts, the types of risk instruments used, and approaches adopted in recent years. Second, we provide an overview of the reasons why risk assessment has become attractive to so many lawmakers and policy-makers in recent years and concerns critics have raised about the design of instruments. Third, we describe a wide variety of risk assessment tools in use, ranging in complexity from simple checklists or scorecards to algorithms.

In Part II, we present three results of the studies in Virginia: (1) the use of nonviolent risk assessment; (2) the results of a novel survey of judges; and (3) the results of an analysis focusing on local treatment resources. We found highly inconsistent uses of risk assessment and highly divergent attitudes toward risk among judges. The local availability of behavioral health resources is substantially correlated with variation in the use of alternative sentencing.

In Part III, we turn to the experience with risk assessment in other states. That evidence similarly suggests the challenge of integrating risk assessment into judicial and other official decision-making.

In Part IV, we develop a model for how to better regulate and integrate risk assessment into criminal justice decision-making. First, we draw from administrative procedures used in other fields that regulate risk management using accepted criteria, with a public approval process, validation, and ongoing review. Second, we draw from research on decision-making regarding risk and communication of risk. We suggest that risk assessment information should be better explained, in its training and in its presentation, to be understandable in the context in which judges and other criminal justice officials work. Third, we discuss solutions aimed at better structuring the manner in which risk information is incorporated into decisions. Fourth, we recommend ongoing validation not just of risk assessment instruments, but also of their use in practice by the decision-makers. We conclude by offering a six-part roadmap for how to regulate the use of risk assessment in the criminal justice system.

The need to get risk assessment right has never been more pressing. Every year, upwards of ten million people are arrested in the United States.[25] Judges must decide whether to jail people pending trial. Every year, millions of people are convicted in the United States. Incarceration, which had reached record levels, began to modestly decline starting in 2008, but so far we have halted growth rather than substantially reduced the prison population.[26] Efforts to reduce reliance on incarceration have been bipartisan, and have largely focused on efforts to divert offenders, including less risky offenders, from jail and prison.[27] That makes the task of accurately identifying such offenders all the more important. At the same time, mounting criticism has focused on the lack of transparency and potential for bias in the design of risk instruments.[28] Those concerns should extend to how officials use risk assessment in practice. It is not enough to adopt a risk-based tool. To change decision-making, we need to address policy, structure of decision-making, and training. Coherent regulation is needed.

I. The Second Coming of Risk Assessment in Criminal Justice

There has been a movement across many fields of regulation and business to use risk assessment and cost-benefit analysis tools to improve rules and outcomes.[29] Those methods can both involve efforts to measure the frequency and likelihood of adverse outcomes, as well as their severity or cost.[30] Such methods are an everyday aspect of business management, and they are integral to entire fields like insurance and finance, in which there has been longstanding debate regarding the goals of regulation and public policy that affect how risk should be distributed in society.[31] Risk-based methods have been widely incorporated into federal administrative policy-making, but they have had an uneven history in criminal justice.

Beginning in the early twentieth century, risk assessment tools were adopted in criminal justice in the United States. In the 1970s, those experiments with the use of quantitative tools to evaluate criminal cases were largely neglected, as criminal justice focused more on retributive approaches to punishment. The largely retributive approach has changed in the past decade and a half—policy-makers turned back towards rehabilitation—and more quantitative research on recidivism began to inform policy.[32] We are in the midst of a risk assessment revolution in criminal justice. In the pretrial context, New Jersey and Kentucky, for example, along with many local jurisdictions, use the Public Safety Assessment (PSA) developed by Arnold Ventures.[33] State Supreme Courts, such as the Indiana Supreme Court, the Kentucky Supreme Court, the Nebraska Judicial Council, and the New Jersey Supreme Court, have ordered studies or sweeping changes.[34] States are using risk-based instruments to assess conditions of confinement as well.[35] That said, in criminal justice, unlike in other fields, one typically does not have regulations concerning use of risk assessment. Instead, the process has often been ad hoc and has involved conveying risk information to judges and other decision-makers who have retained their traditional discretion. In the sections that follow, we describe three issues: (1) the traditional use of risk assessment in criminal justice; (2) the recent rise in the use of more empirically-informed risk assessment instruments in a variety of criminal justice settings; and (3) the debates regarding the value of risk assessment in criminal justice.

A. The Traditional Use of Risk Assessment

The most widely used definition of risk assessment describes it as the process of using risk factors to estimate the likelihood (i.e., probability) of an outcome occurring in a population.[36] “Risk factors” are simply variables that (1) statistically correlate with recidivism, and (2) precede recidivism in time. In the case of pretrial decision-making, the relevant population consists of persons facing criminal charges. In the case of sentencing, the relevant population consists of convicted offenders.

In the area of pretrial detention, traditionally the focus in the United States was to impose bail to prevent flight and ensure that the defendant was present in court. Beginning in the 1970s, the focus on preventing the commission of additional crimes increased.[37] The 1984 Federal Bail Reform Act and state laws adopted in almost every state changed that to focus on the risk assessment of a defendant’s failure to appear in court or commission of a new crime, but these laws asked judges to predict failure to appear or commission of new crime without the benefit of empirical evidence.[38] The laws typically defined “dangerousness” extremely broadly, giving judges substantial discretion whether to consider defendants dangerous or not.[39] Unsurprisingly, given this virtually unlimited discretion, empirical work has found judges highly variable in making pretrial detention decisions.[40]

Judicial discretion involving an assessment of the likelihood of recidivism was once a feature of indeterminate sentencing. Judges and parole officials shared the assessment of risk. At the time of conviction, the judge would set a sentencing “range” (e.g., three to five years) and later (i.e., within this three-to-five-year range) parole officials would make a determination that a prisoner was sufficiently unlikely to recidivate and that he or she could be released from prison. As indeterminate sentencing gave way to determinate sentencing, the obligation fell increasingly (and later, almost exclusively) on the judge to fix sentence lengths. Risk assessment began to disappear from the practice of sentencing because judges became tasked with assigning backward-looking retributive sentences and not forward-looking sentences based on risk of future crime.[41]

B.The Rise of Modern Risk Assessment

Risk assessments are now commonplace at each stage of the criminal process, from police investigations, pretrial settings, sentencing, corrections, during parole and community supervision, as well as in specialized courts such as juvenile or mental health courts. Risk assessment involves, as noted, a process designed to predict outcomes. What outcome is being predicted may depend on the criminal setting in which it is being used, and the risk may be defined to include types of reoffending or failure to appear, substance abuse, or other outcomes. Risk assessment is distinct from risk management, which tries to reduce risk through supervision or treatment interventions. Further, it matters not just how risk is defined, but for what time frame an outcome is to be predicted. In general, risks increase as time frames increase.

Two types of errors may result whenever predictions are made. The individual, despite the assessment that the person is “low” risk, may commit a crime (or fail to appear in court, or violate probation, etc.) which would constitute a “false negative” prediction. Alternatively, the individual may be assessed as “high” risk, but may not commit the relevant type of violation, which would constitute a “false positive” prediction. False negatives may be particularly salient when, despite a prediction to the contrary, an offender commits a serious crime. In contrast, false positives are hard to detect. If a person is erroneously given a lengthy sentence, they cannot easily show that they would not have reoffended had they been released into the community.

The types of risk assessment tools vary, depending on their designs and the uses to which they are put. There are over four hundred structured risk assessment instruments used around the world.[42] Many are largely interchangeable, however, and involve the same or similar risk factors.[43] The developers of some of these tools claim not only to assess risk, but also to address the rehabilitative “needs” of a person.[44]

Researchers describe how these instruments have evolved from first-generation tools, consisting of clinical judgment and the experience of a decision-maker, to second-generation tools relying on static risk factors (such as criminal history, age, and gender), to third-generation instruments looking at both risks and needs, as well as static and dynamic risk factors such as educational status and employment, and finally to fourth-generation instruments, which provide individualized plans based on assessment of static and dynamic factors. A fifth generation of these tools may use machine learning techniques to predict recidivism in real time, using far more complex analysis. This “generation” terminology should not necessarily be taken to mean that more recent and complex tools necessarily perform better, however.[45]

Jurisdictions have widely experimented with using risk instruments pretrial, as an alternative (or a supplement) to reliance on cash bail. Jails have become increasingly overburdened in the United States, with most of those in jail serving pretrial detention.[46] In response, state supreme courts, such as the Indiana Supreme Court, the Kentucky Supreme Court, the Maryland Supreme Court, the Nebraska Judicial Council, and the New Jersey Supreme Court have ordered studies and changes to pretrial bail decision-making, sometimes accompanied by legislation or changed exclusively through legislation.[47] The pace of change has been rapid. Every state has adopted new pretrial policies, with five hundred enactments from 2012 to 2017, and at least fourteen states have adopted statistical risk assessment pre-trial since 2012.[48] The American Bar Association recommends the use of pretrial risk assessment, as does the National Association of Counties and the Conference of State Court Administrators.[49]

Research has shown that quantitative assessments are more reliable in their predictions than those of individual decision-makers.[50] One study found that 42 percent of people would be released pretrial if New York State used a risk assessment instrument to make decisions concerning pretrial release, rather than bail and subjective judicial assessments.[51] Over two dozen local jurisdictions and the entire state of New Jersey are now using the Public Safety Assessment tool funded by Arnold Ventures.[52] The tool was based on analyses of criminal cases in three hundred American jurisdictions. It is freely available, and it was designed to remove factors associated with racial disparities in pretrial detention, such as arrest history, and instead relies on factors such as conviction history.[53] It relies on static factors, and not on information gleaned from interviews with a subject, because it is designed to be used early on in the criminal process.

Other jurisdictions have relied on risk assessment in sentencing to divert outright certain classes of offenders from criminal punishment. There are many important legal and policy differences between the pretrial and sentencing contexts. In the pretrial context, the question is whether a person will appear in court and whether they might pose a danger of recidivism pretrial. However, while retribution is an (arguably) appropriate concern in sentencing, it is (inarguably) an inappropriate concern pretrial, before a person has been convicted of any crime.

Use of risk assessment during sentencing has increased in recent years. Virginia was an early adopter of this approach.[54] As efforts to reduce incarceration have become more prominent in the states, almost half of the states now use risk-based instruments at sentencing, at least in some types of cases.[55] State supreme courts have approved the use of risk assessment in sentencing.[56] Some states, such as Kentucky, Ohio, Oklahoma, Pennsylvania, and Washington, have required judges to consider risk assessment during sentencing.[57] A 2007 National Center for State Courts report encouraged this movement towards empirically informed sentencing approaches.[58] As part of the adoption of realignment or Justice Reinvestment Initiative legislation, states have increased the use of alternatives to incarceration. In addition, a number of states have enacted bail reform legislation. In California, realignment legislation requiring the reduction of statewide prison populations in order to comply with prison-crowding-related court rulings led to the use of county-specific plans that reduced imprisonment quite dramatically, including through the use of a range of different risk-assessment tools.[59] In New York, lawmakers enacted statewide bail reform legislation that went into effect on January 1, 2020.[60]

The juvenile justice system has similarly seen a shift towards the use of risk assessment instruments to identify high-risk offenders for greater sanctions or for rehabilitation. While in 1990, only one-third of state juvenile justice systems used risk assessments, by 2003, 86 percent used risk assessments.[61] While more work has been done on the validity of risk instruments used to predict violence for adults, a systematic review of instruments used in the juvenile setting suggested, on average, the same predictive ability as in adult settings.[62]

In the area of policing, police officers have long made predictions regarding neighborhoods and individuals that may be more likely to engage in criminal activity, and agencies can deploy resources based on those assessments of risk. Increasingly, police departments have used data analysis to inform predictive decision-making.[63]

Still other jurisdictions focus on risk assessment in the probation stage to better utilize community corrections, including as an alternative to incarceration. The National Institute of Corrections has described how its recommended evidence-informed approach uses risk assessments.[64] Similarly, the federal probation office and many state probation agencies use risk-based instruments to make probation decisions.[65] Probation and reentry generally is an area where risk assessment similarly plays a far greater role than it did in the past.

C. Risk Assessment Debates

Risk assessment in criminal justice has never been more widespread and never more debated among policy-makers and scholars. The Model Penal Code, as revised by the American Law Institute (ALI) in 2017, encourages the use of “actuarial instruments or processes to identify offenders who present an unusually low risk to public safety.”[66] One reason for this endorsement is the following:

In virtually every decision-making situation for which the issue has been studied, it has been found that statistically developed predictive devices outperform human judgments . . . This is one of the best-established facts in the decision-making literature, and to find otherwise in criminal justice settings would be surprising (at best) and suspicious or very likely wrong (at worst).[67]

The Model Penal Code recommends sentencing commissions “develop actuarial instruments or processes, supported by current and ongoing research, that will estimate the relative risks that individual offenders pose to public safety” to be incorporated into sentencing guidelines.[68] The Code also calls for needs assessments to match offenders with rehabilitative interventions.[69]

1. Penal Theory

As noted above, there are particularly serious questions regarding theory of criminal punishment in the sentencing context (as opposed to when risk assessment is used during pretrial proceedings, incarceration, or community supervision). The ALI view of risk assessment in sentencing reflects a hybrid approach towards criminal punishment, incorporating both retributive and utilitarian approaches. Broadly put, the retributive approach focuses backwards on an offender’s moral culpability for crimes committed in the past, while the utilitarian approach focuses forwards on deterring future criminal acts by the offender or other would-be offenders (or by incapacitating the offender). Retributivists believe that an offender’s moral culpability for past criminal acts should be the sole consideration in determining punishment.[70] Offenders should be punished “because they deserve it, and the severity of their punishment should be proportional to their degree of blameworthiness.”[71] Those who follow a utilitarian approach believe punishment is justified principally by its ability to decrease future criminal acts by the offender or to deter other would‐be offenders.[72]

Many scholars argue that any workable theory of criminal punishment should address both retributive and utilitarian concerns.[73] In an influential hybrid approach, Norval Morris developed a theory of “limiting retributivism” in which retributive principles may set an upper (and perhaps also a lower) limit on the severity of punishment, but within that range, utilitarian concerns can be taken into account.[74] The Model Penal Code approach reflects this hybrid view, in which risk assessment does not entirely define sentencing, but in which risk assessment can be used to identify low-risk offenders eligible for reduced sentences.[75]

Those who share a purely retributive approach towards criminal punishment would object to the use of risk assessment (or any other utilitarian considerations) in criminal justice. The use of risk assessment in sentencing involves a normative choice: to make decisions regarding predicted outcomes. As a result, scholars, criminal justice policy-makers, and decision-makers, like judges, may not share that normative preference. Some, such as Bernard Harcourt, have objected to the use of prediction generally on grounds that justice should not be based on future outcomes, explaining that risk assessment is not compatible with traditional theories of just punishment.[76] Our focus here is on critics who have raised a series of concerns that operate within the assumption that some hybrid of retributivism and utilitarianism are acceptable goals of criminal punishment. However, we want to highlight that judges and others who reject the use of risk assessment may do so for some of the same reasons that scholars may criticize it: they view risk assessment, correctly, as incompatible with a wholly retributivist view of criminal punishment.

2. Predictive Accuracy

Another set of concerns has to do with the quality of the predictions that risk assessment instruments provide. For example, it is far more challenging to predict uncommon violent offenses (e.g., homicide) than to predict common property offenses, for which there is far more data. An additional concern is with the time period during which the data relied upon are collected. The Model Penal Code recommends that risk assessment tools be periodically reviewed for their reliability.[77] Indeed, many risk assessment instruments used by parole boards, for example, have never been independently validated, have not been validated on in-state populations, are not regularly updated, and are sometimes altered to add additional factors that are not part of the original instrument.[78]

3. Inequality

Others argue that policy-makers have failed to attend to the real distributional consequences of risk assessment instruments. Expressing that view, as noted in the Introduction, then-Attorney General Eric Holder questioned the use of risk assessment as potentially causing “fundamental unfairness.”[79] The concepts of fairness and accuracy as used in the risk assessment context may themselves need to be clarified and can create difficult trade-offs.[80] However, a central and important fairness concern is that risk assessment can result in racial disparities or disparities based on other invidious criteria.[81] For example, Pennsylvania considered a sentencing approach in which rural offenders were given fewer points in their risk scores than urban offenders,[82] an approach that would have strongly correlated with race, and that was for good reason rejected in 2017.[83]

Gender is a factor explicitly taken into account by some risk assessment instruments. Sonja Starr has argued that doing so violates the Equal Protection Clause because under intermediate scrutiny, such assessments use a gender classification without a substantial state interest to justify doing so.[84] Others have responded that use of gender as a factor in discretionary risk assessment does not constitute a classification, and does not raise constitutional concerns, because (1) gender is empirically a highly predictive risk factor for many types of offenses, and (2) avoiding gender as a risk factor for recidivism works to the disadvantage of female offenders.[85]

4. Transparency

A separate concern is with the transparency of the risk assessment instruments used. Certain private companies have marketed software without making public the factors relied upon in complex algorithms; one such algorithm, COMPAS, marketed by Northpointe, apparently relies on socioeconomic and family factors.[86] Critics argue that, although race is of course not a static factor in the algorithm, these dynamic factors likely correlate closely with race.[87] A ProPublica report called the COMPAS software “biased against blacks.”[88] In the widely discussed Loomis case, the Wisconsin Supreme Court rejected an appeal by a person who objected, at his sentencing, that he could not review the basis for the Northpointe software’s risk assessment, nor whether invidious factors or factors that have a disparate impact based on invidious criteria were relied upon.[89] The U.S. Supreme Court ultimately denied certiorari in the case, but the case raised a serious due process concern that is likely to be litigated in future years.

5. Efficacy

Others fear that using risk assessment is too incremental an approach towards the problem of mass incarceration and that more forceful interventions are needed. Thus, Jessica M. Eaglin has questioned the use of risk assessment, arguing that evidence-based approaches are not the right way to reduce mass incarceration.[90] To date, risk assessment has not been used to reduce incarceration to the extent some advocates prefer.

6. Implementation

The question that has been least explored, in our view, in the literature is how judges use risk assessment. Critics have assumed that instruments, whether desirable or not, are uniformly applied by judges.[91] The Model Penal Code has recommended advisory use of risk assessment.[92] Where that is the case, the defendant can challenge the findings of an assessment “in open court,” and can “contest any adverse findings.”[93] However, research suggests that decision-makers do not always accurately perceive risk and that whether risk estimates are actually used may depend on the manner and format with which risk information is communicated.[94] Whether judges appropriately follow the advisory or even presumptive recommendations concerning risk assessment has been little studied in the literature. That is the problem we turn to in the next Part.

II. Empirical Studies of Risk Assessment in Virginia

If risk assessment is desirable because, “[i]n virtually every decision-making situation for which the issue has been studied, it has been found that statistically developed predictive devices outperform human judgment,” then the question remains how well human decision-makers actually use their judgment to incorporate the information from risk assessment.[95] That question has not been carefully examined in the past. Now that risk assessments are so commonly used by legal decision-makers, it is a question that can and should be examined. We examine that question by focusing first on the experience in Virginia with risk assessment in sentencing, based on several studies that we conducted, and then we turn to evidence from other jurisdictions concerning the use of risk assessment instruments.

A. The Virginia “Nonviolent Risk Assessment” Instrument

The Virginia model for the use of risk assessment in sentencing is important to study because Virginia was the first state to make risk assessment a formal part of sentencing. As a result, the Virginia model has received a great deal of attention from policy-makers, judges, and scholars, including, as noted, the American Law Institute in the revised Model Penal Code.[96] The use of risk assessment in Virginia has been hailed as a “valuable test” case and a model for “ratchet-down” approaches in sentencing that reduce prison populations: it identifies low-risk individuals.[97] The Virginia risk instrument has been lauded for its substance and for the process used to develop it. The instrument is transparent, publicly available, and validated within the state, and more recently revalidated.[98] Virginia’s instrument is simple and easy to understand. It involves “placing risk discretion in the sentencing courtroom.”[99] We sought to assess that discretion: how is this risk instrument used by judges in practice?

In 1994, the Virginia Legislature adopted the so-called truth-in-sentencing legislation and abolished parole in the state. To avert a resulting fiscal “collapse”[100] of the state’s prison system, the legislature simultaneously adopted risk assessment “to reduce the use of incarceration for nonviolent criminals, in order to offset the increased prison stays for violent offenders that were built into the original Virginia guidelines.”[101] The Legislature directed the newly formed Virginia Criminal Sentencing Commission (VCSC) to develop an empirically based risk assessment instrument.[102] The goal lawmakers set out for the VCSC was to divert 25 percent of the “lowest-risk, incarceration-bound, drug and property offenders” from prison to alternative sanctions such as jail, probation, community service, outpatient substance-abuse or mental health treatment, or electronic monitoring.[103] In the words of Richard Kern, the first Director of the VCSC, among the “main goals of the 1994 sentencing reforms” was to “expand alternative punishment/treatment options for some non-violent felons” by adopting statistical instruments “to divert low risk offenders” from prison.[104]

The VCSC developed the instrument by 1996; it was implemented at pilot sites in 1997.[105] The risk assessment instrument is administered only to offenders for whom the state’s sentencing guidelines recommend incarceration in prison or jail. In addition, offenders must meet certain eligibility criteria (e.g., a criminal history of only nonviolent offenses). If the offender’s total score on the instrument is below a given cut-off, he or she is recommended for an alternative, community-based sanction. If the offender’s score on the instrument is above that cut-off, the prison or jail term recommended by the sentencing guidelines remains in effect. The National Center for State Courts (NCSC) evaluated the use of the instrument at the pilot sites from 1998 to 2001.[106] The VCSC then conducted a validation study of the instrument to prepare it for state-wide use; the goal was to designate as lower-risk the group of individuals with an approximately 85 percent likelihood of not having a felony conviction within three years of release from confinement.[107]

The instrument was adopted statewide in 2002, for all felony larceny, fraud, and drug cases.[108] Thus, in 2002, the Nonviolent Risk Assessment Instrument (NVRA) was included as one of the sentencing worksheets to be completed for all eligible offenders convicted of one of four drug and property crimes—Larceny, Fraud, Drug Schedule I/II (e.g., possession of cocaine), and Drug/Other (i.e., marijuana distribution). In 2012, the Commission re-validated its instruments on large samples of eligible drug and larceny/fraud offenders.[109] Recidivism was defined, for those purposes, as a reconviction for any felony offense within three years of release. The Commission found sizable differences between those designated as lower and those designated as higher risk offenders; 12 percent of lower risk drug offenders recidivated, as compared with 44 percent of higher-risk drug offenders, for example.[110]

The NVRA instruments for each offense examine only the following static factors: (1) offender age at the time of the offense; (2) gender; (3) prior adult felony convictions; (4) prior adult incarcerations; (5) whether the person was legally restrained at the time of the offense[111]; and additionally, for drug offenses, (1) prior juvenile adjudication, and (2) prior arrest or confinement within the past twelve months (as opposed to legal restraint) at the time of the offense.[112] If the offender’s total score on the instrument is below the cut-off, the offender is recommended for an alternative sanction. If the offender’s score on the instrument is above the cut-off, the prison or jail term recommended by the sentencing guidelines remains in effect.

The NVRA instrument was adopted to be used alongside the sentencing guidelines in 1994, and therefore, the use of the NVRA is not considered a “departure” from the sentencing guidelines. Rather, an alternative sentence, when imposed after using the NVRA, is considered to be in compliance with the guidelines. After the NVRA is completed, judges have discretion whether to follow the recommendation for an alternative sentence. Judges also have discretion regarding which alternative sentence, if any, to impose.

We aimed to study whether the nonviolent risk assessment instrument is being used to divert from prison 25 percent of the “lowest-risk, incarceration-bound, drug and property offenders,” and if so, what types of alternative sentences they receive.[113]

B. Judicial Reliance on the Nonviolent Risk Assessment Instrument

First, we reviewed fiscal year (FY) 2016 sentencing data the VCSC shared with us concerning the use of the NVRA instrument.[114] The use of this instrument is important in a large number of cases; over 8,000 people were convicted of eligible offenses in Virginia in FY 2016. In that year, the Commission received a total of 23,713 sentencing guidelines worksheets.[115] Thus, one quarter of felony convicts were convicted of crimes that were eligible for the use of the NVRA. Of those, 6,787 people were eligible offenders for whom a risk assessment form was received. Over a thousand additional offenders were eligible offenders for whom a risk assessment form was either not filled out or not shared by the judge with the VCSC. The table below displays our analysis of receipt of alternative sentences in fiscal year 2016.

Of the entire population of 8,443 offenders eligible for risk assessment under the NVRA, 3,396, or 45.8 percent, scored in the category of least violent offenders and were therefore eligible for an alternative sentence. Of those, only 42.2 percent (1,433 people) did in fact receive such an alternative sentence. Of violent offenders, 23.4 percent (941 people) received alternative sentences. Of the group of offenders for whom NVRA information was missing, 39.7 percent (408 people) received alternative sentences. For that group, the NVRA information was not included in the sentencing record; it is not known whether that information was considered.[116]

Second, we examined what types of alternative sentences were offered under the NVRA. Those alternatives range from jail time to release for time served or release under supervised probation, and they also include rehabilitative options such as drug treatment. Table 2 below displays, for all eligible offenders who received an alternative sentence in FY 2016, which types of alternative sentences were imposed. Since cases may, and often do, involve more than one type of alternative sanction, the totals add up to more than 100 percent of cases.

Table 2: Types of Alternative Sanctions Imposed in NVRA Cases

The most common alternative sanction offered was supervised probation, with almost half of those receiving alternative sentences receiving jail time. While local jail and state prison both involve incarceration, it is easier for relatives to maintain visits to a local jail. Further, the jail sentences, which are one year or less, are far lower than the thirty-month median prison sentence imposed for the relevant offenses, according to VCSC data.[117]

The variation between judicial districts and between judges was also striking. There are 120 Circuit Courts in Virginia, organized into 31 Circuits. The Circuits varied widely in the ratios of those persons diverted to alternative sentences, in the ratios of missing NVRAs, and in the provision of alternative sentences to persons who scored low risk versus high risk under the NVRA. The thirty-one Circuits had a mean alternative sentencing rate of 33 percent, with a minimum of 19 percent of sentences diverted and a maximum of 54 percent diverted. Low-risk alternative sentencing rates varied by Circuit from 22 percent to 67 percent. Higher-risk alternative sentencing rates varied by Circuit from 11 percent to 51 percent. The variation among individual Circuit Court judges was as follows: judges had a mean alternative sentencing rate of 32 percent, with a minimum of 11 percent of sentences diverted and a maximum of 65 percent diverted. Broken down further, low-risk alternative sentencing rates varied from 7 percent to 85 percent among individual judges while higher-risk alternative sentencing rates varied from 0 percent to 60 percent among individual judges.

If the NVRA was intended to provide more uniformity among courts and judges, that goal does not appear to have been achieved. That said, we express no view on the optimal level of consistency among courts and judges. Different courts or different judges may have different offender populations or resources available to them. The judicial survey, which we describe next, addresses some of those differences.

C. Judicial Survey Findings

In addition to analyzing sentencing data concerning low-risk offenders in Virginia, we aimed to study why judges differ so widely in their use of risk assessment information. There have been very few efforts to survey judges to assess their attitudes regarding their use of risk assessment in the United States.

One recent survey by Steven Chanenson and Jordan Hyatt asked whether judges use risk assessment well and noted how little data about that question was available. They conducted a survey of judicial attitudes, surveying 137 judges in 37 states, including both state and county-level judges.[118] Acknowledging that it was a “non-representative and small sample,” the authors noted they only provided suggestive data on the question. Many responding judges were familiar with risk assessment but believed that “their judgment was more accurate than actuarial at-sentencing assessments.”[119] Judges were more favorable towards the use of risk assessment to release low-risk individuals than to imprison high-risk offenders, and the vast majority supported the use of risk assessment in sentencing in some manner.[120] We also located a survey of judges and probation officers concerning juvenile justice in four states; it found that half of the judges and probation officers frequently override recommendations from needs assessments and risk assessments (the distinction between the two is discussed more in Part IV). Judges were particularly likely to override recommendations where there was limited availability of community and residential placements.[121] The authors of the study concluded that lack of training, inadequate resources, and inappropriate instruments hindered the use of risk assessments in juvenile justice.[122]

Judges in many states, including Virginia, are the primary “consumers” of risk assessments at sentencing, yet their views on risk assessment are not typically solicited. In Virginia, since the NVRA instrument was adopted statewide in 2002, we expected that judges would be familiar with the instrument and would be able to comment on how they use it. We sent judges a brief survey asking them to answer several questions concerning the use of the risk assessment.[123] The survey was anonymous. We also included postcards that judges could separately mail if they agreed to be interviewed further concerning their views on the use of the NVRA, with results to be similarly described with anonymity for participating judges. We conducted the survey between November 2017 and January 2018, sending the survey by mail to all 161 Circuit Court judges in Virginia. In Virginia, each city and county has a Circuit Court, which handles all criminal felony cases. There are 31 Circuits and 161 Circuit Court judges. Virginia is one of two states in which judges are selected by legislative election, based on a majority vote of the Virginia House of Delegates and the Senate. Circuit Court judges serve for terms of eight years.[124]

We received responses from eighty-five judges (a response rate of 52.8 percent). To our knowledge, this is the highest response rate reported to a statewide judicial survey. (We note that surveys of federal judges conducted by the U.S. Sentencing Commission have reported similarly high response rates.)[125] We found, in summary,[126] the following: eight out of ten judges surveyed believed that sentencing drug and property offenders should be based not only on the seriousness of the crime committed and the offender’s blameworthiness, but also on the risk that the offender will commit another crime in the future. In addition, eight out of ten judges stated that they were either “familiar” or “very familiar” with the use of the NVRA instrument in sentencing drug and property offenders. Only approximately half of all judges, however, stated that they “always” or “almost always” considered the results of the NVRA instrument in sentencing drug and property offenders, and approximately one-third stated that they “usually” did so. Indeed, approximately half of the judges stated that they relied equally on the NVRA instrument and on their judicial experience in sentencing a drug or property offender, and approximately one-third stated that they relied primarily on their judicial experience.

In explaining how they exercised discretion, we found that seven out of ten judges rated availability of alternative interventions—such as outpatient drug or mental health programs—within their jurisdiction as “less than adequate,” and 5 percent rated such alternatives as “virtually non-existent.” In addition, three-quarters of the judges responded affirmatively when asked whether an increase in availability of alternative interventions for drug and property offenders would change their sentencing practices.

We also asked whether adopting a policy requiring judges to provide a written reason for declining to impose an alternative intervention on an offender who scores as “low risk” would increase the likelihood of judges imposing such alternative interventions. Six-in-ten judges believed that such a policy would increase the use of alternatives, and four-in-ten believed that it would not do so. However, when asked if they favored or opposed the adoption of the policy regarding providing a written reason described in the previous question, one-third of the judges responded that they favored adopting such a policy, and two-thirds opposed it.

After completing the survey, judges also had the opportunity to provide additional written comments. Judges were candid in their views of risk assessment. Some were quite opposed to the use of risk assessment generally. Others described the need for more treatment resources in their districts.

Our survey yielded three primary conclusions. First, a strong majority of Virginia judges endorsed the principle that the sentencing of eligible offenders should include a consideration of the risk that the person will commit new crimes. The judges were familiar with the use of the NVRA in sentencing, and usually or always considered the results of the NVRA in relevant cases. Judges reported that the NVRA “constitutes a useful tool within the general sentencing scheme.” Another judge said, “I support the use of these risk assessments under current usage—specifically the risk assessment is used to reduce and not increase incarceration recommendations.” A more skeptical judge said that “[i]t should be clarified to judges and litigants alike that Evidence Based Practices like the Nonviolent Risk Assessment are but another tool that aids but does not supplant judicial judgment.” In contrast, a significant minority of judges excluded considerations of risk when sentencing eligible drug and property offenders and were largely unfamiliar with the NVRA.

Second, a strong majority of judges found the availability of alternative interventions for eligible drug and property offenders in their communities to be inadequate at best. Those judges stated that they believed that an increase in the availability of alternative interventions would change their sentencing practices.

As one judge put it: “The assessment is useful. The problem is the lack of useful alternatives. In several counties in my Circuit, there are no inpatient treatment options.” Another judge said, “We need more alternative options—lack sufficient treatment programs and follow-up. Unfortunately, that costs money which communities are reluctant to provide.” Yet another judge said, “bona fide alternative programs must first exist.” One judge said, offering still more detailed concerns: “There is presently no valid alternative in our area. Referral to local mental health takes thirteen weeks for the initial interview. Who knows how long to start treatment . . . We need a statute which requires that all areas of the state have equal access to drug treatment.”

Third, a majority of judges believed adopting a policy requiring a written reason for declining to impose an alternative intervention on eligible offenders who score as “low risk” would increase the likelihood that such sentences would be imposed. Currently, Virginia judges are asked to provide their reasons for departing from sentencing guidelines, but the use or non-use of the NVRA is not considered a sentencing departure. Requiring judges to express reasons might affect their behavior. However, a majority of judges oppose the adoption of such a policy. One judge responded, for example: “Having to write out reasons for Guidelines departure is already an added time and effort burden on the sentencing process. To add another requirement to explain the sentencing decision would simply complicate and drag out the sentencing even more.” Another judge added, “At some point someone must realize that adding more paperwork . . . takes time and when court staffing remains the same, this takes time away from hearing cases, deciding cases, reading, signing orders, etc.” Another judge similarly noted the time it would take to provide reasons for not following NVRA recommendations: “Requiring judges to take 3–10 minutes per such sentencing to explain will be an unnecessary drag on our criminal dockets.”

D. The Treatment-Resource Hypothesis

In a related analysis, we examined the hypothesis that, because they lack treatment resources, judges do not always provide alternative sentences.[127] We obtained data on availability of mental health and substance abuse treatment resources across judicial districts in Virginia, and found a substantial correlation between those resources and the imposition of alternative non-jail sentences. A “treatment-resource hypothesis” explains a significant portion of the variation in such sentences. This correlation suggests that judges’ statements in the survey and the interviews that treatment resources matter are reflected in the data. Further, this suggests that more non-incarceration alternative sentences may require not just more training on the use of risk assessment, for example, but rather additional investment in treatment resources.

In conclusion, we found that most judges are familiar with and embrace risk assessment as a major consideration in sentencing property and drug offenders, but that judges find that community alternatives to imprisonment in their jurisdictions are often scarce. Further, most judges oppose the adoption of a policy requiring them to write out their reasons for declining to impose alternative interventions on low-risk offenders. We discuss next how judicial education in structured risk assessment,[128] better-structured risk information, and increased resources for community programs addressing criminogenic needs may help realize the promise of risk assessment as a means of reducing mass incarceration.

III. Empirical Evidence from Other Jurisdictions

As risk assessment has been increasingly used in criminal justice settings, data has begun to emerge regarding how judges and other decision-makers use such assessments in practice. The evidence is not all encouraging—but it is highly informative. Consistent with research on structured decision-making generally, the evidence suggests decision-makers do not take account of quantitative information when they do not value it, or when it is not incorporated into the structure of their work, or perhaps, when inadequate resources are provided to make use of the information. Below we discuss evidence from risk assessment in pretrial decision-making in Kentucky and evidence from risk assessment in probation in Pennsylvania, as well as evidence from jurisdictions that have not adopted risk assessment in certain contexts.

A. Pretrial Risk Assessment in Kentucky

Kentucky is one of a growing group of states that now requires risk assessment in pretrial decision-making.[129] Kentucky is one of only a handful of states to eliminate commercial bail, which it did in 1976 legislation.[130] In 2011, lawmakers enacted the Public Safety and Offender Accountability Act, which required use of “evidence-based practices to reduce the likelihood of future criminal behavior”—this included the use of risk assessment to make decisions regarding pretrial release.[131] The pre-existing state criminal procedure statute had required that a judge release a defendant on personal recognizance or on an unsecured bail bond unless the judge decided, “in the exercise of its discretion,” that such release would not “reasonably assure” the appearance of the defendant in court; the statute stated that such discretion shall rely with “due consideration” on recommendations of the pretrial services agency.[132]

Implementing this legislation in 2013, the Kentucky Supreme Court issued an order requiring all judges to expedite release of pretrial offenders who scored a low to moderate risk based on the Public Safety Assessment (PSA) risk assessment developed by the Laura and John Arnold Foundation.[133] If judges chose to impose cash bail, they had to provide written reasons for doing so. However, many judges did not follow the recommendations of the instrument.[134] Some judges simply wrote “flight risk” or “danger” as their reason for not using the PSA.[135] One prosecutor even created bumper stickers objecting to risk assessment: “Catch and release is for fish not felons.”[136] In 2017, the Kentucky Supreme Court issued a new rule making the program “uniform” for judges, expanding the applicability of risk assessment to new classes of defendants, and asking pretrial services to provide bi-annual reports to the Chief Justice to monitor the judicial use of the risk assessment.[137]

In a recent article, “Assessing Risk Assessment in Action,” Megan Stevenson analyzed data from Kentucky from 2009 to 2016 (pre-dating the most recent 2017 Kentucky Supreme Court rule).[138] Analyzing over 1.5 million criminal cases originating with a new criminal offense (rather than with a probation or parole violation or a violation of conditions of pretrial release), Stevenson found that the adoption of pretrial risk assessment sharply increased release rates in 2011, but this rate of change immediately began to fall.[139] When the Kentucky Supreme Court adopted the PSA in 2013, the same effect was observed—an initial increase in release rates, followed by a decline in release rates.[140] Over time, Stevenson describes, judges reverted to their prior habits in bail-setting pretrial; the increase in release rates was “short-lived,” and by 2015, release rates were lower than they had been prior to the 2011 legislation.[141] Stevenson found no meaningful change in the pretrial arrest rate during this time period, and found an increase in pretrial failure to appear.[142] Stevenson also found very different responses based on type of jurisdiction, with urban regions far more likely to experience a decline in their pretrial release rates.[143] Stevenson notes that, in part due to this variation between urban and rural jurisdictions, there was no detectable impact of the adoption of risk assessment on racial disparities in pretrial release in Kentucky.[144]

Stevenson’s results are consistent with the frustration the Kentucky Supreme Court has itself expressed in adopting new measures to try to corral judges into implementing risk assessment pretrial. James Doyle has written that this experience suggests that “covert work rules, workarounds, and informal drift will always develop, no matter what the formal requirements imposed from above try to require.”[145] Stevenson documents not only that judges may return to their prior practices, but also that increased use of risk assessment may be uneven and may fail to improve outcomes as intended. Time will tell whether the Kentucky court rule adopted in 2017 is any more successful than prior interventions. As we describe in the next Part, a different approach towards educating judges and restructuring their decision-making may be needed.

B. Pretrial Risk Assessment in New Jersey

In contrast with the accounts provided so far, in which risk assessment has not resulted in a significant change over time, the state of New Jersey has also adopted the PSA statewide[146] but experienced quite dramatic changes. New Jersey has reported its data online and reports more consistent use of the instrument and a large reduction in jail population. They describe, for example, that only 19 percent of people are confined to detention pretrial, with over two-thirds receiving pretrial monitoring and services.[147] The New Jersey courts describe a 55 percent decline in pretrial jail population, from almost nine thousand people in 2016 to about five thousand people at the end of 2018.[148] The acting administrator of the New Jersey courts reported on these declines: “New Jersey has successfully transformed an antiquated money bail system into a modern, risk-based system that relies on empirical evidence to better identify the risk a defendant poses.”[149] Why has PSA worked in New Jersey and not in other jurisdictions? One reason may be that the decision-making by judges is more structured, with a pretrial decision-making framework provided to judges.[150] However, other challenges remain: without relying on fines and fees imposed pretrial, the courts face serious budget shortfalls and inadequate resources for pretrial services and treatment.[151]

C. Additional Risk Assessment Jurisdictions

Despite the use of risk assessment instruments at almost all phases of the criminal justice process across a large number of state and local jurisdictions, we typically do not know how jurisdictions are actually using risk assessment instruments in practice.[152] Thus, in Arizona, a pretrial risk instrument has been encouraged state-wide but not required and not incorporated into guidelines. Steven Chanenson and Jordan Hyatt report anecdotal evidence that lawyers and judges in Arizona sometimes consider that pretrial risk information, but the impact of it is unclear.[153] There is no systematic evidence documenting those results.

To provide another example from the sentencing context, in Utah, judges are provided evidence from a risk assessment instrument as part of pre-sentencing reports. But there has been no evaluation of the use of that instrument or study of its use by researchers. The 2015 Justice Reinvestment Initiative legislation emphasized that judges should use risk assessment, but we do not know if they in fact are doing so or how they are doing so.[154]

In 2016, a journalist reported the results of a review of over 1,500 cases in Chicago in which judges 85 percent of the time made pretrial decisions contrary to a new risk instrument to be used to help determine pretrial decisions.[155] The review found that the amount and conditions of bail varied widely by judge. An Illinois Supreme Court justice raised the concern that these judges are not just unwilling to use risk assessment but “they are not being sufficiently trained and supervised and are not being held accountable by the system.”[156] Attitudes towards types of crimes may also explain the judicial resistance to the risk assessment. The review noted that bail was most commonly required for gun possession charges, with bail set for 98 percent of suspects, while the risk assessment called for it in only about 5 percent of such cases. [157]

In Maryland, in response to concerns about the constitutionality of the use of bail, the Court of Appeals adopted rules intended to end the use of cash bail. In response, according to a study, detention rates have apparently increased, as judges have issued “no bail” detention orders rather than use cash bail.[158] The University of Baltimore Pretrial Justice Clinic analyzed data from Prince George’s County, and found that while cash bail declined 11 percent, detention without bond rose almost 15 percent.[159] Professor Colin Starger commented that: “In a time where judges are politically accountable, there’s a fear you’re going to release someone who will go on to commit a crime so there’s a lot of public pressure to detain people.”[160]

A study by Richard Berk examined the use of risk assessment by the Pennsylvania Board of Probation and Parole, which commissioned a machine-learning study to develop prediction instruments. The information generated by the instrument was provided to the Board members as they made decisions, beginning in 2013. The risk information, if used, could have reduced recidivism substantially; 58 percent of those released on parole reoffended within two years, while the risk instrument predicted that if those recommended had been released, the recidivism rate would have been 27 percent.[161] The study concluded that when they began to use the instrument, “[t]he patterns for release proportions were roughly the same whether or not the machine learning forecasts and reliabilities were available to the Board.”[162] Perhaps the Board largely followed their same practices, or perhaps their decision-making relied on the same data sources as the machine learning.[163] Another study by Berk examined risk assessment in corrections, and found that using a risk assessment instrument better predicted inmate misconduct than traditional classifications used by prison administrators.[164] A final study by Jill Viglione, Daniell S. Rudes, and Faye S. Taxman examining the use of risk and needs assessment in two adult probation districts in a state found that the tool was overwhelmingly administered but “rarely” linked the resulting scores to their management or supervision decisions.[165]

D. Non-Risk-Assessment Jurisdictions

We can also learn from jurisdictions that do not use risk assessment. One case in point is North Carolina. In North Carolina, risk assessment is not used, except in a pilot jurisdiction concerning bail and pretrial decision-making.[166] However, the relevant statute does not preclude a judge’s consideration on pretrial risk. In an open-ended way, it provides for a right to have a judge set conditions for release.[167] Inconsistent with that preference, the statute then conveys the notion that pretrial risk should be assessed by focusing on what the criminal charges are—and that the way to address the risk posed by those charges is to impose money bail. Thus, North Carolina statutory provisions require judicial officials to impose a secured bond “[i]f the judicial official determines that the defendant poses a danger to the public,” or if a defendant commits a crime on pretrial release, an official can double the amount of money required.[168] Such statutes focus judges on pretrial risk, but suggest that risk be considered as reflected in criminal charges, and not by other evidence such as risk assessment. For that reason, a task force in North Carolina suggests that in order to change decision-making, the system must overcome “faulty assumptions,” which will require amending the governing statutes in addition to adopting risk assessment.[169] That said, a number of jurisdictions within North Carolina are using risk and needs assessments—as well as pretrial services—to address needs, court appointment reminder systems, and special dockets to reschedule missed appearances.[170] The lack of a clear statewide approach has not prevented local experimentation.

IV. Regulating Risk Assessment in Criminal Justice

The experience of a range of jurisdictions with risk assessment adds support to our concern that far more attention must be paid to the structure of decision-making that is supposed to be informed by risk assessment. Empirical research on judges has generally asked whether extra-legal factors influence their decision-making. Researchers have found that judges, like all decision-makers, can be influenced by forms of biasing information.[171] These problems are not unique to the risk assessment setting. For example, in the area of forensic science, judges have been criticized for not applying scientific criteria for screening out unreliable and invalid forensic evidence.[172] Efforts to encourage judges to use more stringent gatekeeping in the forensic science area have been largely unsuccessful.[173] In this Part, we develop proposals for improving how decision-making can be designed to encourage actual deliberation and more thoughtful use of risk assessment and we discuss how they fit in with a Due Process framework for judicial review. We provide a roadmap for: (1) presenting risk information in a more comprehensible way to decision-makers; (2) structuring decision-making to better make use of that information; and (3) accompanying these reforms with ongoing monitoring—through judicial review and open peer review—of data to assess on-the-ground use of risk assessment. Section A discusses (1) the presentation of risk information. Section B discusses (2) how to structure decision-making. Section C turns to (3) the regulation of risk information both through judicial review and peer review by independent researchers.

A. Conveying Risk Information to Judges

One challenge in conveying risk information to judges and other decision-makers is that they may prefer information presented to them in categories rather than frequencies or probabilities, which then raises important policy questions regarding how risk categories are defined. That problem suggests why the design of structured decision-making processes must be preceded by a more fundamentally transparent process: the regulation of risk assessment requires a set of rules designed to ensure public approval and ongoing review of risk assessment in criminal justice.

An initial question is whether changing the way that risk information is conveyed can improve its use. Social science scholarship studying how to communicate risk of recidivism in criminal justice has been the subject of decades of scholarship, but it is comparatively thin compared with studies on how to communicate assessment of risk for violent behavior in other legal contexts, such as involuntary civil commitment.[174] That literature describes how it is not enough to simply provide information about risk. Such information, to be used effectively, must be presented in a way that overcomes misunderstandings about its meaning and emphasizes its relevance to the decision-making task at hand.[175]

Even when communicating valid risk information, perceptions of risk can differ depending on how one communicates risk.[176] The format and framing of risk may matter.[177] Not all decision-makers are numerate and comfortable with quantitative information.[178] Presenting information in the form of frequencies (twenty out of one hundred offenders), versus probabilities (20 percent of offenders), versus categorical terms (such as “high risk”) can powerfully impact how well decision-makers use the information.[179] Judges and other decision-makers in the criminal system tend to prefer categorical information, such as estimates of low, moderate, or high risk. However, such classification raises crucial questions about how to define these categories.[180] Decisions regarding thresholds to place individuals into risk categories should be made through a public regulatory process. In Virginia, for example, a public agency categorized “low risk” offenders to be recommended for “alternative sanctions” through an open process.[181] When private companies design risk assessment instruments, categorization is often neither public nor transparent.[182]

Some jurisdictions have used visual models to convey risk information and to incorporate the information into decisions, and research shows that graphs can sometimes improve assessments of risk (although perhaps less so for less numerate people).[183] A risk-detention matrix can display and convey the recommended structure of decision-making, rather than simply supply risk information and ask the decision-maker to incorporate it in their judgment. However, research is currently inconclusive regarding whether conveying risk assessment using images—graphs, charts, or histograms—improves decision-making in criminal justice contexts.[184]

Studies show that like all decision-makers, judges can be influenced by a range of factors, including their political beliefs, desire for promotion, and consideration for reappointment or election.[185] Particularly relevant to risk assessment of offenders is the concern that judges may not want to be perceived as “soft” in their sentencing practices, out of concern for reappointment and reelection; the recall of the judge in the Brock Turner case is a high profile example of the public scrutiny that can result from sentencing decisions.[186]

Separate from any career-oriented influence on behavior, there is research finding that judges, like all individuals, rely on shortcuts or heuristics when processing information.[187] That suggests that judges may not give sufficient weight to statistical evidence that fails to confirm their prior beliefs. Studies have found that judges ostensibly rely on a range of factors in setting bail. For example, judges often rely solely on prosecutor’s recommendations, rather than data they receive. Even judges who believe they rely on many types of information in fact rely “almost exclusively on prosecutorial recommendation.”[188] Studies have also found troubling evidence that judges rely on an offender’s race when making decisions concerning sentencing.[189]

The approach taken by the Pennsylvania Commission on Sentencing has been to convene focus groups of judges, district attorneys, public defenders, and probation officers, in order to create a survey and solicit evidence on what methods for communicating risk at sentencing might be effective.[190] Not only may it create more legitimacy and accountability to involve a range of actors in the decisions whether and how to use risk assessment, but input may be useful regarding how to present and incorporate risk information in decision-making.

Whether attorneys present risk assessment information to the decision-maker at the time the decision is made may also make a difference. In the pretrial context, studies suggest that provision of counsel at hearings can powerfully affect release rates.[191] Further research is needed on the role of counsel regarding risk-based decision-making.

B. Structured Decision-Making and Risk Assessment

This Section focuses on how to structure decision-making to improve the use of risk information. Structured decision-making is simply a formal or rule-based procedure for making decisions. Some states have incorporated risk assessment into decision-making frameworks, providing structured guidance on how to make use of risk assessment information. Other jurisdictions, like Virginia, may generally preserve unfettered discretion of decision-makers, regarding whether and how to make use of risk assessment information.[192] It may be that even requiring reasoned decision-making and reviewing data collection on decision-making is not enough. Efforts to incorporate quantitative information into decision-making may need to rethink the fundamental structure of that decision-making process. The danger of structuring professional judgment is that the value judgments made may not be consistently or reliably reflective of the underlying risk information.[193] Decision-makers must carefully assess the relationship between risk information and the structure or categories recommended. Such actors may need to alter fast-paced bail decision-making to permit the time for the efficient use of risk assessment information. Sentencing guidelines may need to be made more binding, and more user-friendly when incorporating risk assessment. This may necessitate substantial judicial education efforts.

1. Presumptive Risk Assessment

An alternative approach could simplify decision-making to make the use of risk assessment presumptive, rather than relying on the traditional exercise of discretion by judges or other decision-makers. Studies suggest that providing a numerical “anchor” can affect judges’ risk assessments.[194] A sentencing recommendation that judges impose no jail or prison time to the lowest risk offenders might be far more salient than a notation that a person is a low-risk offender, without any guidance as to what alternative sentence is appropriate in that situation.

Another approach would require the sentencing judge to state on the record a cogent reason whenever he or she disregards the sentence-lowering implications of a low-risk designation. Asking judges to give reasons in writing may result in more careful reasoning than a quick decision without any reasoned justification. The approach has its drawbacks, however. The Kentucky experience suggests that judges that are opposed to the risk-based recommendation can readily offer cursory explanations for not using it.[195] That information, though, can influence policy-makers. State sentencing commissions could periodically review the “cogency” of these rationales from presumptive deference to empirical findings of low risk.

Providing feedback to judges concerning their use of risk assessment may improve their performance. It is difficult and uncommon for there to be a successful appeal of a pre-trial bail decision or a sentencing decision.[196] Judges are essentially unreviewed by appellate courts in settings where they make risk assessments. The same is typically true in juvenile cases and for parole or probation decisions as well. The only accountability may come in the form of election or reappointment by legislators or supervising administrators. Moreover, judges and other criminal justice officials with large caseloads have strong incentives not to spend time on matters that they are unlikely to revisit ever again. A real concern could arise regarding the way feedback could skew incentives. Some jurisdictions have many more alternative sentencing and treatment options than others, and some judges could be penalized for not being in a high-resource jurisdiction in which risk-based sentencing is more feasible. Any effort to review and provide feedback to judges would have to take account of the options available to judges or other decision-makers.

2. Automating the Use of Risk Assessment

One approach that might reduce the perceived burden on judges or other decision-makers would be to automate part of their work. In Virginia, for example, one development that might assist judges in applying the NVRA and explaining reasons for not granting alternative sentences under the NVRA, is the adoption of the automated web-based Sentencing Worksheets and Interactive File Transfer (SWIFT) program, which has now been pilot-tested throughout the state.[197] This program will allow for much more rapid recording of all types of information required at sentencing, including the Nonviolent Risk Assessment for eligible offenders. Drop-down menus could supply the most common reasons a judge might decline to impose an alternative intervention on an offender who scores as low risk (e.g., “I believe the offender’s risk is higher than indicated by the Nonviolent Risk Assessment,” “No appropriate community program to address this offender’s needs exists in this jurisdiction,” or “This offender appears not to be responsive to treatment intervention.”). Automated systems can help to structure decision-making by setting out the order of potential decisions and requiring entry of information to explain each decision.

3. Needs and the Mitigation of Risk

In a range of settings, risk assessments may accompany needs assessments. These may include mental health screenings, substance abuse screenings, and educational assessments where the goal is not just to assess risk but also to mitigate it by providing rehabilitative interventions.[198] For example, in juvenile justice decision-making, it is common to consider risk assessment alongside needs assessments of various types.[199] Substance abuse screenings may address whether a person could benefit from drug treatment. When decision-makers are evaluating not just future risk but also how to mitigate it through treatment and other rehabilitation, they will need to look at more than just risk assessments. They must then have guidance on how to evaluate and incorporate information from very different sources to examine not just risk, but needs.

Risk assessment instruments do not tell decision-makers anything about whether an alternative to punishment (or bail, or supervised probation) might mitigate risk. Indeed, risk factors on such instruments that are not causal are not directly relevant to the reduction of risk.[200] For example, substance abuse is a causal risk factor for recidivism, but it can be changed using a treatment intervention, which reduces the risk of recidivism.[201] Similarly, to reduce failure to appear in the pre-trial setting, messaging and reminders help.[202] Other risk factors, such as a prior criminal history or age, are not changeable by any intervention.

As we found in Virginia, treatment needs may not be met, and inadequate resources may explain why decision-makers vary their alternative sentence use. Judges and other decision-makers may face hard choices concerning both risks and needs and tradeoffs between the two. For example, judges may be reluctant to divert a low-risk offender from prison if no community program to address the offender is available in the court’s jurisdiction. Correspondingly, an offender assessed as being at high risk of recidivism might benefit most from a community treatment program, but diverting that offender from prison to community treatment might well be precluded on public safety grounds even if such community treatment were available. As the National Center for State Courts evaluation of Virginia’s approach noted:

One of the primary concerns of judges and probation officers is the difficulty many young males have qualifying for alternative punishment. Unemployed, unmarried, males under age 20 begin with a score of nine points [on the NVRA], and any additional points render them ineligible for a diversion recommendation. Judges and probation officers know that VCSC research shows this type of offender has a high rate of recidivism, but they also believe this is the group most in need of services.[203]

We similarly found that Virginia judges often raised this concern in their survey responses. In doing so, they noted a lack of treatment resources. The NVRA and other risk assessment tools can tell decision-makers that an offender is low-risk and might be diverted from prison without endangering public safety. However, such instruments do not inform judges, probation officials, or corrections officials which offenders might benefit the most from messaging, drug treatment, mental health treatment, or jobs programs.

In regulating risk assessment, policy-makers should separate the tasks that we are asking judges to perform: (1) recommend incarceration or not, whether pretrial or as punishment for a conviction, and (2) separately recommend any treatment interventions. When multiple sources of information are provided, it is all the more important that a structured decision-making process provides guidance on how to incorporate information from more than one source. Otherwise, risk assessment may be at odds with needs assessment.

4. Institutional Incentives

Criminal justice actors may not readily incorporate risk assessment, or any other guidance to inform their discretion, if it conflicts with their pre-existing incentives. If judges have crushing dockets and very little time to consider pretrial or sentence determinations, providing detailed quantitative information in a manner that would slow down the process is unlikely to succeed at expanding the use of risk assessment data. Realigning those incentives may involve incorporating new approaches into official guidelines, providing resources to allocate decision-making time to consideration of new data, and rewarding decision-makers for their use of it.

In addition to practical, professional, and reputational considerations, financial incentives and resource constraints may also strongly impact decision-makers. If judges or prosecutors are funded based on their number of felony convictions, then alternative sentencing may reduce their institutional budgets. If there are no resources for community treatment alternatives to prison, then judges will not be able to divert offenders to such programs. Approaches to re-orient institutional incentives include adopting a “justice reinvestment” model for localities to fund diverting low-risk offenders. In this legislative scheme, localities use the fiscal benefits of reducing incarceration rates to offset the costs of alternatives to incarceration (e.g., community-based mental health and substance abuse programs). In California, for example, a “realignment” approach, enacted in legislation designed to comply with court orders to reduce prison overcrowding, provides localities fiscal incentives to reduce jail incarceration rates.[204]

C. Judicial Regulation of Risk Assessment

This Section focuses on judicial review as a tool to regulate risk assessment as used in practice. Judicial review, once viewed as implausible, has recently been utilized to highlight constitutional issues that may arise within practical applications of risk assessment. As we noted earlier, the Fifth Circuit largely affirmed a federal judge order that the cash bail system in Harris County, Texas, violated the Due Process Clause.[205] The Fifth Circuit found a state-protected liberty interest in pretrial release and affirmed the judge’s findings that the system as administered was arbitrary. The opinion emphasized that the judges departed from release recommendations by pretrial services as much as 66 percent of the time and that when pretrial services informed the court that a person was indigent, judges insisted on bail knowing full well that the result would be detention.[206] What was also noteworthy, and consistent with our findings in Virginia, was that this pattern of behavior was not based on any written procedures, but rather on an “unwritten custom and practice that was marred by gross inefficiencies, did not achieve any individualized assessment in setting bail, and was incompetent to do so.”[207] As a result, the Fifth Circuit described how bail had been imposed “almost automatically on indigent arrestees” without having meaningfully considered risk or ability to pay.[208]

However, the panel also emphasized that it would be unduly burdensome to require that judges provide written decisions explaining their reasons for denying release or bail.[209] Moreover, that ruling, while noteworthy as a constitutional ruling, did not provide a roadmap for the improvement of judicial decision-making using risk information. Indeed, while it found the then-current system in Harris County to have been arbitrary due to the rote insistence on cash bail for arrestees, the ruling also encouraged individualized decision-making without any minimal accountability in the form of reason-giving. Perhaps the new procedures that are developed on remand will put an end to a process that automatically detained indigent arrestees through the use of cash bail. However, the procedures the Fifth Circuit focused on, such as providing written reasons, may not be the best way to ensure consistency and reliability in judicial decision-making. If judges provide “individualized, case-specific reasons” but continue to automatically impose cash bail on indigent arrestees, the remedy will change nothing.[210]

We believe that the Due Process and Equal Protection Clauses demand additional assurances of consistency and reliability beyond the minimal requirement that some individual decision-maker theoretically consider the relevant criteria and state some reason for a decision.[211] Moreover, an entrenched, unwritten custom and practice may take a more forceful intervention to displace. Additional language in the Fifth Circuit opinion supports our view, focusing on how in practice, judges did not heed pretrial services and did not apply informed criteria to their decision-making.[212] Further, the Eleventh Circuit’s Walker v. City of Calhoun, Georgia ruling and other rulings distinguishing Walker similarly focused on the bare procedural form of a hearing but not the criteria followed or the quality of the decision-making that results.[213]

D. Statutory Regulation of Risk Assessment: The First Step Act

The federal First Step Act, enacted in December 2018, invokes the concept of “risk” no fewer than one hundred times, and tasks the Department of Justice and the Bureau of Prisons (BOP) with design and implementation of a risk assessment instrument to classify all 180,000 federal prisoners. The timelines imposed for this effort were stringent: 180 days to develop a new risk assessment instrument. Remarkably, that deadline was met. On July 19, 2019, the Department of Justice reported the development and initial validation of a new assessment instrument called the Prisoner Assessment Tool Targeting Estimated Risk and Needs (PATTERN). [214]

The PATTERN was designed to predict the likelihood of general and violent recidivism for all BOP inmates over a three-year follow-up period.[215] It contains static—unchangeable—risk factors as well as dynamic risk factors that experts believe are associated with either an increase or a reduction in risk.[216] Static items on the PATTERN include criminal history, age at first conviction, age at time of assessment, and educational attainment.[217] Dynamic risk factors on the PATTERN include the number of infractions committed during the current incarceration and the number of “beneficial” programs (e.g., educational, parenting, drug treatment, and vocational programs) successfully completed.[218] In what is sure to be controversial, the PATTERN is sex-specific, i.e., it was developed and validated for males and females separately.[219]

The PATTERN instrument contains many of the procedural and substantive elements of a sound risk-assessment approach. The First Step Act authorizing the development of the instrument was enacted through highly bipartisan legislation, which involved public deliberation over the choice of a risk-assessment approach towards federal incarceration. It was not adopted by judges or other decision-makers without legislative or regulatory process. The Act specified that prisoners must be classified into one of four risk categories—minimum, low, medium, and high—and the PATTERN report specified what likelihoods of recidivism are reflected in those categories.[220] The Act required the appointment of an Independent Review Commission, composed of scientists who have studied risk assessment, to vet possible instruments and report to the Attorney General—which they have done.[221] The Act specifies that risk should not be the exclusive focus and that treatment needs of prisoners should also be an emphasis. The Act further provides resources for rehabilitative programming in prisons and directives to assess those programs, which has been reflected in the PATTERN.[222] The Act also states that whatever existing instrument was selected, or new instrument was developed, it will be made public on the DOJ website so that researchers can evaluate it. No prior legislation had taken that step. The DOJ has since published its report on the PATTERN and, as of this writing, received detailed critical comments regarding the proposed instrument, with a review process underway.[223] Researchers and policy-makers will undoubtedly devote careful study to whether the PATTERN instrument is as predictive as its authors state and whether it performs well in practice.

The First Step Act addresses many of the concerns that we have raised in this Article concerning regulating risk assessment. The Act focuses on rehabilitative treatment in addition to risks, it ensures that the risk assessment instrument and data are made public, it was the result of legislative deliberation, and it calls for implementation through training, auditing, and evaluation by scientists, the Comptroller General, and the Attorney General. The Act calls for initial and continuing training of prison officials on its use.[224] The Act also asks that there be biannual evaluations of how prison officials use the assessments and that every two years the Comptroller General conduct a review of demographic disparities in how the assessments are used.[225] However, the Act also raises concerns consistent with those raised in this Article. While the Act calls for ongoing public release of data after an instrument is adopted, it did not call for such release of data concerning the initial design and validation of the instrument; nor did any occur. Other questions will arise during implementation, including whether adequate resources are devoted to reentry programming.[226] The Act represents an extremely large-scale adoption of evidence-based risk assessment. It will be important to learn from the implementation of the Act as policy-makers design, implement, and regulate additional risk-assessment measures.[227]

E. A Roadmap for the Regulation of Risk Assessment

In the subsections that follow, we discuss what elements should go into the design of a structure for judging risk and why they assist in assuring a non-arbitrary, constitutional, and sound risk-assessment procedure. The roadmap we provide focuses on the key features that we have discussed in this Article: (1) publicly specifying the criteria for risk assessment instruments; (2) defining the relevant risks and needs to be measured; (3) making the risk instrument public and accessible to researchers; (4) presenting risk information in a comprehensible way to decision-makers; (5) structuring decision-making to make better use of that information; and (6) accompanying these reforms with ongoing monitoring, through judicial review and by making data accessible to researchers.

1. Publicly Specifying the Risk Assessment Criteria

The questions of what type of risk assessment instrument to use, what risks and needs to measure, and at what thresholds, should be public choices. Too often, risk assessments have been adopted without public discussion or disclosure. Yet, as with bail schedules, substantive offense definitions, sentencing guidelines, and parole criteria, risk assessments represent normative choices that affect outcomes in criminal cases. While criminal justice actors’ decision-making has typically lacked transparency and deliberation,[228] some jurisdictions have incorporated public discussion and input into their development of risk assessment instruments.[229] For example, Virginia adopted risk assessment in sentencing in a statute, as described, and specified that the lowest risk 25 percent of offenders should be given alternative sentences. While the statute did not specify how risk assessment should be conducted, the statute[230] directed the Virginia Criminal Sentencing Commission to develop a risk assessment instrument. The Commission made the instrument it developed for use in sentencing transparent and sought input on its design.[231] Similarly, the First Step Act involved a legislative deliberation and delegation of risk assessment design and implementation to the Department of Justice, with advice from a new scientific review committee. There should be such an approval process before any risk assessment is used in the criminal justice setting.

2. Defining the Relevant Risks and Needs to be Measured

As described, selecting what threshold to define various categories of risk and selecting which are the relevant risks to be measured are policy choices. While some jurisdictions treat non-appearance as an important concern pre-trial, others reconsidered that choice and made efforts to reschedule court appearances and excuse certain non-appearance. Likewise, in sentencing, how “recidivism” is defined—whether as re-conviction or return to prison, for the commission of any crime or only the commission of a felony, and over what time period (e.g., within one year or within three years after release from prison)—are questions of policy, not science. Regarding needs, there are broader questions regarding what types of treatment are effective at reducing relevant risks.

3. Making the Risk Instrument Public and Accessible to Researchers

The growing use of risk assessment has often occurred without public disclosure of the instrument itself. Just disclosing a checklist or the weights associated with an instrument does not make public how it was validated and how those weights were selected. The First Step Act is an important departure from this approach, making the instrument and validating studies public. Virginia also made the instrument and validating studies public as well as certain underlying data upon request. Without such disclosure, researchers cannot assess whether a risk assessment does what it was set out to do or whether there are superior methods to accomplish those goals.[232]

4. Presenting Risk Information in a Comprehensible Way to Decision-Makers

As discussed in the previous sections, decision-makers commonly do not use risk-assessment information. One reason why concerns how that information is presented to them. The information must be presented in a clear, comprehensible manner. Providing counsel at legal proceedings involving risk information is another way to ensure that sound information is presented. For example, where pro se litigants appear at informal pre-trial hearings, the presentation may be highly abbreviated and unfair. One weakness of both the Fifth Circuit ruling and the First Step Act is the lack of focus on indigent representation to ensure a fair process to present and contest risk and needs determinations by decision-makers.

5. Structuring Decision-Making to Better Make Use of Risk Information

Focusing on how decision-makers actually use risk information is essential. As we have described, decision-making itself can be structured to make use of risk assessment information so that it does not seem extraneous to the decision-maker. Some legislation and judicial rulings have neglected this dimension of the problem. For example, the Fifth Circuit ruling, with its emphasis on custom and practice, addressed implementation but did not call for any structural changes to how judges make decisions. The district judge noted that even once Harris County adopted a risk assessment approach, the concern remained that the results of the risk assessment were only informative. Thus, judges might ignore them like they ignored 66 percent of the recommendations of pre-trial services.[233] Indeed, there is evidence that in the first year of implementation, some judges have “fallen back on their old ways, of trying to issue orders of preventive detention by setting money bail at an amount they know the person can’t afford.”[234]

In contrast, comprehensive efforts to restructure decision-making that involve decision-makers have been more successful. For example, Charleston, South Carolina, dramatically reduced its jail population by assembling a multi-disciplinary team of law enforcement, judges, mental health professionals, and others. The team not only implemented a new risk assessment approach, but it also relied on alternatives to arrest (e.g., issuing a ticket), placement in a community program instead of jail, as well as provision of public defenders at pre-trial proceedings to ensure representation.[235] The risk assessment is only one part of an overall strategy to rely less on jail and more on treatment and diversion.

6. Ongoing Monitoring, Including Judicial Review and Making Data Accessible to Independent Researchers

Adopting a risk assessment instrument is just the beginning. There must be monitoring of how it is implemented in practice. That monitoring can be delegated to a sentencing commission or scientific committee. By making data regarding implementation, not just initial validation, available to researchers, there can be careful study of whether the risk assessment is actually improving outcomes. The review of risk assessment instruments has not always been public or conducted by researchers independent of those who developed the instrument. If instruments are only validated in-house, or by the originators of the instrument, then the validations cannot be verified independently. An additional concern, as described in Part I, with risk assessment use has been that the data, even if sound when the risk assessment instrument is adopted, may be out of date if criminal offending patterns change or if innovations in pretrial supervision are implemented (e.g., telephone or text reminders of scheduled court appearances). Instruments should be re-validated over time at reasonable intervals and with attention to local variation in populations, resources, and crime patterns.[236]

Whether judicial review can improve this monitoring process is less clear. Courts may increasingly become involved in the use of risk assessment and real constitutional questions may be raised, as discussed. The scope of that review may depend on which stage of the criminal justice system it is used, as well as the accompanying regulations. In the sentencing context, courts may be less deferential given the stakes of applying or failing to apply risk assessment to impose prison sentences. The Iowa Supreme Court recently opined that “the shiny legal penny of a new risk assessment tool should be carefully scrutinized by the courts.”[237]

Judicial review may be more deferential in the pre-trial setting. The Fifth Circuit analysis noted the deference due to criminal justice actors in a due process analysis.[238] While liberty of arrestees, the court noted, is a very important interest, the court also noted that an efficient process stands to benefit arrestees who desire prompt resolution of their cases.[239] As a result, the panel was reluctant to burden judges with reporting requirements, such as statements of written reasons. The ruling is consistent with the larger practice in criminal justice. We note also that for risk assessment used by judges at sentencing, or by other decision-makers such as corrections officials or parole officials, the same liberty interest cannot be asserted. The Due Process Clause provides far less protection in those settings.[240] We nevertheless hope that the framework and practices that we discuss here will be used in those settings.

The Fifth Circuit discussion in O’Donnell v. Harris County, Texas did not suggest a robust role for judges in evaluating data concerning risk assessment. A serious Due Process analysis could emphasize the need not just for individual decision-making, but also accurate decision-making. Indeed, the district court decision in the case discussed empirical research in detail, including by academics who had studied the bail process in Harris Country, Texas, and found that denial of bail had systematic harmful effects on outcomes in criminal cases.[241] The district court also cited studies showing cash bail is not more effective than alternatives at assuring appearances for misdemeanor arrestees.[242] The district court described how Harris County had agreed to use a new risk assessment tool, the PSA, to inform pre-trial decision-making, and was in the process of implementing this new risk-assessment-based approach.[243] The consent decree that, as of this writing, has been submitted to the district court for approval, calls for ongoing data collection and monitoring of Harris County pre-trial outcomes, exactly as we would recommend. [244]

Our findings suggest the need for ongoing study regarding how judges and other decision-makers use risk assessment instruments in the various contexts they are asked to do so.[245] The same is true for other types of decision-makers who use risk assessments, such as police or probation officers. In the juvenile justice setting, one study found that “there was a positive relationship between frequency of use and perceptions of value.”[246] It may be that over time, judges and other decision-makers become accustomed to and appreciate using structured decision-making. But if decision-makers widely vary in their appreciation of risk assessment tools, then their decisions may become even more non-uniform over time. In addition, as described above, evidence regarding how risk assessments are used in practice should be incorporated into training and feedback that judges and other decision-makers receive.

Just as judges receive direction from sentencing guidelines and know that departures will be reviewed more carefully than within-guideline sentences, judges and other actors should understand that ignoring risk assessments will similarly result in increased scrutiny. The use of risk assessment must inform the discretion of criminal justice decision-makers such that they are accountable if they routinely fail to incorporate risk into their decision-making.

Conclusion

A central problem facing criminal justice today is that accurate and fair risk assessments will “make little difference if the decision-makers do not understand the information, which is a serious possibility.”[247] In this Article, we have explored these challenges. We set out the results of several studies of the use of risk assessment in Virginia, long held out as a national model for the use of validated and legislatively authorized use of risk at sentencing, and reviewed additional studies concerning risk assessment in other criminal justice settings and jurisdictions. We find that judges are not using risk assessments consistently, and they frequently discard them entirely. That should not be a surprise given that judges and other decision-makers typically receive almost no training in risk assessment, and their discretion to ignore risk assessment is virtually unchecked. Judges themselves have begun to express constitutional concerns with the unguided use of risk assessment. A concurring judge in the Iowa Supreme Court case of State v. Guise elaborated: “Even if the emerging risk assessment tools are found to have a place in sentencing as a ‘relevant’ factor, our law does not allow mere conclusions to be mounted on spikes and paraded around our courtrooms without statistical context.”[248]

The use of risk assessment in criminal justice should be regulated with rules to inform and structure decision-making, and a process for developing those rules should be articulated. The Due Process Clause and other constitutional criminal procedure sources do not provide sufficiently definitive or informative guidance in this new world where risk is judged at each stage of the criminal process. We view the Fifth Circuit ruling regarding pre-trial decision-making in Harris County as a significant step in the right direction—but also a missed opportunity. What was important in that ruling was its emphasis on how judges work in practice. What was missing was a plan for ongoing oversight of how judges incorporate validated risk assessments into decision-making, and a procedural framework to ensure consistency and accuracy. The consent decree settling the Harris County litigation will also supply the missing ongoing data collection and monitoring. The new federal First Step Act similarly provides for ongoing monitoring. It requires the risk instrument be made public, along with data on its validation, and periodic assessments. We hope such approaches become more common.

We set out a roadmap for better regulating the use of risk assessment in criminal justice with six key elements: (1) public deliberation regarding criteria for risk and needs assessment instruments; (2) defining the relevant risks and needs to be measured; (3) making the risk instrument public and accessible to researchers; (4) presenting risk information in a comprehensible way to decision-makers; (5) structuring decision-making to better make use of risk and needs information; and (6) ongoing monitoring, through judicial review and by making data accessible to independent researchers.

Using risk assessment to reduce reliance on incarceration and improve criminal justice outcomes is a salutary goal. As Richard Frase has argued, “with respect to low-risk assessments, can we afford to renounce any major sources of mitigation, given our inflated American penalty scales and overbroad criminal laws?”[249] Risk assessment could be used to dramatically reduce reliance on incarceration. So far that promise has not been realized. Unless policy-makers address the problem of judging risk assessment by the decision-makers tasked with using it, the risk revolution in criminal justice will not achieve its intended goals.

DOI: https://doi.org/10.15779/Z38B56D515.

Copyright © 2020 Brandon L. Garrett, L. Neil Williams, Jr. Professor of Law and Director, Center for Science and Justice, Duke University School of Law.

Copyright © 2020 John Monahan, John S. Shannon Distinguished Professor of Law, University of Virginia School of Law.

Many thanks to Rachel Barkow, Curtis Bradley, Jamie Boyle, Sam Buell, Sharon Dolovich, Jessica Eaglin, Jeff Fagan, Bernard Harcourt, Corinna Lain, Anna Lvovsky, Ralf Michaels, Erin Murphy, Justin Murray, Alice Ristroph, Carol Steiker, Megan Stevenson, Jonathan Wiener, and the participants at a faculty workshop at Duke University School of Law, and at the 2018 Criminal Justice Roundtable conference at Harvard Law School, for their invaluable comments on drafts. We thank Meredith Farrar-Owens, Jody Fridley, and the Virginia Criminal Sentencing Commission for their extraordinary help, both in sharing data and in assisting in our understanding and use of them. We could not be more grateful to Judge Ted Hogshire for his assistance and to Chief Justice Donald W. Lemons and Joanne B. Rome for their support in surveying Virginia judges. We thank Alexander Jakubow and Anne Metz, who did invaluable work analyzing data summarized here. Study findings were presented to the Virginia Criminal Sentencing Commission on April 9, 2018, and we are grateful for the feedback we received at that meeting. Empirical research presented in this Article was funded by a grant from the Charles Koch Foundation. The views expressed are those of the authors and do not necessarily represent those of either the Commission or the Foundation.

3. See, e.g., Cecelia Klingele, The Promises and Perils of Evidence-Based Corrections, 91 Notre Dame L. Rev. 537, 564–67 (2015) (critiquing use of risk assessment in parole and probation); Sonja B. Starr, The Risk Assessment Era: An Overdue Debate, 27 Fed. Sent’g Rep. 205, 205 (2015) (“[W]e are already in the risk assessment era.”); see also John Monahan & Jennifer L. Skeem, Risk Assessment in Criminal Sentencing, 12 Ann. Rev. Clinical Psychol. 489, 493–94 (2016) (describing use of risk assessment in sentencing). See generally Sandra G. Mayson, Dangerous Defendants, 127 Yale L.J. 490 (2017) (describing use of risk assessment in bail); Sonja B. Starr, Evidence-Based Sentencing and the Scientific Rationalization of Discrimination, 66 Stan. L. Rev. 803 (2014) (critiquing use of risk assessment at sentencing); Note, Bail Reform and Risk Assessment: The Cautionary Tale of Federal Sentencing, 131 Harv. L. Rev. 1125 (2018) (critiquing use of risk assessment in bail-reform practices). In addition, risk assessments are used in the civil context, such as in civil commitment to a mental hospital. For an excellent overview, see International Perspectives on Violence Risk Assessment 9–10 (Jay P. Singh et al. eds., 2018). ↑

4. Jonathan Simon, Reversal of Fortune: The Resurgence of Individual Risk Assessment in Criminal Justice, 1 Ann. Rev. Law Soc. Sci. 397, 413–18 (2005). ↑

5. Model Penal Code § 6B.07 (Am. Law Inst. 2017). ↑

6. First Step Act of 2018, Pub. L. No. 115-391, 132 Stat. 5194 (2018). ↑

7. For example, then-Attorney General Eric Holder questioned the use of risk assessment, stating: “Using group data to make an individualized determination, I think, can result in fundamental unfairness.” Joshua Barajas, Holder: Big Data is Leading to ‘Fundamental Unfairness’ in Drug Sentencing, PBS News Hour (July 31, 2014), https://www.pbs.org/newshour/politics/holder-big-data-leading-fundamental-unfairness-drug-sentencing [https://perma.cc/REF5-NU65]. For additional scholarly critics, see David Arnold et al., Racial Bias in Bail Decisions, 133 Q. J. Econ. 1885 (2018) (examining the impact of racial bias in bail decisions); Laurel Eckhouse et al., Layers of Bias: A Unified Approach for Understanding Problems with Risk Assessment, 46 Crim. Just. & Behav. 185 (2019); Klingele, supra note 1; Mayson, supra note 1; Sandra G. Mayson, Bias In, Bias Out, 128 Yale L. J. 2218 (2019) (critiquing the use of risk assessment that is not race conscious); David G. Robinson, The Challenges of Prediction: Lessons from Criminal Justice, 14 I/S: J. L. & Pol’y for Info. Soc’y 151 (2018) (evaluating different quantitative assessment tools in the criminal justice system); Starr, supra note 1. See in particular the recent debate occasioned by a statement of twenty-seven scholars, Chelsea Barabas et al., Technical Flaws of Pretrial Risk Assessments Raise Grave Concerns (July 17, 2019), https://dam-prod.media.mit.edu/x/2019/07/16/TechnicalFlawsOfPretrial_ML%20site.pdf [https://perma.cc/V2VM-5GDE]. These scholars concluded that there are “fundamental, technical problems with actuarial risk assessment instruments. These technical problems cannot be resolved. We strongly recommend turning to other reforms.” Id. at 4. A summary of the statement has received much attention. See Chelsea Barabas et al., The Problems With Risk Assessment Tools, N.Y. Times (July 17, 2019), https://www.nytimes.com/2019/07/17/opinion/pretrial-ai.html?searchResultPosition=1 [https://perma.cc/GC2P-JGZL]. Other scholars strongly dispute the conclusions reported in the statement. See Sarah Desmarais et al., Risk Assessment Tools Are Not A Failed “Minority Report”, Law 360 (July 19, 2019), https://www.law360.com/articles/1180373/print?section=access-to-justice [https://perma.cc/LSR9-T5Z7]; Sarah Lustbader, Risk Assessment Tools Are Flawed—Should We Throw Them Away?, Appeal (July 25, 2019), https://theappeal.org/risk-assessment-tools-are-flawed-should-we-throw-them-away [https://perma.cc/RAY9-79X3]; see also infra Part I.A. for a further discussion of traditional uses of risk assessment. ↑

8. ODonnell v. Harris County, 892 F.3d 147, 154 (5th Cir. 2018), aff’g, rev’g, and modifying in part, 882 F.3d 528, 536 (5th Cir. 2018). ↑

9. See id. at 154. However, the court held that judges need not be required to provide written decisions explaining their reasons for denying release or bail. Id. at 160 (“We decline to hold that the Constitution requires the County to produce 50,000 written opinions per year to satisfy due process.”). The court also relied on empirical analysis of outcomes in the county. See Paul Heaton et al., The Downstream Consequences of Misdemeanor Pretrial Detention, 69 Stan. L. Rev. 711, 786–87 (2017) (discussing the consequences of not using assessment to release defendants before trial). ↑

10. Consent Decree, ODonnell et al. v. Harris County, No. CV H-16-1414, 2019 WL 6219933 (S.D. Tex. Nov. 21, 2019). ↑

11. See, e.g., Walker v. City of Calhoun, 901 F.3d 1245, 1266 n.12, 1266–67 (11th Cir. 2018) (distinguishing ODonnell, and finding district court abused discretion in ordering preliminary injunction in favor of arrestee, concluding that a procedural due process analysis did not warrant such relief); see also Schultz v. Alabama, 330 F. Supp. 3d 1344, 1359–60, 1376 (N.D. Ala. 2018) (distinguishing Walker and granting preliminary injunction to enjoin practice of pretrial bail); Edwards v. Cofield, 301 F. Supp. 3d 1136, 1139 (M.D. Ala. 2018) (denying both plaintiffs’ motion for preliminary injunction and defendants’ motion for summary judgment in challenge to pretrial bail practices); Buffin v. San Francisco, No. 15-cv-04959-YGR, 2018 WL 424362 at 12 (N.D. Cal. Jan. 16, 2018) (denying motions for summary judgment in class action challenge to system of pretrial bail). For a description of the role of the American Bar Association in the Fifth Circuit as well as pending Sixth and Eleventh Circuit litigation, see Lorelei Laird, ABA Files Amicus Brief Challenging Money Bail System as House of Delegates Considers Resolution, ABA Journal (Aug. 10, 2017), http://www.abajournal.com/news/article/aba_amicus_brief_money_bail_5th_circuit [https://perma.cc/4ZX5-4A4J]. In addition, the Fifth Circuit has reversed as overly expansive an injunction entered on remand that called for the immediate release of any inmate not provided with a hearing within forty-eight hours. See *ODonnell v. Goodhart, 900 F.3d 220, 228 (5th Cir. 2018). ↑

12. See Erwin Chemerinsky, This is Not the Way to Reform California’s Bail System, Sacramento Bee (Aug, 20, 2018), https://www.sacbee.com/opinion/op-ed/article217018990.html [https://perma.cc/A79V-48CT] (explaining the dangers of discretion in money-bail decisions). ↑

13. See First Step Act of 2018 § 103, Pub. L. No. 115-391, 132 Stat. 5194 (2018) (requiring that the Comptroller General conduct an audit every two years to report on the implementation of the new risk assessment used in federal prisons); id. § 3632(f) (requiring that prison staff receive biannual auditing including of their “interrater reliability,” presumably requiring, then, testing of their consistency in decision-making). ↑

14. For example, the failed federal Regulatory Improvement Act specified the use of risk assessment for any “major rule” by a federal agency. Regulatory Improvement Act of 1998, S. 981, 105th Cong., § 623(2)(A)(iii) (1998). ↑

15. For a piece pointing out that abolition of parole meant “transferring risk discretion” to judges, see Kevin R. Reitz, “Risk Discretion” at Sentencing, 30 Fed. Sent’g. Rep. 68, 70 (2017). ↑

16. One excellent article has examined judicial variation and use of the pretrial risk assessment instruments used in recent years in Kentucky. See generally Megan Stevenson, Assessing Risk Assessment in Action, 103 Minn. L. Rev. 303 (2018). We discuss one survey of judicial opinion and media accounts of judicial practices in local jurisdictions in Part III. See also Richard Berk, An Impact Assessment of Machine Learning Risk Forecasts on Parole Board Decisions and Recidivism, 13 J. Experimental Criminology 193, 194 (2017) (noting we have “scant information about how actuarial risk assessments have affected practices and outcomes”). A working paper posted as this article went to press, by Megan Stevenson and Jennifer Doleac, examines Virginia sentencing data and finds that judges inconsistently use risk assessment. See Megan T. Stevenson & Jennifer L. Doleac, Algorithmic Risk Assessment in the Hands of Humans 18 (Nov. 18, 2019) (unpublished manuscript), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3489440 [https://perma.cc/3HUH-D6L2]. They found that recidivism was not affected by use of the risk assessment instrument during the time period that they examined and that while racial disparities did not increase through the use of the instrument, there were larger disparities in the districts in which judges relied more on the risk assessment instrument. Id. at 18–22. ↑

17. Model Penal Code § 6B.07 n.(c)(2) (Am. Law Inst., Proposed Final Draft 2017). ↑

18. See Brian J. Ostrom et al., Offender Risk Assessment in Virginia: A Three-Stage Evaluation 9 (2002), http://www.vcsc.virginia.gov/risk_off_rpt.pdf [https://perma.cc/L8TQ-JDUN]; see also Va. Criminal Sentencing Comm’n, 2010 Annual Report 38–41 (2010), http://www.vcsc.virginia.gov/2010AnnualReport.pdf [https://perma.cc/YN2E-BSHG]. ↑

19. Brandon L. Garrett, Alexander Jakubow & John Monahan, Judicial Reliance on Risk Assessment in Sentencing Drug and Property Offenders: A Test of the Treatment Resource Hypothesis, 46 Crim. Just. & Beh. 799, 802 (2019). ↑

20. See id. ↑

21. Brandon L. Garrett and John Monahan, Assessing Risk: The Use of Risk Assessment at Sentencing, 103 Judicature 42 (2019). ↑

22. Id. ↑

23. See infra Part II.C. ↑

24. Nor is the problem of how to best implement generalized data to inform individual decision-making unique to the risk assessment setting. See, e.g., David L. Faigman et al., Group to Individual (G2i) Inference in Scientific Expert Testimony, 81 U. Chi. L. Rev. 417, 455–57 (2014) (providing examples of risk assessment in the use of testimony). ↑

25. Rebecca Neusteter & Megan O’Toole, Vera Inst. of Justice, Every Three Seconds: Unlocking Police Data on Arrests 1 (2019), https://www.vera.org/publication_downloads/arrest-trends-every-three-seconds-landing/arrest-trends-every-three-seconds.pdf [https://perma.cc/7XEE-SNGL]. ↑

26. See Lauren E. Glaze & Danielle Kaeble, Bureau of Justice Statistics, Correctional Populations in the United States, 2013, at 2 (2014), https://www.bjs.gov/content/pub/pdf/cpus13.pdf [https://perma.cc/VQ5E-RFSV]. ↑

27. See, e.g., Right on Crime, http://www.rightoncrime.com [https://perma.cc/N4DZ-8577]; see also Newt Gingrich & Pat Nolan, Prison Reform: A Smart Way for States to Save Money and Lives, Wash. Post (Jan. 7, 2011), http://www.washingtonpost.com/wp-dyn/content/article/2011/01/06/AR2011010604386.html?noredirect=on [https://perma.cc/WE96-LELX]. ↑

28. See Jessica M. Eaglin, Constructing Recidivism Risk, 67 Emory L.J. 59, 88–101 (2017) (explaining how developers construct risk reduction tools); Julia Angwin et al., Machine Bias, ProPublica (May 23, 2016), https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing [https://perma.cc/9QYG-ETV8] (providing examples of biases in risk assessment tools); Anna Maria Barry-Jester et al., The New Science of Sentencing, Marshall Project (Aug. 4, 2015) https://www.themarshallproject.org/2015/08/04/the-new-science-of-sentencing [https://perma.cc/JN4U-MPRP] (examining the danger of using risk assessment in sentencing); see also Rebecca Wexler, Life, Liberty, and Trade Secrets: Intellectual Property in the Criminal Justice System, 70 Stan. L. Rev. 1343, 1348 (2018). ↑

29. See, e.g., Carnegie Comm’n on Sci., Tech., & Gov’t, Risk and the Environment: Improving Regulatory Decision Making 1, 73 (1993), http://www.ccstg.org/pdfs/RiskEnvironment0693.pdf [https://perma.cc/SS76-LGUA]. ↑

30. See Fred Anderson et al., Regulatory Improvement Legislation: Risk Assessment, Cost-Benefit Analysis, and Judicial Review, 11 Duke Envtl. L. & Pol’y F. 89, 89–90 (2000). ↑

31. See, e.g., Kenneth S. Abraham, Four Conceptions of Insurance, 161 U. Pa. L. Rev. 653, 672, 678 (2013) (critiquing risk-utility or cost-benefit approaches to insurance and describing how public policy influences questions or redistribution inherent in allocating risk). ↑

32. See Simon, supra note 2, at 413–17. ↑

33. See Laura & John Arnold Found., Public Safety Assessment: Risk Factors and Formula, at 2, http://craftmediabucket.s3.amazonaws.com/uploads/PDFs/PSA-Risk-Factors-and-Formula.pdf [https://perma.cc/K56R-M47M] (“LJAF created the PSA using the largest, most diverse set of pretrial records ever assembled—1.5 million cases from approximately 300 jurisdictions across the United States. Researchers analyzed the data and identified the nine factors that best predict whether a defendant will commit new criminal activity (NCA), commit new violent activity (NVCA), or fail to appear (FTA) in court if released before trial.”); see also Matthew DeMichele et al., The Public Safety Assessment: A Re-Validation and Assessment of Predictive Utility and Differential Prediction by Race and Gender in Kentucky (Apr. 30, 2018) (unpublished manuscript), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3168452 [https://perma.cc/LV65-Q9A9]. ↑

34. See, e.g., Ill. Staff, Supreme Court Issues Order on Pretrial Release, Ind. Lawyer (Dec. 23, 2014), https://www.theindianalawyer.com/articles/35974-supreme-court-issues-order-on-pretrial-release [https://perma.cc/RB7B-JU4E] (noting the review by the Supreme Court of Illinois); Pretrial Release Outcomes Studied by Nevada Judiciary, Nev. Judiciary (Oct. 2, 2015), http://nvcourts.gov/Supreme/News/Pretrial_Release_Outcomes_Studied_by_Nevada_Judiciary [https://perma.cc/5QFQ-ZXH6] (describing Nevada’s process). For a discussion of Kentucky Supreme Court orders, see infra Part III.A. ↑

35. See generally Francis T. Cullen et al., Eight Lessons from Moneyball: The High Cost of Ignoring Evidence-Based Corrections, 4 Victims & Offenders 197 (2009) (explaining the benefit of using actuarial results to examine conditions in prisons); Christopher Slobogin, Principles of Risk Assessment for Researchers and Practitioners, 36 Behav. Sci. & L. 507 (2018). ↑

36. Helena C. Kraemer et al., Coming to Terms with the Terms of Risk, 54 Arch. Gen. Psych. 337, 340 (1997). ↑

37. Samuel R. Wiseman, Pretrial Detention and the Right to Be Monitored, 123 Yale L. J. 1344, 1351 (2014) (“Historically, the U.S. system of bail and associated pretrial detention was employed solely to prevent pretrial flight.”); see Sarah L. Desmarais & Evan M. Lowder, Pretrial Risk Assessment Tools: A Primer for Judges, Prosecutors, and Defense Attorneys 5–6 (2019), http://www.safetyandjusticechallenge.org/wp-content/uploads/2019/02/Pretrial-Risk-Assessment-Primer-February-2019.pdf [https://perma.cc/4U7C-YX2B]. ↑

38. See Shima Baradaran & Frank L. McIntyre, Predicting Violence, 90 Tex. L. Rev. 497, 506–07 (2012). ↑

39. Jeffrey Fagan & Martin Guggenheim, Preventive Detention and the Judicial Prediction of Dangerousness for Juveniles: A Natural Experiment, 86 J. Crim. L. & Criminology 415, 422–24 (1996). ↑

40. Arpit Gupta et al., The Heavy Costs of High Bail: Evidence from Judge Randomization, 45 J. Legal Stud. 471, 477–78 (2016) (explaining how magistrate judges in Pennsylvania make risk assessment decisions on bail hearings); Megan T. Stevenson, Distortion of Justice: How the Inability to Pay Bail Affects Case Outcomes, 34 J. L. Econ. & Org. 511, 514–16 (2017) (describing the process and discretion of magistrate judges in Pennsylvania). ↑

41. See Michael Tonry, Legal and Ethical Issues in the Prediction of Recidivism, 26 Fed. Sent’g Rep. 167, 170–72 (2014) (providing context for the philosophical justifications of punishment in a risk-assessment context). See generally Kirk Heilbrun et al., Risk Assessment for Future Offending: The Value and Limits of Expert Evidence at Sentencing, 53 Court Rev. 116, 121 n.40 (2017) (describing and critiquing the use of risk assessment historically and currently); John Monahan & Jennifer L. Skeem, Risk Redux: The Resurgence of Risk Assessment in Criminal Sanctioning, 26 Fed. Sent’g Rep. 158 (2014) (providing historical context for the use of risk-assessment practices). ↑

42. Nicholas Scurich, An Introduction to the Assessment of Violence Risk, in International Perspectives on Violence Risk Assessment 3, 10 (Jay P. Singh et al. eds, 2016). ↑

43. Id. The tools available include the following: Correctional Offender Management Profiling for Alternative Sanctions (COMPAS); Correctional Assessment and Intervention System (CASI); HCR-20; Level of Service Inventory-Revised (LSI-R); Level of Service/Case Management Inventory (LS/CMI); Minnesota Screening Tool Assessing Recidivism Risk 2.0 (MnSTARR 2.0); Modified Wisconsin Risk Assessment (WRN); the Ohio Risk Assessment System (ORAS); the federal Post Conviction Risk Assessment (PCRA); the federal Pre-Trial Risk Assessment tool (PTRA); and the Static Risk and Offender Needs Guide-For Recidivism (STRONG-R). See, e.g., Kevin S. Douglas et al., HCR-20 Violence Risk Assessment Scheme: Overview and Annotated Bibliography 6 (2014) (reviewing the HCR tool for risk assessment); Northpointe, Practitioner’s Guide to COMPAS 1 (2015), http://www.northpointeinc.com/downloads/compas/Practitioners-Guide-COMPAS-Core-_031915.pdf [https://perma.cc/4UPN-LHLN] (providing an overview of one risk-assessment tool available to criminal justice practitioners); Timothy P. Cadigan et al., The Re-validation of the Federal Pretrial Services Risk Assessment (PTRA), 76 Fed. Probation 3, 6 (2012) (evaluating a pretrial risk-assessment system); Christopher T. Lowenkamp et al., The Federal Post Conviction Risk Assessment (PCRA): A Construction and Validation Study, 10 Psychol. Servs. 87, 88 (2013) (evaluating a postconviction risk-assessment tool); David J. Simourd & P. Bruce Malcolm, Reliability and Validity of the Level of Service Inventory-Revised Among Federally Incarcerated Sex Offenders, 13 J. Interpersonal Violence 261, 264 (1998) (describing a risk-assessment tool for sex offenders); see also Jenifer Cox, et. al., An Update and Expansion on the Role of the Violence Risk Appraisal Guide and Historical Clinical Risk Management-20 in United States Case Law, 36 Behav. Sci. & L. 517, 519–21 (2018) (describing various risk-assessment tools). ↑

44. For an overview of each of these types of risk assessment instruments, see Brian K. Lovins et al., Validating the Ohio Risk Assessment System Community Supervision Tool with a Diverse Sample from Texas, 3 Corrections 186, 187–88 (2017). See also James Bonta & D.A. Andrews, Risk-Need-Responsivity Model for Offender Assessment and Rehabilitation 4 (2007) (calling such instruments “fourth generation” risk assessment). ↑

45. J. Stephen Wormith, Automated Offender Risk Assessment: The Next Generation or a Black Hole?, 16 Am. Soc’y. Criminology 281, 288–93 (2017). ↑

46. See Justice Policy Inst., Bail Fail: Why the U.S. Should End the Practice of Using Money for Bail 5–8 (2012) (explaining the costs of money bail). ↑

47. See, e.g., Lorelei Laird, Court Systems Rethink the Use of Financial Bail, Which Some Say Penalizes the Poor, A.B.A. J. (Apr. 1, 2016), http://www.abajournal.com/magazine/article/courts_are_rethinking_bail [https://perma.cc/858M-JDUK] (describing the effect of Maryland’s revocation of cash bail on poor communities); Ovetta Wiggins & Ann E. Marimow, Maryland’s Highest Court Overhauls the State’s Cash-Based Bail System, Wash. Post (Feb. 7, 2017) (describing Maryland’s reconsideration of money bail). ↑

48. Amber Widgery, Nat’l Conference of State Legislatures, Trends in Pretrial Release: State Legislation 1 (2015); Nat’l Conference of State Legislatures, Trends in Pretrial Release: State legislation Update 1–2 (2018), https://www.ncsl.org/portals/1/ImageLibrary/WebImages/Criminal%20Justice/pretrialEnactments_2017_v03.pdf [https://perma.cc/8MSW-8XAP]. ↑

49. See, e.g., Conference of State Court Adm’rs, 2012-2013 Policy Paper Evidence-Based Pretrial Release 5–9 (2012–2013); Nat’l Ass’n Ctys Legislative Conference 2017, Adopted Interim Policy Resolutions, Resolution on Improving Pretrial Justice Process 11 (2017); Standards for Criminal Justice: Pretrial Release 10-1.10 (Am. Bar Ass’n). See generally Thomas H. Cohen & Christopher T. Lowenkamp, Revalidation of the Federal Pretrial Risk Assessment Instrument (PTRA): Testing the PTRA for Predictive Biases (June 15, 2018) (unpublished manuscript), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3197325 [https://perma.cc/NTW6-A2LA] (evaluating the effectiveness of the PTRA). ↑

50. See generally William M. Grove, et al., Clinical Versus Mechanical Prediction: A Meta-Analysis, 12 Psych. Assessment 19 (2000) (comparing clinical and mechanical predictions); Christopher Slobogin, A Jurisprudence of Dangerousness, 98 Nw. U. L. Rev. 1, 1–2 (2003) (providing background on the efficacy of determinations of danger in various trial situations). ↑

51. Jon Kleinberg et al., Human Decisions and Machine Predictions 2 (Nat’l Bureau of Econ. Research, Working Paper No. 23180, 2017) (“[A] policy simulation shows crime can be reduced by up to 24.8 percent with no change in jailing rates, or jail populations can be reduced by 42.0 percent with no increase in crime rates.”). ↑

52. Public Safety Assessment: Risk Factors and Formula, supra note 31; see also Jon Schuppe, Post Bail, NBC News (Aug. 22, 2017), https://www.nbcnews.com/specials/bail-reform [https://perma.cc/S46L-NQU7]. ↑

53. Public Safety Assessment: Risk Factors and Formula, supra note 31. ↑

54. Ostrom et al., supra note 16; Va. Criminal Sentencing Comm’n, 2010 Annual Report, supra note 16, at 38–41. ↑

55. See Monahan & Skeem, supra note 39, at 159–60; J.C. Oleson, Risk in Sentencing: Constitutionally Suspect Variables and Evidence-Based Sentencing, 64 S.M.U. L. Rev. 1329, 1399–402 (2011). See generally Jennifer Elek et al., Ctr. for Sentencing Initiatives, Using Risk and Needs Assessment Information at Sentencing: Observations from Ten Jurisdictions (2015) (surveying ten jurisdictions using risk-based instruments at sentencing). ↑

56. See Malenchik v. State, 928 N.E.2d 564, 571–73 (Ind. 2010); State v. Gauthier, 939 A.2d 77, 81, 85–86 (Me. 2007). ↑

57. Ky. Rev. Stat. Ann. § 532.007 (2019); Ohio Rev. Code Ann. § 5120.114 (2019); Okla. Stat. tit. 22, § 988.19 (2019); 42 Pa. Cons. Stat. § 2154.7 (2019); Wash. Rev. Code § 9.94A.500 (2019). See generally Rhys Hester, Risk Assessment at Sentencing: The Pennsylvania Experience, in Predictive Sentencing, 213 (Jan W. de Keijser et al. eds., 2019) (discussing Pennsylvania’s approach to using risk assessment in sentencing). ↑

58. Roger K. Warren, Evidence-Based Practice to Reduce Recidivism: Implications For State Judiciaries (2007). See generally Pamela M. Casey et al., Nat’l Ctr. for State Courts, Using Offender Risk and Needs Assessment Information at Sentencing (2011) (providing guidance to courts on using risk assessment in sentencing). ↑

59. Susan Turner & Julie Gerlinger, Risk Assessment and Realignment, 53 Santa Clara L. Rev. 1039, 1045–47 (2013). ↑

60. For an overview of the New York Legislation, see VERA Institute of Justice, New York, New York, Highlights of the 2019 Bail Reform Law (July 2019), at https://www.vera.org/downloads/publications/new-york-new-york-2019-bail-reform-law-highlights.pdf [https://perma.cc/NB3P-EY3Y]. ↑

61. Nat’l Ctr. for Juvenile Justice, State Juvenile Justice Profiles (2003), http://www.ncjj.org/stateprofiles [https://perma.cc/S7TS-6TEA]. Another study found that most states used some type of structured decision-making, but only a minority used empirically validated risk assessment. See Donna B. Towberman, A National Survey of Juvenile Risk Assessment, 43 Juv. & Fam. Ct. 61, 61, 67 (1992). ↑

62. Craig S. Schwalbe, Risk Assessment for Juvenile Justice: A Meta-Analysis, 31 L. & Hum Behav. 449, 458 (2007). ↑

63. See, e.g., David Robinson & Logan Koepke, Upturn, Stuck in a Pattern: Early Evidence on “Predictive Policing” and Civil Rights (2016); Mara Hvistendahl, Can ‘Predictive Policing’ Prevent Crime Before it Happens?, Science (Sept. 28, 2016), https://www.sciencemag.org/news/2016/09/can-predictive-policing-prevent-crime-it-happens [https://perma.cc/GX5V-QJ7Z]. For an overview of predictive policing methods and research, see Walter Perry et al., RAND Corp., Predictive Policing: The Role of Crime Forecasting in Law Enforcement Operations (2013). ↑

64. Nat’l Inst. of Corrections, Implementing Evidence-Based Policy and Practice in Community Corrections 2 (2d ed. 2009), https://s3.amazonaws.com/static.nicic.gov/Library/024107.pdf [https://perma.cc/N9X6-2V2K]. ↑

65. Evidence-Based Policies and Practices, N.Y.C. Probation, https://www1.nyc.gov/site/probation/about/evidence-based-policies-and-practices.page [https://perma.cc/9YLN-UYHB] (“Evidence-based policies and practices (EBPP) use current research and the best available data to guide decisions and produce the outcomes that our stakeholders—probation clients, victims, and communities—expect.”). ↑

66. Model Penal Code § 6.06, cmt. m (Am. Law Inst. Final Draft 2017). ↑

67. Stephen D. Gottfredson & Donald Gottfredson, Accuracy of Prediction Models, in Criminal Careers and “Career Criminals” 212, 247 (Alfred Blumstein et al. eds., 1986). ↑

68. Model Penal Code § 6B.09(2). ↑

69. Id. § 6B.09(1). ↑

70. Andreas Von Hirsch, Deserved Criminal Sentences (2017). ↑

71. Richard S. Frase, Just Sentencing: Principles and Procedures for a Workable System 8 (2013). ↑

72. See generally Christopher Slobogin, A Defense of Modern Risk-Based Sentencing, in Predictive Sentencing, supra note 55, at 107; Christopher Slobogin, Prevention as the Primary Goal of Sentencing: The Modern Case for Indeterminate Dispositions In Criminal Cases, 48 San Diego L. Rev. 1127 (2011) (discussing indeterminate sentencing entailing broad sentencing ranges and individualized assessment). ↑

73. See, e.g., Kevin R. Reitz, Sentencing, in Crime and Public Policy 467, 472 (James Q. Wilson & Joan Petersilia eds., 2011). ↑

74. Norval Morris, The Future of Imprisonment (1974). ↑

75. See Model Penal Code § 6B.09, cmt. d (Am. Law Inst. Final Draft 2017) (“From an actuarial perspective, attempts to identify persons of low recidivism risk are more often successful than attempts to identify persons who are unusually dangerous.”). See generally Richard A. Berk & Justin Bleich, Statistical Procedures for Forecasting Criminal Behavior: A Comparative Assessment, 12 Criminology & Pub. Pol’y 513 (2013). ↑

76. Bernard E. Harcourt, Against Prediction: Profiling, Policing, and Punishing in an Actuarial Age (2007). ↑

77. Model Penal Code § 6B.09(1). ↑

78. .See Sarah L. Desmarais et al., Performance of Recidivism Risk Assessment Instruments in U.S. Correctional Settings, 13 Psychol. Servs. 206, 216 (2016) (“For most instruments, predictive validity had been evaluated in one or two studies that met our inclusion criteria. Those studies often were completed by the developers of the instrument under investigation. Perhaps one our most striking findings, only two of the 53 studies reported on the interrater reliability of the risk assessments.”). See generally Handbook of Recidivism Risk Assessment (Jay P. Singh et al. eds., 2018) (surveying risk assessment instruments in US correctional settings). ↑

79. Barajas, supra note 5. ↑

80. See Richard Berk et al., Fairness in Criminal Justice Risk Assessments: The State of the Art, Soc. Methods & Res. 1 (2018). See generally Alexandra Chouldechova, Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments, 5 Big Data 153 (2017) (discussing fairness and bias in risk assessment instruments). ↑

81. For a discussion of the constitutional claims potentially raised in the area, see Aziz Z. Huq, Racial Equity in Algorithmic Criminal Justice, 68 Duke L. J. 1043 (2019). ↑

82. See Harcourt, supra note 74. ↑

83. See Monahan & Skeem, supra note 39, at 163. ↑

84. Starr, Evidence-Based Sentencing and the Scientific Rationalization of Discrimination, supra note 1, at 824, 827; see also Oleson, supra note 53, at 1395–98 (discussing options available to judges considering evidence-based sentencing, including those options involving the use of suspect variables like gender). ↑

85. *See generally Jennifer L. Skeem et al., Gender, Risk Assessment, and Sanctioning: The Cost of Treating Women Like Men, 40 Law & Hum. Behav. *580 (2016). ↑

86. See Northpointe Software Suite, Equivant, https://www.equivant.com/northpointe-suite/ [https://perma.cc/H2F2-PTCU]. ↑

87. Angwin et al., supra note 26. ↑

88. Id. But see Anthony W. Flores et al., False Positives, False Negatives, and False Analyses: A Rejoinder to “Machine Bias: There’s Software Used Across the Country to Predict Future Criminals. And it’s Biased Against Blacks.”, 80 Fed. Prob. 38 (2016) (rebutting Angwin article for contradicting studies indicating that race and gender bias is avoidable); Jennifer L. Skeem & Christopher T. Lowenkamp, Risk, Race, and Recidivism: Predictive Bias and Disparate Impact, 54 Criminology 680 (2016) (finding that claims that risk assessment tools suffer from racial bias are exaggerated). ↑

89. See generally State v. Loomis, 881 N.W.2d 749 (Wis. 2016), cert. denied, 137 S. Ct. 2290 (2017); Alyssa M. Carlson, The Need for Transparency in the Age of Predictive Sentencing Algorithms, 103 Iowa L. Rev. 303, 316, 320–22 (2017) (discussing Loomis). ↑

90. See, e.g., Jessica M. Eaglin, Against Neorehabilitation, 66 S.M.U. L. Rev. 189 (2013); see also Jacob Kang-Brown et al., Vera Inst. of Justice, The New Dynamics of Mass Incarceration (2018) (discussing the failure of reformed sentencing policies to reduce the risk of incarceration); Katherine Beckett, The Politics, Promise, and Peril of Criminal Justice Reform in the Context of Mass Incarceration, 1 Ann. Rev. Criminology 235 (2018). ↑

91. Slobogin, A Defense of Modern Risk-Based Sentencing, supra note 70. ↑

92. Model Penal Code § 6B.09(3) (Am. Law Inst. Final Draft 2017). ↑

93. Model Penal Code § 6B.09 cmts. a, e. ↑

94. See generally Jorge G. Varela et. al., Same Score, Different Message: Perceptions of Offender Risk Depend on Static-99R Risk Communication Format, 38 Law & Hum. Behav. 418–27 (2014).
 ↑

95. Gottfredson & Gottfredson, supra note 65, at 247. ↑

96. Model Penal Code: Art. 6B, reporter’s note d. ↑

97. Reitz, supra note 13, at 70. ↑

98. Id. at 70–71. ↑

99. Id. at 71. ↑

100. Richard P. Kern, Overview of Virginia’s Truth-in-Sentencing System 15, 20 (PowerPoint presentation), http://sfc.virginia.gov/pdf/committee_meeting_presentations/June%2019%20meeting/Virginia%20Felony%20Sentencing%20Guidelines.pdf [https://perma.cc/K4D8-RWTG]. ↑

101. Reitz, supra note 13, at 70; see also Richard P. Kern & Mark H. Bergstrom, A View from the Field: Practitioners’ Response to Actuarial Sentencing: An “Unsettled” Proposition, 25 Fed. Sent’g Rep. 185, 188 (2013) (explaining that the adoption of risk assessment in Virginia was driven in large part by “budgetary concerns”); Richard Kern & Meredith Farrer-Owens, Sentencing Guidelines with Integrated Offender Risk Assessment, 16 Fed. Sent’g. Rep. 165, 169 (2004) (“The non-violent risk assessment tool adopted as part of the discretionary sentencing guidelines serves to safely divert a significant share of low risk felons away from expensive prison beds into less costly alternative punishment programs.”). ↑

102. For a description of this process, see Va. Criminal Sentencing Comm’n, 2004 Annual Report 35 (2004). ↑

103. Kern & Farrar-Owens, supra note 99, at 165; Meredith Farrar-Owens, The Evolution of Sentencing Guidelines in Virginia: An Example of the Importance of Standardized and Automated Felony Sentencing Data, 25 Fed. Sent’g. Rep. 168, 170 (2013). ↑

104. Kern, supra note 98, at 15, 20. ↑

105. 2004 Annual Report, supra note 100, at 35. The risk factors included on the original assessment tool developed by the VCSC consisted of six types of variables: offense type, whether the offender is currently charged with an additional offense, “offender characteristics” (i.e., gender, age, employment, and marital status), whether the offender had been arrested or confined within the past 18 months, prior felony convictions, and prior adult incarcerations. ↑

106. Nat’l Ctr. for State Courts & Va. Criminal Sentencing Comm’n, Offender Risk Assessment in Virginia: A Three Stage Evaluation (2002), http://www.vcsc.virginia.gov/risk_off_rpt.pdf [https://perma.cc/2L2Y-LB8M]. ↑

107. 2004 Annual Report, supra note 100, at 35, 78, 84 (“According to the Commission’s data, less than 17 percent of the offenders recommended for an alternative sanction by the risk instrument were identified as recidivists.”). ↑

108. Va. Criminal Sentencing Comm’n, Re-validation of the Nonviolent Offender Risk Assessment Instrument: Study Update (2011), http://www.vcsc.virginia.gov/Nonviolent%20Offender%20Risk%20Assessment%20Update%2011-14-11%20HANDOUT.pdf [https://perma.cc/9N73-A8RY] (PowerPoint presentation). ↑

109. In these samples, 63 percent of drug offenders scored in the low-risk group and 37 percent scored in a higher-risk group, while 43 percent of the larceny/fraud offenders scored in the low-risk group and 57 percent scored in a higher-risk group. Recidivism in this research was defined as reconviction for a felony offense within three years of release from incarceration. Of drug offenders designated as low risk, 12 percent recidivated; by comparison, 44 percent of higher-risk drug offenders recidivated. Of larceny/fraud offenders designated as low risk, 19 percent recidivated; by comparison, 38 percent of higher-risk larceny/fraud offenders recidivated. Va. Criminal Sentencing Comm’n, 2012 Annual Report (2012), http://www.vcsc.virginia.gov/2012VCSCAnnualReport.pdf [https://perma.cc/C948-4AF8]. ↑

110. Id. ↑

111. Va. Sentencing Guidelines, Larceny, Section D., http://www.vcsc.virginia.gov/worksheets_2012/Larceny.pdf [https://perma.cc/YVB8-9LWV]. ↑

112. Va. Sentencing Guidelines, Drug/Schedule I/II, Section D., http://www.vcsc.virginia.gov/worksheets_2016/Drug-I_II2.pdf [https://perma.cc/Y7MA-2CZX]. ↑

113. Farrar-Owens, supra note 101, at 170. See generally Re-validation of the Nonviolent Offender Risk Assessment Instrument: Study Update, supra note 106, at 4 (summarizing data from 2003 to 2010, and showing beginning in 2005, a consistent group of about 50 percent of eligible low-risk offenders were sentenced to alternative sentences). ↑

114. These data were presented to the Virginia Criminal Sentencing Commission at its April 9, 2018 meeting. The materials presented, including PowerPoint, are available on the VCSC website: http://www.vcsc.virginia.gov/meetings.html [https://perma.cc/ANZ6-K5R9]. ↑

115. Va. Criminal Sentencing Comm’n, 2016 Annual Report 14 (2016), http://www.vcsc.virginia.gov/2016Annualreportfinal.pdf [https://perma.cc/K9PS-YARX]. ↑

116. The cases in which the NVRA was missing are systematically different than those in which the NVRA was filled out in the following main ways: the sentencing information was far more likely to be prepared by a commonwealth attorney (83 percent vs. 53 percent) and the cases were far more likely to include a written plea agreement (62 percent vs. 39 percent) and/or a guilty plea (94 percent vs. 87 percent). It is possible that commonwealth attorneys and defense attorneys sometimes considered the NVRA when negotiating plea bargains, even if it was not filled out. Ongoing research aims to assess what role the NVRA plays in plea bargaining in Virginia. See John Monahan et al., Risk Assessment in Sentencing and Plea Bargaining: The Roles of Prosecutors and Defense Attorneys, Behav. Sci. & L. 1 (2019). ↑

117. The median prison sentence for property crimes and for drug crimes in Virginia in 2016 was thirty months. See Va. Criminal Sentencing Comm’n, Crime and Criminal Justice Trends in Virginia 23 (2018), http://www.vcsc.virginia.gov/2018meetings/Criminal%20Justice%20Trends%20in%20Va%2009-10-2018.pdf [https://perma.cc/6Q2P-ECVZ] (PowerPoint presentation). ↑

118. Steven L. Chanenson & Jordan M. Hyatt, The Use of Risk Assessment at Sentencing: Implications for Research and Policy 10 (2016), https://digitalcommons.law.villanova.edu/cgi/viewcontent.cgi?referer=https://www.google.com/&httpsredir=1&article=1201&context=wps [https://perma.cc/85BJ-KM5Q]. ↑

119. Id. ↑

120. Id. ↑

121. Jeffrey J. Shook & Rosemary C. Sarri, Structured Decision Making in Juvenile Justice: Judges’ and Probation Officers’ Perceptions and Use, 29 Child. & Youth Servs. Rev. 1335, 1344 (2007). ↑

122. Id. at 1335. ↑

123. We are grateful to Chief Justice Donald Lemons of the Virginia Supreme Court, who provided a cover letter encouraging Virginia judges to participate in the survey. These data were presented to the Virginia Criminal Sentencing Commission at its April 9, 2018, meeting. The materials, including a PowerPoint, are available on the Commission’s website: http://www.vcsc.virginia.gov/meetings.html [https://perma.cc/3VT9-T7GT]. Statistical analyses of survey data can be found in John Monahan et al., Judicial Appraisals of Risk Assessment in Sentencing, 36 Behav. Sci. L. 565 (2018); see also Anne Metz et al., Risk and Resources: A Qualitative Perspective on Low-Level Sentencing in Virginia, 47 J. Community Psych. 1476 (2019). ↑

124. Twenty-four judges were interviewed. See Metz et al., supra note 121. ↑

125. See, e.g., U.S. Sentencing Comm’n, Results of 2014 Survey of United States District Judges: Modification and Revocation of Probation and Supervised Release 3 (2015), https://www.ussc.gov/sites/default/files/pdf/research-and-publications/research-projects-and-surveys/surveys/20150225_Judges_Survey.pdf [https://perma.cc/CZ5P-APWB]. ↑

126. First, the survey asked Virginia judges whether sentencing of drug and property offenders should be based only on the seriousness of the crime, or also on the risk that the offender will commit another crime in the future. Second, the survey asked how familiar judges were with the use of the NVRA for sentencing drug and property offenders in Virginia. Third, the survey asked how often the judge considered the results of the NVRA before sentencing a drug or property offender. Fourth, the survey asked whether judges relied on their judicial experience or on the NVRA to determine the risk that an offender would commit another time. Fifth, the survey asked how judges would rate the current availability of alternative sanctions, such as outpatient drug or mental health programs. Sixth, the survey asked whether availability of more alternative sanctions would change sentencing practices. Finally, the survey asked whether it would affect the use of alternative sanctions if the NVRA were presumptive and if judges were required to provide a reason in writing for declining to impose an alternative sanction. We also asked judges whether they would favor the adoption of such a procedure. ↑

127. See generally Brandon L. Garrett et al., Judicial Reliance on Risk Assessment in Sentencing Drug and Property Offenders: A Test of the Treatment Resource Hypothesis, 46 Crim. Just. & Behav. 799 (2019). ↑

128. We asked the Virginia Criminal Sentencing Commission to share the training that judges receive on the use of the NVRA. That training consisted of two PowerPoint slides that are a part of a twenty-six slide pre-bench training that newly appointed judges receive. That training briefly explains what the NVRA is, states that alternative sanctions are possible for eligible offenders, and notes that “compliance with the risk assessment recommendation is discretionary.” In addition, failure to follow sentencing guidelines is not reviewable on appeal in Virginia. No further judicial training is provided on the NVRA at present. ↑

129. John Clark, Pretrial Justice Inst., The Fundamentals of Pretrial Risk Assessment: Achieving Inter-Rater Reliability 10 (2014) (listing Colorado, Delaware, Hawaii, Kentucky, New Jersey, and West Virginia as states that used pretrial risk assessment at that time); Shaila Dewan, Judges Replacing Conjecture with Formula for Bail, N.Y. Times (June 26, 2015), http://www.nytimes.com/2015/06/27/us/turning-the-granting-of-bail-into-a-science.html [https://perma.cc/BU88-NL2H] (describing use of pretrial risk assessment in twenty-one states). ↑

130. Ky. Rev. Stat. Ann. § 431.510(1) (2019). ↑

131. Ky. Rev. Stat. Ann. § 532.007; see also § 196.288 (requiring budgetary funds for measurement and documentation of cost savings resulting from reduced incarceration); John D. Minton, Jr. et al., Report on Impact of House Bill 463: Outcomes, Challenges and Recommendations 4 (2012). ↑

132. Ky. R. Crim. Pro. 4.10. ↑

133. Laura & John Arnold Found., Results from the First Six Months of the Public Safety Assessment – Court in Kentucky (2014). ↑

134. See Robert Veldman, Pretrial Detention in Kentucky: An Analysis of the Impact of House Bill 463 During the First Two Years of its Implementation, 102 Ky. L. J. 777, 778 (2013). ↑

135. Alysia Santo, Kentucky’s Protracted Struggle to Get Rid of Bail, Marshall Project (Nov. 12, 2015), https://www.themarshallproject.org/2015/11/12/kentucky-s-protracted-struggle-to-get-rid-of-bail [https://perma.cc/4X7F-WNP9]. ↑

136. Id. ↑

137. Amended Order, In Re: Authorization for the Non-Financial Uniform Schedule of Bail Administrative Release Program, Ky. Sup. Ct. (2016). ↑

138. Stevenson, supra note 14, at 346. ↑

139. Id. at 352–53. ↑

140. Id. at 352–56. ↑

141. Id. at 357. ↑

142. Id. at 359–60. ↑

143. Id. at 364. ↑

144. Id. ↑

145. James M. Doyle, Fewer Prisoners, Less Crime? The Elusive Promise of Algorithms, Crime Report (Nov. 14, 2017), https://thecrimereport.org/2017/11/14/fewer-prisoners-less-crime-the-elusive-promise-of-algorithms [https://perma.cc/7U7D-BH4L] (internal quotation marks omitted). ↑

146. For an overview of the New Jersey approach, see Public Safety Assessment, New Jersey Risk Factor Definitions – December 2018 (2018). ↑

147. See N.J. Courts, Initial Release Decisions for Criminal Justice Reform Eligible Defendants, January 1, 2018 – December 31, 2018 (2018), https://www.njcourts.gov/courts/assets/criminal/cjrreport2018.pdf?cacheID=i1yaARV [https://perma.cc/W7XU-XWED]. ↑

148. Id. ↑

149. S.P. Sullivan, The Good News: N.J. Bail Overhaul is Working. The Bad News: It’s Already Going Broke, NJ.com (Feb. 12, 2018), https://www.nj.com/politics/2018/02/report_finds_nj_bail_reform_is_working_--_but_its.html [https://perma.cc/7AXP-P6SZ]. ↑

150. See N.J. Courts, Pretrial Release Recommendation Decision Making Framework (DMF) (2018), https://www.njcourts.gov/courts/assets/criminal/decmakframwork.pdf?cacheID=7mEYXdb [https://perma.cc/8X3X-XQFA]. ↑

151. Sullivan, supra note 147. ↑

152. For a recent study of pretrial risk assessment in 30 jurisdictions, see Matthew DeMichele et al., What do Criminal Justice Professionals Think About Risk Assessment at Pretrial?, 83 Fed. Prob. 32 (2019). ↑

153. Chanenson & Hyatt, supra note 116, at 7. ↑

154. Id. at 11. ↑

155. Frank Main, Cook County Judges Not Following Bail Recommendations: Study, Chicago Sun-Times (July 3, 2016), https://chicago.suntimes.com/2016/7/3/18325456/cook-county-judges-not-following-bail-recommendations-study [https://perma.cc/6RTV-8L92]. ↑

156. Id. ↑

157. Id. ↑

158. Lynh Bui, Reforms Intended to End Excessive Cash Bail in Md. Are Keeping More in Jail Longer, Report Says, Wash. Post (July 2, 2018), https://www.washingtonpost.com/local/public-safety/reforms-intended-to-end-excessive-cash-bail-in-md-are-keeping-more-in-jail-longer-report-says/2018/07/02/bb97b306-731d-11e8-b4b7-308400242c2e_story.html [https://perma.cc/6C2Z-SVTZ]. ↑

159. Id. ↑

160. Id.; see also Sarah Picard et al., Ctr. for Court Innovation, Beyond the Algorithm: Pretrial Reform, Risk Assessment, and Racial Fairness (2019). ↑

161. Berk, supra note 14, at 193. ↑

162. Id. at 212. ↑

163. Id. at 220. ↑

164. Richard A. Berk et al., A Randomized Experiment Testing Inmate Classification Systems, 2 Criminology & Pub. Pol’y 215, 240–41 (2003). ↑

165. Jill Viglione et al., Misalignment in Supervision: Implementing Risk/Needs Assessment Instruments in Probation, 42 Crim. Just. & Behav. 263, 263 (2015); see also Joel Miller & Carrie Maloney, Practitioner Compliance With Risk/Needs Assessment Tools: A Theoretical and Empirical Assessment, 40 Crim. Just. & Behav. 716, 716 (2013) (surveying a national sample of probation and parole officers and finding that half “were ‘formal’ in their compliance [with risk assessment instruments]: filling out the tools, but often making decisions that did not correspond with tool results”). ↑

166. N.C. Comm’n Admin. Justice & Law, Criminal Investigation and Adjudication Committee Report: Evidence-Based Recommendations to Improve the State’s Criminal Justice System, Appendix C, Pre-Trial Justice 40 n.121 (2016) [hereinafter NCCALJ Final Report], https://nccalj.org/wp-content/uploads/2017/pdf/nccalj_criminal_investigation_and_adjudication_committee_report_pretrial_justice.pdf [https://perma.cc/WM4J-MZDS]. ↑

167. See id. at 32; see also N.C. Gen. Stat. § 15A-533 (2019) (outlining a criminal defendant’s right to pretrial release in capital and noncapital cases). ↑

168. N.C. Gen. Stat. § 15A-534(d2)(1), (d3). ↑

169. NCCALJ Final Report, supra note 164, at 34, 48. ↑

170. For example, Orange County, NC uses each of these mechanisms. Pretrial Release, Orange County, Crim. Just. Resource Dep’t, https://www.orangecountync.gov/439/Pretrial-Release [https://perma.cc/MFN5-G7FG]. ↑

171. Jeffrey J. Rachlinski & Andrew J. Wistrich, Judging the Judiciary by the Numbers: Empirical Research on Judges, 13 Ann. Rev. L. & Soc. Sci. 203, 204 (2017). ↑

172. See generally Nat’l Research Council, Strengthening Forensic Science in the United States: A Path Forward 106 (2009) (“Review of reported judicial opinions reveals that, at least in criminal cases, forensic science evidence is not routinely scrutinized pursuant to the [applicable] standard of reliability . . .”); Stephanie L. Damon-Moore, Trial Judges and the Forensic Science Problem, 92 N.Y.U. L. Rev. 1532 (2017). ↑

173. See generally Brandon L. Garrett & M. Chris Fabricant, The Myth of the Reliability Test, 86 Fordham L. Rev. 1559 (2018). ↑

174. N. Zoe Hilton et al., Communicating the Risk of Violent and Offending Behavior: Review and Introduction to this Special Issue, 33 Behav. Sci. & L. 1 (2015). ↑

175. Id. at 2 (summarizing literature). ↑

176. Daniel A. Krauss et al., Risk Assessment Communication Difficulties: An Empirical Examination of the Effects of Categorical Versus Probabilistic Risk Communication in Sexually Violent Predator Decisions. 36 Behav. Sci. & L. 532 (2018); Varela et al., supra note 92. ↑

177. Hilton et al., supra note 172, at 3 (summarizing literature in the healthcare context). ↑

178. Id. (summarizing literature on numeracy or specifically “risk illiteracy”). ↑

179. Many studies have shown how participants’ risk perception differs when risk is framed as a frequency versus a percentage. See, e.g., Nicolai Bodemer et al., Communicating Relative Risk Changes with Baseline Risk: Presentation Format and Numeracy Matter, 34 Med. Decision Making 615 (2014); Leam A. Craig & Anthony Beech, Best Practice in Conducting Actuarial Risk Assessments with Adult Sexual Offenders, 15 J. Sexual Aggression 193, 197 (2009); Hilton et al., supra note 172, at 3; Nicholas Scurich & Richard S. John, The Effect of Framing Actuarial Risk Probabilities on Involuntary Civil Commitment Decisions, 35 Law & Hum. Behav. 83, 88 (2011); Nicholas Scurich et al., Innumeracy and Unpacking: Bridging the Nomothetic/Idiographic Divide in Violence Risk Assessment, 36 Law & Hum. Behav. 548, 549 (2012); Paul Slovic et al., Violence Risk Assessment and Risk Communication: The Effects of Using Actual Cases, Providing Instruction, and Employing Probability Versus Frequency Formats, 24 Law & Hum. Behav. 271, 290 (2000). Additional research suggests categorical presentations are still more effective, although they raise questions concerning the definitions and consequences of the categories. ↑

180. See, e.g., Stephanie A. Evans & Karen L. Salekin, Involuntary Civil Commitment: Communicating with the Court Regarding “Danger to Other,” 38 Law & Hum. Behav. 325, 333 (2014) (finding in a survey of judges that judges gave the same weight to frequency-based and probabilistic presentations but categorical messages were viewed as more probative); Hilton et al., supra note 172 at 7–8; Nicholas Scurich, The Case Against Categorical Risk Estimates, 36 Behav. Sci. & L. 554, 556 (2018). ↑

181. See Reitz, supra note 13, at 70–71. ↑

182. See Eaglin, supra note 26, at 61, 117. ↑

183. Am. Civil Liberties Union N.J., Nat’l Ass’n Criminal Def. Lawyers, N.J. Pub. Defs., The New Jersey Pretrial Justice Manual, 11 (2016); Hilton et al., supra note 172, at 3 (summarizing literature on use of graphs to communicate risk). ↑

184. See Hilton et al., supra note 172, at 6. ↑

185. For an excellent summary of the literature, see Rachlinski & Wistrich, supra note 169. ↑

186. Christal Hayes & John Bacon, Judge Aaron Persky, Who Gave Brock Turner Lenient Sentence in Rape Case, Recalled From Office, USA Today (June 6, 2018), https://eu.usatoday.com/story/news/2018/06/06/judge-aaron-persky-who-gave-brock-turners-lenient-sentence-sanford-rape-case-recalled/674551002 [https://perma.cc/G6Z2-H66V]. ↑

187. Rachlinski & Wistrich, supra note 169, at 212–13. ↑

188. Id. at 213. ↑

189. Id. at 221. ↑

190. Pennsylvania Comm’n on Sentencing, Risk/Needs Assessment Project, Interim Report 8: Communicating Risk at Sentencing (2014). The study, however, received responses from only 21 percent or 200 of the 962 surveyed. Id. at 6. That study found that respondents found risk information easiest to understand and interpret when receiving full information and reported that respondents found that tables were more useful than graphs. Id. at 12; see also R. Barry Ruback et al., Communicating Risk Information at Sentencing in Pennsylvania: An Experimental Analysis, 80 Fed. Prob. 47, 51–52 (2016) (discussing the way recidivism risk information affected the post-risk judgments for burglary crimes). ↑

191. American Council of Chief Defenders, Policy Statement on Fair and Effective Pretrial Justice Practices 5 (2011); Douglas L. Colbert et al., Do Attorneys Really Matter? The Empirical and Legal Case for the Right of Counsel at Bail, 23 Cardozo L. Rev. 1719, 1719 (2002). ↑

192. Va. Criminal Sentencing Comm’n, 2019 Annual Report 12 (2019), http://www.vcsc.virginia.gov/2019VCSCAnnualReport.pdf [https://perma.cc/Y9BK-3D73] (“When risk assessment for nonviolent offenders is applicable, a judge may sentence a recommended offender to an alternative punishment program or to a term of incarceration within the traditional guidelines range and be considered in strict concurrence.”). ↑

193. Scurich & John, supra note 177, at 84. ↑

194. Rachlinski & Wishtrich, supra note 169, at 215. ↑

195. See Stevenson, supra note 14, at 369–72. ↑

196. Dorothy Weldon, More Appealing: Reforming Bail Review in State Courts, 118 Colum. L. Rev. 2401, 2419–36 (2018). ↑

197. See Sentencing Worksheets and Interactive File Transfer (SWIFT), Va. Sentencing Criminal Commission, http://www.vcsc.virginia.gov/swift.html [https://perma.cc/KT9Q-5D2C]. ↑

198. For an excellent overview of a wide range of screening and assessment instruments used in the area of co-occurring mental and substance abuse disorders, see Substance Abuse & Mental Health Servs. Admin. (SAMHSA), Screening and Assessment of Co-Occurring Disorders in the Justice System (2015). ↑

199. See Shook & Sarri, supra note 119, at 1342 (describing interviews with juvenile justice professionals in several states and describing how 35 percent described using three forms of decision-making: risk assessment, needs assessment, and security classification). ↑

200. Monahan & Skeem, supra note 1, at 497. ↑

201. See id. at 498 (noting difficulty in identifying such causal risk factors where many programs “are aimed at multiple factors at the same time in a ‘blunderbuss fashion’” that makes it difficult to conduct randomized controlled trials). ↑

202. See, e.g., Pretrial Justice Inst., 2009 Survey of Pretrial Services Programs 50 (2009) (describing results of a survey of pretrial programs that include messaging options); Stanford Legal Design Lab, 
 https://law.stanford.edu/organizations/pages/legal-design-lab/#slsnav-our-4-themes [https://perma.cc/G5LS-CQ4Z] (describing a software application designed to send automated text message reminders to youth involved in juvenile justice proceedings in order to improve Failure to Appear rate). ↑

203. Nat’l Ctr. for State Courts & Va. Criminal Sentencing Comm’n, supra note 104, at 5. ↑

204. See Turner & Gerlinger, supra note 57. ↑

205. ODonnell v. Harris County, 892 F.3d 147, 152 (5th Cir. 2018). ↑

206. Id. at 154. ↑

207. See id. at 153. ↑

208. Id. at 159. ↑

209. Id. at 160 (“We decline to hold that the Constitution requires the County to produce 50,000 written opinions per year to satisfy due process.”). The Court also relied on empirical analysis of outcomes in the county. Heaton et al., supra note 7, at 786–87. ↑

210. ODonnell, 892 F.3d at 159. ↑

211. The Fifth Circuit largely relied on a due process analysis; for the critique that the proper analysis implicates both Equal Protection and Due process analysis, see Brandon L. Garrett, Wealth, Equality, and Due Process (Jan. 18, 2019) (unpublished manuscript) (draft on file with author). ↑

212. See ODonnell v. Harris County, 892 F.3d 147, 161 (5th Cir. 2018). ↑

213. See, e.g., Walker v. City of Calhoun, 901 F.3d 1245 (11th Cir. 2018) (concluding that a procedural due process analysis did not warrant such relief because a minimally adequate hearing was provided); Schultz v. Alabama, 330 F. Supp. 3d 1344 (N.D. Ala. 2018) (distinguishing Walker and granting preliminary injunction to enjoin practice of pretrial bail based on categorical guidelines for pretrial decision-making). ↑

214. U.S. Dep’t of Justice, The First Step Act of 2018: Risk and Needs Assessment System, at iv (2019), https://www.nij.gov/documents/the-first-step-act-of-2018-risk-and-needs-assessment-system.pdf [https://perma.cc/2HU6-9F2L]. ↑

215. Id. at 43. ↑

216. Id. at 43–49. ↑

217. Id. at 45. ↑

218. Id. ↑

219. See id. at 49 (referring to the gender-specific validation of the PATTERN instrument, but referencing biological sex as the differentiating characteristic). ↑

220. Id. at 50–52. ↑

221. See 18 U.S.C. § 3631(a) (2018) (internally identified as Sec. 107). ↑

222. See id. § 3632(a)(5)(B); see also U.S. Dep’t of Justice, supra note 212, at iv (allocating funding for additional programming in prisons). ↑

223. See 18 U.S.C. § 3632(a); U.S. Dep’t of Justice, supra note 212. For comments on the instrument, see, for example, Comment Letter to Department of Justice on PATTERN First Step Act, Leadership Conf. for Civ. & Hum. Rts. (Sept. 3, 2019), https://civilrights.org/resource/comment-letter-to-department-of-justice-on-pattern-first-step-act [https://perma.cc/6CFQ-P7D5]; Brandon L. Garrett & Megan T. Stevenson, Comment on PATTERN (Sept. 15, 2019), https://sites.law.duke.edu/justsciencelab/2019/09/15/comment-on-pattern-by-brandon-l-garrett-megan-t-stevenson [https://perma.cc/7S7D-N7U8]. ↑

224. 18 U.S.C. § 3632(f); id. § 3621 (internally identified as Sec. 103(5) and appended at the end of 18 U.S.C. § 3621 under “GAO Report”) (calling for evaluation of whether “officers and employees of the Bureau of Prisons are receiving the training described in section 3632(f)”). ↑

225. Id. § 3621 (internally identified as Sec. 103(5) and appended at the end of 18 U.S.C. § 3621 under “GAO Report”) (“Not later than 2 years after the Director of the Bureau of Prisons implements the risk and needs assessment system under section 3621 of title 18, United States Code, and every 2 years thereafter, the Comptroller General of the United States shall conduct an audit of the use of the risk and needs assessment system at Bureau of Prisons facilities.”). ↑

226. See Brandon L. Garrett, Federal Risk Assessment, 41 Cardozo L. Rev. 120–21 (2019) (discussing concerns with design and implementation of the PATTERN). ↑

227. See generally U.S. Dep’t of Justice, supra note 212, at 40–93 (laying out a plan for implementing the PATTERN in order to assess the risk and needs of all BOP prisoners by January 2020). ↑

228. See Rachel E. Barkow, Institutional Design and the Policing of Prosecutors: Lessons from Administrative Law, 61 Stan. L. Rev. 869, 869–74 (2009); Andrew Manuel Crespo, Systemic Facts: Toward Institutional Awareness in Criminal Courts, 129 Harv. L. Rev. 2049, 2050 (2016); Barry Friedman & Maria Ponomarenko, Democratic Policing, 90 N.Y.U. L. Rev. 1827, 1865 (2015); Tracey L. Meares & Bernard E. Harcourt, Foreword: Transparent Adjudication and Social Science Research in Constitutional Criminal Procedure, 90 J. Crim. L. & Criminology 733, 743–44 (2000); Slobogin, supra note 33, at 593–96; Christopher Slobogin, Policing as Administration, 165 U. Pa. L. Rev. 91, 91 (2016). ↑

229. See Press Release, Mesa County Leads Innovation Behind Scientific Criminal Justice Pretrial Innovations, Mesa County (Aug. 28, 2013), https://info.nicic.gov/sites/info.nicic.gov.ebdm/files/docs/Mesa-County-NACOPR.pdf [https://perma.cc/7UFY-TGKP]. ↑

230. Va. Code Ann. § 17.1-803 (2019). ↑

231. Virginia, however, did not develop a framework for judges to use when deciding how to incorporate risk into decision-making in sentencing. This is a contrast to the pretrial risk instrument used in the state, the Virginia Pretrial Risk Assessment Instrument, which is accompanied by a matrix that gives decision-makers a guide as to how to incorporate risk assessment information into pretrial decisions. See Kenneth Rose, Va. Dep’t of Criminal Justice Servs., Pretrial Services Agencies: Risk-Informed Pretrial Decision Making in the Commonwealth of Virginia (2016), http://vscc.virginia.gov/Virginia%20Pretrial%20Services%20Presentation%2012-5-2016.pdf [https://perma.cc/CF3H-FPWV] (PowerPoint presentation). ↑

232. See Hester, supra note 55, at 219 (describing transparency in the Pennsylvania decision-making process). ↑

233. ODonnell v. Harris County, 251 F.Supp.3d 1052, 1125 (S.D. Tex. 2017). ↑

234. Meagan Flynn, Harris County Bail Systems Offers Little Help to Defendants Who Most Need it, Cases Reveal, Houston Chronicle (Jan. 22, 2018), https://www.houstonchronicle.com/news/houston-texas/houston/article/Harris-County-bail-system-shortchanges-defendants-12516456.php [https://perma.cc/623K-NYRV]. ↑

235. Gregory Yee, How Millions Spent on Criminal Justice Reform in Charleston is Paying Off, Post & Courier (Dec. 29, 2018), https://www.postandcourier.com/news/how-millions-spent-on-criminal-justice-reform-in-charleston-is/article_05437456-d8a2-11e8-8dc2-aba07b4bc789.html [https://perma.cc/J5XS-DH8E] (describing nearly 25 percent reduction in jail population due to a range of initiatives, including a diversion center called “The Stabilization Center,” as well as a “risk assessment tool, expansion of public defenders to bond court and other initiatives”). ↑

236. John Logan Koepke & David G. Robinson, Danger Ahead: Risk Assessment and the Future of Bail Reform, 93 Wash. L. Rev. 1725, 1725 (2018) (criticizing “zombie predictions” relying on outdated data). See generally Yee, supra note 233 (discussing how utilizing outdated data and older bail regimes can result in risk assessment providing mathematical legitimacy to historical biases that it is intended to mitigate). ↑

237. State v. Guise, 921 N.W.2d 26, 32 (Iowa 2018). ↑

238. ODonnell v. Harris County, 892 F.3d 147, 159 (5th Cir. 2018). ↑

239. Id. The Fifth Circuit also rejected abstention in the case, where no case-specific remedies were sought. Id. at 156. ↑

240. See Greenholtz v. Inmates of Neb. Penal & Corr. Complex, 442 U.S. 1, 9–11 (1979) (finding no liberty interest in parole decision-making); see also Kimberly Thomas & Paul Reingold, From Grace to Grids: Rethinking Due Process Protection for Parole, 107 J. Crim. L. & Criminology 213, 214 (2017) (discussing how prisoners at parole receive substantially less due process rights than defendants at sentencing). ↑

241. ODonnell v. Harris County, 251 F.Supp.3d 1052, 1106 (S.D. Tex. 2017) (citing Heaton et al., supra note 7). ↑

242. Id. at 1120–21. ↑

243. Id. at 1124–25 (discussing the County’s adoption of “the Arnold Tool,” which refers to the PSA). ↑

244. See Mary Moreno, Harris County Should be Proud of Bail Reform Package, Houston Chronicle (Aug. 3, 2019), https://www.houstonchronicle.com/opinion/outlook/article/Harris-County-should-be-proud-of-bail-reform-14277405.php [https://perma.cc/6KHM-A4PK]. ↑

245. See Koepke & Robinson, supra note 234, at 1796 (“By tracking concurrence, divergence, and why a judge diverged, policymakers may be able to create a positive feedback loop. The more that judges understand how a risk assessment tool works, and the more that the developers of a risk assessment tool understand how judges use—or do not use—their tool, the better.”); see also Jodi L. Viljoen et al., Do Risk Assessment Tools Help Manage and Reduce Risk of Violence and Reoffending? A Systematic Review, 42 Law & Hum. Behav. 181, 210–11 (2018) (“[R]ather than focusing exclusively on predictive validity studies and the development of new [risk assessment] tools, researchers need to pay greater attention to how tools are applied to guide real-world decisions, such as by testing the pathways between risk assessment and risk management, identifying areas of slippage, and developing strategies to facilitate the ability of risk assessments to translate into better risk management efforts. Such initiatives are essential to ensuring that the potential value of risk assessment is more fully realized.”). ↑

246. Shook & Sarri, supra note 119, at 1347. ↑

247. L. Maaike Helmus & Kelly M. Babchishin, Primer on Risk Assessment and the Statistics Used to Evaluate its Accuracy, 44 Crim. Just. & Behav. 8, 8–9 (2017). ↑

248. 921 N.W.2d 26, 32 (Iowa 2018). ↑

249. Richard S. Frase, Recurring Policy Issues of Guidelines (and non-Guidelines) Sentencing: Risk Assessments, Criminal History Enhancements, and the Enforcement of Release Conditions, 26 Fed. Sent’g Rep. 145, 151 (2014). ↑