The Sentinel Event Initiative: Proceedings from an Expert Roundtable

May 21-22
Alexandria, VA

Overview

On May 21-22, 2013, the NIJ convened a roundtable of experts to discuss the potential applicability of a “sentinel-events” approach to improving criminal-justice outcomes. Other fields, especially medicine and aviation, have adopted a sentinel-events approach to learning from error with measurable success. In these fields, a sentinel event is a significant negative outcome that signals underlying weaknesses in the system or process; that likely is the result of compound errors; and that, if properly analyzed and addressed, may provide important keys to strengthening the system and preventing future adverse events or outcomes. Increasingly, aviation, medicine and business fields have invested in a review of sentinel events that involves all stakeholders; refrains from making blame-placing a primary goal; and adopts an ongoing, forward-leaning, system-improving framework for surfacing, analyzing, and resolving sentinel events when they occur. The utility of such a sentinel-events approach in criminal justice was the focus of the roundtable.

Welcome and Opening Remarks

Greg Ridgeway, Ph.D., Acting Director, National Institute of Justice

Dr. Greg Ridgeway thanked everyone for attending the roundtable. He talked about the importance of innovation in conducting and using research to inform practice and the potential for a sentinel event strategy as innovation in criminal justice. Efforts that incorporate some aspects of a sentinel event approach, like the Milwaukee Homicide Review Commission, have contributed to conduct investigations about our understanding of root causes. He stated that Mr. James Doyle was a Visiting Fellow at NIJ who was researching sentinel events in criminal justice. He then introduced Mary Lou Leary, Acting Assistant Attorney General.

Mary Lou Leary, Acting Assistant Attorney General

Ms. Leary stated that she was excited about the meeting. Having worked as a prosecutor for many years, she has seen how the system deals with errors and knows there is room for improvement. The emphasis is often on fixing the blame for errors instead of trying to learn from mistakes. Her key question was whether the criminal justice system can adopt a forward-looking, nonblaming approach. She expressed great hope for this approach and noted that other fields have come far in developing an approach to dealing with mistakes that allows them to examine what happened and to learn from it. She noted, for example, that last year was the safest year ever in aviation. These improvements did not come easily, however, because they require a culture change, including moving away from looking only at specific incidents and specific blame.

Increasingly, medicine and aviation look at sentinel events as organizational accidents, not the result of a single bad actor or a single mistake. Just as occurred previously in these fields, the dominant approach in criminal justice has been to look for blame and liability and respond with civil lawsuits and reports from investigative blue-ribbon commissions. Usually, the reports then go on the shelf. What is missing is a vehicle that allows for an effective non-blaming, forward-looking analysis that will strengthen the underlying system and prevent the next error. In after-action reviews, such as those conducted by the National Transportation Safety Board (NTSB), or in medical reviews, the primary focus is safety: How can we improve safety and prevent tragedies in the future? It is not by pinning blame on an individual. It is rare that the outcome of a review is to blame one person. She indicated that Dr. Gordon Schiff would describe patient safety reviews.

Ms. Leary noted that changing a culture is difficult. At this roundtable, NIJ was asking the participants to think about the concept of the sentinel event approach in criminal justice. Is the criminal justice system ready and capable of this? If so, how can we get there? The idea seems to have great potential, and a sentinel event approach could become a tool that state and local governments could choose. Ms. Leary asked the group to engage in a discussion about this concept and consider: Is this an idea whose time has come?

Opening Presentations

Wrongful Convictions and Other Sentinel Events: Learning From Organizational Accidents in the Criminal Justice System

James Doyle, NIJ Visiting Fellow

Mr. Doyle described the idea of a sentinel event approach in criminal justice as “standing at a threshold” of a new approach. He said that he advocated stepping across the threshold, and he provided background on how the idea had developed to this point. An important factor has been the wave of DNA exonerations, which demonstrate clearly a system-wide fallibility that begins to erode public trust. This has raised questions about how the system should respond to serious errors and mistakes.

He discussed the 1998 meeting convened by then Attorney General Janet Reno to discuss how to respond to serious errors like wrongful conviction and to try to prevent serious errors from happening in the future. At that time, much of the discussion took a linear view of the causes, and these causes were ranked and prioritized. The role of eyewitness identification emerged as a key factor in wrongful convictions. In the first round of lists, serious concerns arose regarding eyewitness identification and how, despite the witness's intention to do the right thing, bad outcomes occurred. Concerns regarding eyewitness identification raised questions about how best to prevent these errors; ultimately, a similar preventive lens was used to examine other potential causes of wrongful conviction as well.

“In criminal justice, these developments have lagged behind the fields of medicine and aviation by 10 or 15 years,” Doyle stated. In these fields, a new paradigm was recognized that sought to identify the multiple contributing causes in, for example, the cases of operations on the wrong patients. In one case, there were 17 mistakes, none of which, on its own, would have caused the wrong patient surgery. These lapses, combined with latent weaknesses in the system, comprised an “organizational accident” that resulted in the erroneous surgery.

The argument is that the experiences in other fields might shed light on some of the criminal justice system's remedial processes in trying to fix processes that can lead to errors. They suggest that we should not try to fix only one component; we have to think about the entire system. We cannot think only about the bad apple (e.g., the inefficient detective or the lazy laboratory technician), which is not likely to prevent future errors. Important system-level questions should be asked, including: who supervised specific individuals; who made resources available or unavailable within the process; and why did a “bad apple's” bad decision look good at the time?

Focusing on placing blame or searching for a single bad apple has the undesirable effect of shutting down the system's ability to detect, identify and examine the large range of incidents and errors in ways that would be helpful to strengthening the system overall. If individuals are worried about being part of disciplinary practices or blame placing, they do not want to be involved. The slogan for a sentinel event approach might be that “every defect is a treasure.” If only bad individuals are targeted, mistakes and latent system weaknesses will be driven underground.

There is a system of screens in criminal justice. This includes the police, the trial system, judges, appellate “screens” or review processes, and others. Although these screens are designed to limit mistakes, thinking of the problem as one of structures that could be fixed with better screens does not work. It is important to know it is crucial to think about how the screens interact with each other — in other words, looking at one screen does not explain how the screens themselves are reacting in the overall environment. When new best practices are developed in one “screen,” they are soon under assault because of their inadequacies and have to change again. The way to look at this is not to convene a commission, but to make understanding and minimizing errors part of regular practice. There should be a way to look at events that includes all the stakeholders at every step in the criminal justice system. Stakeholders should convene in a nonblaming spirit, so mistakes are not driven underground. This is not a problem of structures alone or people alone. When normal people do normal work, the outcome sometimes results in errors.

Learning From Errors in Medicine

Dr. Gordon Schiff

Dr. Schiff addressed the context of medical error. He noted that the field of medicine has not solved all of its problems. Although things are getting better, there is controversy on how far the field has come. There is a long way to go in terms of measurable outcomes, and there is sober caution. For example, “cancer rates”, he noted, “have been reduced only 5 percent in 40 years, despite all the money spent.”

An important bookend in the safety reform movement in medicine is the 1999 U.S. Institute of Medicine (IOM) consensus report To Err is Human: Building a Safer Health System. The report had a significant impact, particularly in the context of its conclusion that 48,000 people died every year from preventable harm. Dr. Schiff noted that some of the lessons documented in medicine could be applied to cases of wrongful conviction and other errors in criminal justice. To Err Is Human highlighted the fact that medical errors are related to bad systems rather than bad people. This was a transforming concept. And, beyond looking at systems, the report also had broader implications with respect to the errors that everyone makes every day. In that regard, Dr. Schiff noted the Swiss cheese analogy first used by James Reason. Every day there are errors, such as people accidentally switching blood specimens, and others. We need to learn from these errors and mitigate them. Experiments can be done to improve the process. The threads worth picking up in criminal justice are learning from errors and creating a culture of nonblame. Change will come from those in the system.

Edwards Deming, an American, was the guru of Toyota and Honda, and later of Ford. Examining the idea of learning from defects, he advocated trying to understand why errors are common in every system. Deming pioneered the concept of Total Quality Management. Later, the field of medicine took a similar approach and looked at using disclosure and apology rather than circling the wagons and hiding errors from patients for fear of being sued. One of the early pioneers was the University of Michigan Health System, which experimented with full disclosure and apologies to patients. Lawsuits and claims dropped dramatically after this policy was put in place. The medical center's chief risk officer said that improving patient safety was more likely to cure malpractice than defensiveness and denial.

The Joint Commission, which accredits and certifies health care organizations, said that the medical profession owes it to patients to tell them what happened and learn from mistakes. They advocate a “culture of patient safety.” The Agency for Healthcare Quality and Research has tools that can measure the culture of safety with specific indicators. People are asked questions, such as whether they feel safe reporting an error. For example, although there are specific steps that can prevent contamination in an operating room, a nurse has to be able to speak up if she sees a doctor making a mistake. This is analogous to a copilot pointing out errors to the pilot. A culture that works is not always one in which a specific individual is the boss.

Dr. Schiff said that medicine has barely begun changing in terms of processes. There has been progress in patient safety, however. He noted that information overload can be as much or more of a problem as information dearth. He asked: “How do we organize and share information and make it more transparent and visible?” This fits with quality improvement processes and human factors. Human factors research addresses how humans work, such as the way medical professionals order medicines using a computer. In this regard, although the way medicines are ordered was redesigned (including computerized tracking of drug interactions and patient allergies), new errors were introduced by having the computers order the drugs. Therefore, it is important to understand that all changes are not fixes, but there must be continual improvement.

Dr. Schiff concluded by acknowledging that, although we don't have the data to address all challenges, we should shine a spotlight on errors and create a culture where people are comfortable sharing information on mistakes.

Discussion Led by James Doyle

Before Mr. James Doyle opened the topic for discussion, he said that NIJ is considering a sentinel-events initiative, which may develop a template for how state and local stakeholders could learn from criminal justice errors. He emphasized that this effort is not about creating any federal mandates regarding the reporting of errors, but rather a template for how state and local stakeholders can use error analysis as a learning tool. The goal of the project could be that some stakeholders would use a sentinel approach at least some of the time. “Although some people may never attempt this type of an error-review analysis,” Doyle said, “the field can still make progress in terms of a new cultural approach. We can provide a platform where stakeholders can disseminate information, accessing each other's experiences and allowing access by other jurisdictions, including researchers, to knowledge gained from a sentinel review.” He also said that we should not limit sentinel events to wrongful convictions: there are also near-misses, which are cases in which someone is exonerated but would have been cleared sooner if evidence had been processed quicker. There are also cold cases that stayed cold too long and cases where the harm/”error” is that too much money is perhaps being spent — such as a geriatric prisoner that cost the state great sums. There will be a question of whether the criminal justice system is thirsty enough to create an all-stakeholder process. The roundtable convened people from all criminal justice communities, with the added criteria of seeking people who had working knowledge of more than one part of the system. He said, “We want the benefit of roundtable participants' opinions, not just on behalf of any group or association with which they may be affiliated,” and he asked that discussants make it clear when they were anticipating a group's response versus expressing their own opinion.

A participant noted that the culture of safety is fairly new in medicine. The first error-in-medicine conference took place in 1997. The medical profession previously had so-called “blame conferences” that seldom looked beyond the case at hand and only at the isolated bad things that happened. The discussions were purely didactic; there was no attempt to learn for the purpose of improving.

Mr. Doyle was asked which method or model attracted him as an example worth following (e.g., checklists, voluntary disclosure to get at precursor conditions, or an NTSB-type review after a crash), suggesting that there were various techniques that could be used to perform quality control within a system. Mr. Doyle said he would like to see a broader and deeper culture of safety considered in general in the criminal justice system. Many of the techniques mentioned can contribute to system improvement, but none is sufficient.

The discussion turned to the usefulness of “near-miss” incidents, in which, for example, an innocent person is released from suspicion/prosecution before a wrongful conviction (that is, the system “worked.”) Some participants noted that a near-miss might be a useful way to jumpstart the idea that even potential errors could be reviewed as worthwhile learning experiences. A participant stated that, in his opinion, there is no such thing as a blameless investigation following a serious accident or error. Mr. Doyle replied that if blame is the exclusive focus and goal of an investigation, there is a different set of people participating in the discussion. An investigation that stops after blaming one person or one component is not the complete answer. Mr. Doyle gave the example of a report on a plane crash. There undoubtedly is not a single wrong decision that led to the crash; it was a series of small steps that created an ineffective system. Blaming the individual small steps does not get you anywhere.

Another participant stated that, initially, his impression of an initiative on sentinel events in criminal justice might be worth exploring, though his initial reaction was that those kinds of errors are outliers. He said the current culture is, “I know I wouldn't do that, and no one I know would do that.” For example, law enforcement has been under siege for a while, and most of what they hear is about blaming cops. There is an insularity that comes from feeling attacked. He recommended acknowledging the good intentions of the initiative, that it would not be designed to go after bad cops. The goal must be to focus on the overall system, the overall operation and process of the system.

In speaking about reviews of officer-involved shootings that he had been involved in, one participant stated that investigators would come to simplistic solutions, such, as “If the officer hadn't done such and such, he would still be alive.” However, he thought that, typically, the fallen officer was just doing what every officer does and it just caught up with him. He gave an example of a traffic stop situation in which a cop stood on the driver side of a vehicle to prevent the suspect from running away. Both officers at the scene were killed. He said part of the discussion had to address the need for regular, systematic audits even before an error or a sentinel event occurs.

Dr. Schiff responded by describing a Failure Mode and Effects Analysis approach developed by reliability engineers in the 1950s. It focuses on weighing how much something contributes to a problem. Every day, people do things in their jobs that depart from normal procedure (i.e., “workarounds”), but they usually have no harm results. A problem occurs perhaps only 1 in 100 times. For example, he said that the professional lives of nurses consist of continual workarounds because so many unanticipated problems arise in the hospital. To ensure quality control however, it is important that leadership address the underlying problems and not focus on the workarounds.

A participant observed that the legal profession may not be at the same place aviation or medicine was when they started a “safety” reform process. In accounting for wrongful convictions, for instance, the commentary from some is that the error stemmed from a technical problem; another observed that, statistically at least, wrongful convictions may not be a problem. Also, it was noted that prosecutors are very risk averse and are often very attentive to how their actions will play in the media, which can be a factor, adding that the legal system overall may seem more concerned with procedure, rather than justice and sentencing. He also said that he thought we have a system where, at the highest level, justices talk about procedural justice — and that people are so concerned with procedural due process that they are willing to accept an unjust outcome. He was not sure criminal justice is ready for this new approach to sentinel events, at least from the standpoint of the bench and prosecutors. He added that the conversation is often, “Look, the number of mistakes is low; there is no problem.” He added that, a review would only reveal the tip of the iceberg, and many problems would never surface in a review. For instance, we really do not know how many wrongfully convicted people are sitting in jails, and we also often rely on unreliable evidence or processes, such as eyewitness identification.

Another participant related the experience of learning based on work of the Innocence Commission and similar entities, like those in North Carolina, New York, Florida and California. “Most are formed by the executive branch, the legislative branch or the bar,” he stated. They convened all the key players from the criminal justice and other agencies, creating a very diverse group that included many with no direct role in the formation of laws or processes. One commission began by meeting for a year and a half, studying the issues, hearing from experts and looking at their best practices. That commission acted by consensus, and its mission is ongoing as it evaluates improvements that were recommended. The roundtable participant queried whether the sentinel-event idea that NIJ is exploring would be similar to the work of such a commission. He added that, inevitably, law enforcement will feel attacked, since “Innocence Commission” sounds like an effort to get offenders off. The committee he described worked because the participants shared the same goal of getting the guilty people and not incarcerating innocent people. He offered the commission's interest in the issue of informants as an example, noting that they looked at what every component of the system can do better. That commission's representatives were from the defense bar, law enforcement, the local prosecutor's office, the mayor's office, the deputy chief of police and the city council. He said he believed that they were doing some things right, such as increasing the use of videotaping. Another participant asked whether the jurisdiction tapes entire interrogations; the response was that taping was done in the police station. While the police were resistant to it at first, it removed possible defense arguments, seemed to help juries and led to important improvements.

One of the participants stated that he wanted to talk about the iceberg, not the tip. He said that defense attorneys feel that the exoneration movement ignores their reality. For the defense, he stated, the iceberg is the 7 million cases that do not have significant resources allocated to them in the first place. The false confessions and other problems with some of these cases are completely invisible. He hoped, through a sentinel-event-type approach, that enough light is shone on these 7 million cases.

He went on to say that, whatever happens with a sentinel-event initiative, a defense lawyer would continue to maintain his “bad-cop directory;” issues of civil rights may be extremely important if such a case is brought and, therefore, attention must be paid to the individual miscreant. And, in the end, the prosecution is the “adversary.”

The speaker ended by turning to the Brady ruling and said that a great amount of mischief has been caused by hidden information. Some prosecutors are great and honorable, but others are not. They have an interpretation of what “favorable” means: it has to be exculpatory, which is not what Brady says. These parameters are different in civil litigation, where the parties have broader access to information. In contrast, in criminal cases in some states, defense counsel may get access to a minimum of information and it can lead to “trial by ambush.” The trend is getting better, but he said, it is central to this issue.

The discussion continued with a focus on defense and exonerations, as another participant described the work of the Innocence Project, where they think of themselves as seekers of truth, basing the defense strategy on adherence to the facts. Their approach includes all the players, but it is also more than just the actors, the speaker said. Whether or not a defendant wins a model-citizen award, if a guilty person goes free, they might harm again. The difference between the medicine/aviation models and criminal justice is that it is easy to care about the innocent patient or the airline passenger, but, generally, it is not as easy to care about defendants in the criminal justice system. “We must attend to this issue,” the speaker said, “if the system is to improve.”

One recent analysis looked at the contributing factors to a wrongful conviction; it is never just one thing, there are seven or eight categories of problems. In the Innocence Project's day-to-day work in reviewing applications and investigating cases, one of the challenges is time and pressure. And keep in mind, the speaker said, that most people plead out. How can we create a system that can take the time to look carefully at each of the 7 million cases and all the factors involved in those cases? You can't ask the people doing the work on the sharp-end to do even more. And do all the actors in the system share a common goal? A suggestion was made to return to this issue later.

Several participants observed that comparisons to the medical system are not straightforward in that everyone agrees that the patient's welfare is primary. With respect to criminal justice errors, however, who is “the patient?” Is it the system, the accused, the community, the prosecutor, the witness? There are many “patients.” It is easier to implement change in the field of medicine, because it is less complex when everyone can focus on one patient. In criminal justice, the focus is diffused. One participant predicted that resource allocation and investment would probably be big stumbling blocks to implementing this approach in the justice system. Even if the patient could be identified, it would take tremendous resources. There are many events worth investigating, including a policeman being terminated or convicted of a crime; learning from those situations might prevent other problems down the road.

A participant noted that there are 17 or 18 thousand law enforcement agencies in the U.S.; perhaps these agencies would have to yield to some standardization? Currently, policing takes place in different ways. Should this be the case since we are all under the same Constitution? Differences arise as a result of differences in police training. In addition, officers often move from one division or operation to another, regardless of whether they have been properly trained for different assignments. When people become first-grade teachers or hold other specialized teaching positions, they have to be certified specifically for that job. There is no process like this for police. There are no standards for being on patrol or having other duties in different divisions. This contributes to the lack of uniform standards across the country. Perhaps the government should standardize how accidents should be investigated in all states and create a model from which to gather standardized information to make comparisons across the country. Then there should be standardized training.

There is an error in the way we are thinking about wrongful convictions and community policing, another participant observed. The sentinel event model is a different way of thinking that transcends a multitude of silos that go beyond the police organization. He opined that the police would not engage the sentinel event idea very well if it addresses only wrongful convictions, adding that a sentinel approach should look at errors more broadly, for example in terms of police recruitment, retention and promotion. It has to be like cognitive-behavioral therapy: Individuals have to begin to think differently. They have to focus on outcomes, errors in thinking about strategies, who gets prosecuted, and the bigger, overarching issues. He expressed concern about discussing only wrongful convictions, and others agreed that a sentinel approach should not be limited to wrongful convictions.

A participant endorsed the idea of a sentinel approach — calling it a great notion that should be broader than wrongful convictions — and said that it is important for NIJ to develop a political strategy while also thinking about the initiative's substance. Unlike the fields of aviation and medicine, criminal justice operates in a political environment. The criminal justice system often makes policies based on a single incident, and high-profile events, such as a parolee killing someone, which can have an outsized impact on any reform or quality-control process. Partly as a result of this, in the last 50 years, the field has been immune to research on incarceration rates. No one would have imagined that the United States would incarcerate as many people as we do. A system would never be designed this way, but we lack the necessary research to guide and influence policy. This is starting to change now with movement toward an evidence-based decision-making. But in essence, the challenge is not only to develop a framework, but to think about all the political and systemic obstacles.

For example, he continued, prosecutors are elected on the basis of several basic statistics, such as how many charges are brought and how many convictions are won. There are no useful performance indicators that tell people how well the prosecutor is actually doing, other than how many people are convicted and sent to prison. Their role is the least transparent in the criminal justice system. Some research has attempted to look whether there are inequities based on race in prosecution and sentencing. One prosecutor's office invited researchers to look at prosecution data for a race effect; then the office tried to improve their operations by overcoming apparent biases. But, this participant opined, not many district attorneys are inviting researchers to look at their prosecution data in this way, and that is a problem. With politics so toxic, he added, you cannot even begin a conversation at this level to talk about race.

Another participant said that, with respect to prosecuting someone suspected of a serious crime, like rape, there is a broad lack of transparency. Saying that it seems like the right time to explore a sentinel event approach, this participant described the accumulation of errors in the criminal justice process as akin to a train wreck, beginning with police and continuing on, as the wrong information is given to the district attorney or prosecutor. If people are not trained correctly, the consequences can be serious. There has to be cooperation across systems. When errors occur, in the vast majority of cases it is never acknowledged that there was a system failure, with errors throughout the system, turning into a cascade of errors. It starts with the police and continues with the prosecutor and becomes a tsunami of system errors.

Another participant raised the issue of the danger of binary thinking. This happens when the goal is to quickly reach a verdict of guilt or innocence. Everyone in every part of the criminal justice system has a moral imperative. For the police, it is to catch the bad guys; for the prosecutor, it is to put the bad guys in prison. They are doing their work in the best way they can. But somehow, the moral imperative needs to be changed. He suggested having a culture of truth as a new shared moral imperative.

Mr. Doyle wrapped up the discussion session by noting that parts of the discussion would make the safety professionals want to chime in. He drew a comparison to the safety field, pointing to the issue of incarceration. At some level, everyone would agree that the current incarceration rate is undesirable. Yet, when you look at the individual events, you can see how the factors of race and poverty affect individual situations. He said you cannot just go “down and in” in the investigation; you have to go “up and out” to consider the larger operating environment. You need to ask yourself: Why did this decision look like a good one at the time? Many events lend themselves to this kind of analysis. Maybe a decision that was made during the process was the least-bad choice a person had. When looking at a sentinel events approach, this perspective will be indispensable. All the variables in operation will not be revealed unless reviewers get into the finer details of individual cases. Framing the discussion in terms of “who is to be blamed” fails to offer the good guys the opportunity to do something other than blaming; they can have a positive role, not just chasing and disciplining bad guys but helping the entire system learn from the error — helping to improve the system by making it less likely that a similar error occurs in the future.

He concluded by noting that people emphasize mistakes by police, but it does not mean the problem stops there. The police make many decisions based on what will happen later in other parts of the system, because their actions result in evidence for the attorneys and affect the judge, the jury and others.

Confidentiality and Other Challenges: The Balance Between Incident Liability and Risk Management, and Other Challenges in a Sentinel Event Initiative

The facilitator asked the group to focus on issues related to confidentiality and the balance between incident liability and risk management. What are the ramifications of notions of blame and shame and protecting potentially culpable practitioners? What are challenges that could stem from the inherent complexities of the systems and the lack of resources? What do we see as the anticipated challenges to implementation and how they can be overcome?

Noting that there are significant challenges, one participant said the purpose of protective disclosure seems to change, depending on the setting. Which system is criminal justice more similar to with respect to confidentiality, aviation or medicine? In aviation, the drivers for change have run out of steam because the death rate due to crashes is very low. Two factors have driven this reduction. The first is the use of forensic exams conducted by NTSB and feeding those lessons learned back into the system; now there are far fewer crashes, making additional data unavailable. The second is how safety quality systems have resulted in piloting systems that are up to 99 percent error-free, which means the critical work has been done and all the checklists are in use. Most crashes now are bizarre and unique events. The aviation industry is currently dealing primarily with catastrophic risk involving many lives where the probability of error is very low. To obtain further enhancements, they would have to work with rare events and on all of the precursor factors at once. The progress of voluntary disclosure is based on precursor events.

The participant opined that these systems were not applicable to criminal justice from the perspective that there are still many sentinel event cases just coming to light, rendering criminal justice at a much earlier stage than aviation in error investigation.

He added that it might be more fruitful to compare criminal justice with the medical field, where there are hundreds of thousands of medical errors, most of which do not come to light. Nurses are encouraged to come forward and report doctors to help populate the data field so the potential system-wide problems might be discovered. This, he said, is more similar to the criminal justice system. For example, the Innocence Project can accept only 5 percent of applications submitted to them for review because there are so many potential errors in the system. One outcome of disclosure through a sentinel-event-type review would be finding more data from more cases. The idea of “protective disclosure” comes into play when an airline, for instance, wants to report its own violation. However, disclosure of a violation may not provide protection from immunity when it involves a serious violation such as a deliberate criminal act, when the regulator was about to discover the violation anyway, or in cases where serious harm occurred. The speaker noted that this is true across federal agencies reporting violations. Criminal justice is more similar to the medical system, in that we are only seeing a small number of the errors being made.

Another participant noted that law enforcement procedures can create risk of an error, such as a SWAT team going to the wrong door. There are a host of procedures about which some data may not be captured, such as police pursuits, use of force and incorrect eyewitness identifications. He said these are not generally regarded as opportunities to learn, so there is little motivation for disclosure or examination. This participant expressed concern regarding liability: For example, how might a sentinel-event review process maintain confidentiality of certain information? Policing generally relies on an “internal affairs” model to determine responsibility, but a sentinel-event-type review would ask police to adopt an approach where “examiners” are hoping to learn lessons to help prevent similar errors in the future? No one in the police department wants to talk to Internal Affairs, which, the discussant noted, is a crucial factor to consider because many errors occur at the police level.

The challenge of applying a sentinel-events approach to wrongful convictions was raised by another participant who noted that the opportunity to learn could be affected by the fact that the event may have occurred so long ago. A sentinel-event review could play a role in events such as a critical incident and other policing responses, which they may be more amenable to reviewing from a system-wide perspective. But there would be challenges because of the culture that currently exists. To push for protective disclosure to get frank discussions, there could be a concern that the “bad apple” would not be fully held to account. Some stakeholders would simply not be happy with a sentinel-event-type review. In addition, such a culture change would also have to change the expectations of the general public. When someone is shot, law enforcement cannot say to the community, “We found no one to blame, but we really learned from this case.” With respect to using a wrongful conviction as a sentinel event, there could also be a challenge ensuring multiagency involvement, which doesn't exist in the same way as with hospitals. The criminal justice system embraces agencies that may want to blame each other for an error.

The police will bear the brunt for opening the door to this kind of review process, said another participant. Many errors start early in the criminal justice process, with the police, and he predicted strong resistance from the police. Another participant suggested taking a broader view, noting that police are seen as the starting point of the case, so, if the outcome of such a review reveals errors in other parts of the system, it would help the police. With respect to crime labs, International Organization for Standardization (ISO) requirements and other safeguards attempt to ensure very strict standards and quality control. If the accreditation process is followed, detailed questions are asked about why mistakes were made. Corrective action and root cause analysis is the standard in crime labs. Therefore, looking at root causes of error from a system-wide perspective would be a good starting place.

Another participant noted that the discussion seemed to be focusing on wrongful convictions, but there are also wrongful acquittals, defense misconduct and other events that might form the basis of a sentinel-event review. He also observed that people sometimes believe that the end justifies the means. On the prosecution side, for example, the attitude can be: I'm fighting the good fight; I'm the last line between evil and good. The system is inherently adversarial; someone has the moral high ground, and someone is on the defense. The reality is that there are many errors occurring and the community suffers, because if the wrong person was convicted, the perpetrator is still free. Opining that there are many corrupt practices in all parts of the system, this discussant said he would like to see a dialogue about wrongful outcomes in the system and an effort to identify factors they share and where problems can be fixed. Wrongful convictions open the door, but they are not the entire problem. He suggested looking at many different kinds of sentinel events, but acknowledged potential difficulties such as liability and risk-management.

A participant described his experience with a process that tried to confront system errors; it was referred to as a review process, also a reform committee. They intentionally did not use the word “innocence.” They came together on several issues. They all agreed that there are some bad defense lawyers, that judges are not always doing the best job they can, and that states do not have enough money for DNA analysis. They got over the idea that the process was against any one person. They started looking for reliable ways to prevent wrongful acquittals. He said that once you get beyond your own perspective, you look at it as: How can we contribute to reliability overall? Once they addressed it from that angle, they could work together. He said the defense's job is the most difficult, because they are not institutionally concerned with finding the truth; their job is to show reasonable doubt. However, this type of commission must be concerned with truth, which means that some participants would have to give up some of the tools of their trade.

The facilitator asked the group to consider the difficulties in bringing law enforcement representatives together. One participant said the police do get blamed first, but that's just a piece of the puzzle. A reform commission he was familiar with was eventually able to get around this by communicating that the commission was not created to get more defendants off. The committee gained experience through talking to judges and innocence commissions. Initially, there was resistance on the committee to including an eyewitness identification expert, for example, but there was, indeed, a role for this type of participant. And the jurisdiction's police department is coming to the conclusion that the process works better if they cooperate. Everyone must understand that they have the same goals.

No one disagrees with learning from errors, another participant noted. But there are choices concerning who should conduct such an initiative. For example, if you are writing the ISO standard for police agencies, this will increase acceptance of the idea that it will benefit police. As NIJ thinks about who will oversee the initiative, he said to consider which risks the agency would want to control. The laboratories control the problems that are really about the laboratories. However, for complex issues that cut across the criminal justice system, there must be a coordinated approach, perhaps at the local level. He said it is unlikely that the federal government can fill this role in conducting a review. The federal role could be more effective in diffusion of innovations. Management of the review and reform process is best done at the local level.

The overall goal, another participant stated, is a means to some larger cultural change in the way criminal justice professionals think and talk. This kind of culture change can inform a broad range of issues and that is great. It makes the effort look less like an “innocence project.” He discouraged the use of the bad apple metaphor, which implies that there are rotten people. He also said it should not be framed as solely addressing systemic issues, as that is a false dichotomy. It is not just the nondeliberate mistakes that should be prevented. What are best practices for such a review? He stated that best practices were getting a short shrift in the discussion. He was in agreement with truth being a goal, and added that accuracy should be a shared goal in all parts of the system. Unlike medicine and aviation, he said criminal justice has not decided what the right ways would be to conduct an investigation, including how to interview a cooperative witness, how to get a confession or how to do a lineup. He said they could not go very far with a sentinel-events initiative when they did not know the best way to go about it. He suggested adding anecdotal evidence.

A participant noted that another way criminal justice errors are unlike medical accidents is that the contributing actors are independent of each other in criminal justice. He said that it is a statistical fluke if the surgeon cuts off the wrong arm. In contrast, criminal justice errors are not random and disconnected; these errors occur and compile during a single case, so best practices are needed to prevent the errors from spreading.

One participant noted that there were many obstacles that must be overcome. NIJ must lead the effort, followed by the leading U.S. police organizations. They should come to consensus on what the investigatory framework would look like. Then they should address how to promote this strategy and embed the standardized framework in the system. He suggested getting some opinion leaders onboard, such as representatives from large U.S. cities, by telling them that we want to do things better. It could be a demonstration project. Over time, the practices could be diffused if they are successful. Another participant noted that this approach sounded similar to the way community policing came together in that community policing was a new way of doing things that involved collaboration. Now everyone knows what it is.

Another key difference between the criminal justice system and the medical and airline models, a participant noted, is that in medicine or aviation, someone makes a choice about seeking medical treatment and what airline to fly. In the criminal justice system, the state usually intervenes in someone's life, and the system decides where to focus its attention. He said they would never get to the place where system change can be isolated from this reality. Eventually, there would be choices made if a civil case is filed. He suggested perhaps taking steps similar to those made by individual agencies to minimize risk exposure. He said that maybe they do not need to be as secretive as they have been. Mistakes are going to be exposed anyway. Gordon Graham, who works in risk management, started a website where individual firefighters can anonymously post errors that they made on the job. He asked why this could not be done for police. Also, he agreed that wrongful convictions are not the totality of the problem, but he said the issue is a great place to start and provide lessons learned. They might be able to learn from case studies in which individual cases are thrashed out. Anyone who has been in police leadership has had to argue about not going public with certain information. If you are going to get sued anyway, you might as well learn to live with it. Maybe they do not need to worry about it so much.

One participant with experience in the medical field said he was hearing a bit of déjà vu, because when reform efforts began in medicine, they said aviation is “different from us” and that attitude still exists among some. There are many things that can be learned from other fields; the similarities are greater than the differences. Concerning confidentiality, two things happening in medicine are relevant. There are confidentiality protections in the peer review process in hospitals, and some are more protective than others. They talk about errors after the fact, but the conversation of the official quality committee is, by and large, not discoverable. That is positive as it gives people freedom from fears. In addition, there are patient safety organizations that allow them to share information outside the hospital. The second relevant thing that is happening in medicine is the “apology and disclosure” movement started by Rick Boothman, the CEO of the University of Michigan health service. He advocates conducting root cause analyses and finding out everything, and if they get sued, so be it. There is none of that circle-the-wagons approach. The University of Michigan has had a 41 percent decrease in liability since adopting this policy, so it is right morally and it is right as a business case. It sets a different tone. How does that translate into the criminal justice arena?

He also noted that medicine has very complex interactions and consists of highly complex interacting systems. Just as in criminal justice, there is not just a single system. To investigate an error, they must get all the actors together and have teamwork solutions. There must be operational definitions in place across the system boundaries, and that involves teamwork.

Finally, he added, regarding near-misses, there is great value to be gleaned from medicine. The patient did not die, and the suspect did not get hung, so let us investigate without so much symbolism and so many heavy consequences. Do not be defensive; let us create a less threatening arena for discussion of the near-misses — since they are probably similar to bad-outcome situations.

Another participant referred to the need for creating “safe space” for this kind of review. An interim space could be created, such as quality improvement discussions across agencies and review boards. She shared her experience with a review board where everything was always in “draft” and people had the chance to talk to each other and admit that things went wrong. Some of these incidents were in police departments, but many legal cases take place across systems and need the support of an executive (e.g., the mayor, heads of agencies) and the people at the ground level, in the front lines. Meetings included all parts of the system. They needed to know that each person would protect the others, and that they would be thanked for sharing the truth. People at the ground level need to feel protected. That environment has to be created. The question was posed: How long did it take? Not too long, was the reply, because what are the alternatives? None of us have perfect systems.

We are stepping across a threshold with this idea, a participant noted. A sentinel-event-review initiative would not have to be a mandate, but rather a cross-agency effort addressed locally by volunteers with federal help, with the goal of developing a template. The participants would have to be self-selected, and they would have to choose their own events to examine. This could provide a way to learn. How would we talk about this if confidentiality were still an issue? Would we seek legislation? He advocated that folks scale back this type of discussion and, instead, think about what the first step would be, then take that, and learn something from it. He noted a previous point made regarding the value of the near-miss, which would address some objections, including that it is all about wrongful convictions, where a review might only happen years later. By looking at near-misses, they would learn about the error and the resiliency of the organization, and it would address the time-lag problem associated with examining only wrongful convictions. They could look at a near-miss sentinel event that has both an error and a correction, such as a person convicted because of snitch testimony, who was later exonerated by DNA evidence. Other questions to consider: Where would we want to experiment with this approach? In big or small jurisdictions? For what types of events? Will we learn enough to know what the second step will be? Near-miss situations might have fewer obstacles.

One individual spoke about his experience on a commission that made nonbinding resolutions. He recalled that people were very reluctant to talk. Themes in the current discussion reminded him of that effort. He emphasized the importance of setting up a framework that is neutral. He noted that, when his commission's resolutions were rolled out, they were undermined by law enforcement, which regarded them as an attack on police. Eventually, the professional organizations supported it, because they had discussions about using safe, controlled settings to talk about things. He said it is challenging work, in which high-level persons like mayors might need to be included. 

Another participant said the American Civil Liberties Union (ACLU) has sometimes been seen as adversarial. Part of the reason is that they are excluded from discussions, but, through working with them, he realized that they have valuable input and are not as adversarial as the police think they are. He offered the analogy of the movie 12 Angry Men in which each juror had a lot to say but no one had ever asked for his opinion before. When the ACLU is not asked to the table, they voice their views from their bunker. He recommended bringing people into a sentinel-event-type review discussion who may traditionally not be asked.

Another participant noted that they talked with the ACLU before they put up cameras on the street, and the group provided good advice on how they could use them. One participant brought up the time challenge, noting that near-misses would be hard to work on, because people are very busy. If the system ultimately got a case right, who has time to think about how it could have been done better? She said the entire system is overworked. On the board she worked on, someone was tasked with writing, and there was staff work that had to be done; they needed resources. Another participant said it might be a good idea to look at a case while the information is fresh and available. How much time would be allowed for a review, the first speaker asked. Another participant noted that some auto manufacturers stop the production line if there is an error. People then swarm to figure out the problem. There are corrosive effects of letting damaged cars go by on the line. Stopping the line also allows workers to see that management cares about fixing errors in real time. Root-cause analysis in medicine takes quite a bit of time, possibly 100 hours for a case; a hospital may, for example, do one or two dozen a year. The Veterans Administration has a model on how to conduct root-cause analysis that is probably on the Internet, which poses a series of questions on why something happened. One participant noted that a plane crash investigation typically conducts a thorough review in 18 months. A participant asked whether any organization is currently doing this type of work. Another offered an example, in D.C., where they investigate when a child dies in foster care. Another participant noted examples such as a review process for elder abuse and independent reviews and large-scale investigations, such as the Henry Louis Gates matter, around which an all-stakeholders study was performed. What seems to be missing from these is the non-blaming stakeholder review that captures everything from the initial investigation through adjudication. There are some things that come pretty close to a non-blaming approach, but nothing that takes in the whole system with all of its actors and the influence of one set of actors on another set of actors. NIJ is assembling information on comparable efforts largely related to criminal justice.

The question was posed about police reaction to a sentinel-event-type review process. One participant with police experience said that many would say they are under enough scrutiny now, and this seems like another level of scrutiny. They would resist participation, depending on the types of events reviewed. For example, a shooting incident where an officer was shot but nobody died already goes through several levels of review; and, if someone was injured, there will also likely be a civil suit, where facts are reviewed. A safe space is very important, he said, and there also would need to be state-level legislation, similar to what is in place for doctors. For states with collective bargaining, they would need model contract language. He said the Toyota stop-the-line policy was similar to Harley Davidson, where workers — who take pride in building the best motorcycles in the world — fought to preserve the right to stop the line. Not many police departments would give that kind of authority to police officers. If they did, it might be only to someone with 10 or more years of experience on the force.

Another question was posed, regarding whether anyone was ever disciplined as a result of information that came out in the death reviews in which some participated. Was there a moderator who prevented inappropriate disclosure and the blame game? A participant with experience in death reviews (which began when 10–15 inmates were committing suicide annually) stated that the main goal was to sort through the information, although there were some elements of the blame game. The agencies decided who came to the table; it was a small number. She did not know about the disciplinary processes that may also have arisen. Another participant noted that blame and culpability are still available by other means, in other places. An example would be someone who commits an egregious rule violation; there is another process for that.

A participant addressed possible reactions of police to the initiative. He noted that the Department of Justice (DOJ) has come out with best practices formulated by researchers for law enforcement; so why had DOJ not come together and implemented sentinel-event review processes? We all operate under the same laws, but procedures that hold everyone accountable are not standardized nationwide. People blame the police for wrongful convictions. The criminal justice system is involuntary. Law enforcement has to look at its own practices; they need a standardized process, if they can construct one. The subculture of law enforcement will not allow progressive change: The police commander will simply say no, we are not doing that. Law enforcement is actually a dynamic, fluid process. Citizens expect a lot from law enforcement. Law enforcement should realize that a more open exploration of the system is not a bad thing. He thought police would change only if forced to by the state or federal government, or if shamed into it by the media. He said police do not have the mindset that exists in medicine or aviation, which has standards for the actors. Stating that the police use antiquated techniques, he said he did not understand why the best practices on the books are not in use. NIJ could help criminal justice stakeholders embrace a sentinel-event-type review philosophy at the national level, but law enforcement is very behind the times.

Another participant noted that the attitude toward change in policing runs along a wide spectrum. There are cops who refuse to change and others who are more open to it. Some police agencies do debriefings with their team members and have an error review process. Policing has review components, but they are called something else. This participant was aware of places where police tried to implement small, incremental processes at various levels through reviews, normalizing discussions about what worked well and what did not work. Because street cops want to discuss successes as well as errors, it became easier for the police to talk openly after they had normalized the process by talking about positive things. In fact, police may be doing some of this already. We should look at where we can make changes and look for the low-hanging fruit. Policing is a high-risk, very ambiguous business, so if the review of events and processes are normalized, we can get there; it could be a step up from what police already do every day, which could be more than people think. Research on system change in the private sector may be helpful.

Police in some U.S. cities have been seeking accreditation for years without success. This kind of effort needs to come from the leadership first. Any new step is difficult until people learn to like it. Often, police put as little information as possible in a report so as to not be questioned on it during cross-examination; in reality, however, providing more information shows that someone did their job better.

The level of change that has occurred in policing in the last 40 years is remarkable, offered another participant. There is a spectrum of this, across all of the country's jurisdictions, however, and nationally there is still a long way to go. New officers could contribute enormously. If a sentinel-event review process is developed, the key is probably in the prosecutors' offices and the standards they offer.

One participant asked whether the communities in medicine and aviation know when mistakes are made and if a root-cause analysis is conducted. The general public typically does not realize that errors occur in criminal justice. They are astonished that police might hide evidence or pay a jailhouse informant. They do not believe there is corruption or errors. The community thinks criminal justice is getting it right most of the time. She noted that elevators require certification and restaurants post sanitation standards, but what about the criminal justice system? She loves the idea of a nonblaming process, but there also must be accountability. She expressed frustration because there are police who do not follow their own best practices. She thinks a sentinel-event review process should lead to best practices, adding that we are only as strong as our weakest link.

Another participant said that, with respect to medicine, there is a range of community awareness and sometimes the media demands answers; as a rule, however, there is not enough transparency. At some hospitals, an Intranet contains internal information that may be made public.

From Concept to Reality

To start off the final session of the day, NIJ's Dr. Thomas Feucht said NIJ's ongoing goals include making decisions about research priorities and how to explore knowledge in important ways. He noted that Mr. Doyle spent more than a year working on the concept of a sentinel-event review process — and, in that regard, NIJ has identified three testable questions: (1) Could something like a non-blaming review process really be put in place in the justice system? (2) Are there measurable public safety outcomes, such as enhanced administration of justice or fewer errors? (3) Is there a potential for sustainability beyond any funding that the federal government might provide? Would such an effort survive afterwards? He encouraged the group to focus on the first question for the balance of the discussion. Mr. Doyle summarized his thinking on the minimum components of a template and asked whether the group could develop a minimum set of features that people might be willing to try, with room for experimentation around a three-part core:
  • All stakeholders would need to be involved.
  • They would use a non-blaming approach.
  • They would determine what the safe space is and how it would be protected.

Additional discussion could address other questions: What would a sentinel event be? Who could nominate sentinel events? Who keeps the records? What should be included in the process? He said NIJ is also thinking about producing a document for dissemination that would be useful for practitioners and researchers.

A participant stated that this approach is eminently doable. He assumed the police would be the starting point and there needs to be a way to translate these big ideas for the police in the front lines. There is low-hanging fruit to start with. Any such initiative would need to prove to skeptics that it can be done without risk. Ideally, there would be legislation for protection, but otherwise, they could use the attorney-client privilege model. The initiative must address the question — “What's in it for me?” — from the police officer's standpoint (e.g., officer safety, better connections). There should be training and standards. What are the skill sets we should be thinking about? It is a knowledge management system, so social business software should be used.

Another participant brought up the firefighter's near-miss website, which recommended finding a way to translate near-misses into a disciplinary strategy. Clearly, any sentinel-events approach would have to anticipate and address a desire by some to focus on discipline; for example, a street cop needs to trust the chief, or he would not talk. He suggested that officer safety-and-wellness-related events constitute a type of low-hanging fruit for sentinel-event review, as they can involve error. He gave the example of an officer with mental health issue — known to the force — who committed suicide. He further noted that people need risk-taking and knowledge management skills to embrace a sentinel-event process. When errors are eliminated, public safety improves, he said. He asked, “Where is that safe space?” It needs to be carved out. A primary concern that police officers could have is their career, because there could be disciplinary consequences. The participant also noted that there should be a role for middle managers in any sentinel-event review process.

He went on to say that not everything has to be ready in order to begin; they can get started. But there has to be a great deal of talk about what is being done throughout the process. Cops will not put themselves at risk for something they do not have to do. The concept is not that difficult when you understand it and it is not foreign to street cops, although it has to be made meaningful to them. The effort would require two sorts of “safe space:” — a legal one and an organizational one.

Another participant noted that, with respect to police, legal protections exist. He also observed that the small number of wrongful convictions to-date has shown that change is possible over time. People change through a process of education. He agreed that addressing issues that are of most concern to law enforcement is very important, including legal protections, because police officers are the most vulnerable in the system; others are protected or have immunity.

Should victims be involved in a sentinel-event review? Yes, but they are not monolithic, just as police departments are not all the same. There are other community players that are important as well. A participant suggested starting small and letting jurisdictions define what they care about and their goals. The template should not be too directive. Local choices are important. She was concerned about data measurement problems, as she has seen such challenges. For example, a police department may say they do not keep records on the number of interviews conducted by individual officers or on other regular activities; the medical examiner will not be able to provide raw data; and the prosecutor may be the only one who has some important information. Simply put, people often do not write things down.

Another participant, who noted that one of his jobs is to help law enforcement agencies to change, said that the more something is seen as routine, the more it is accepted. If something is seen as “special,” or time limited, there is less acceptance. Also, the review should be focused on how officer decisions have been affected by other people's decisions. There are some key questions that must be answered. Have the officers been well trained? Do they have enough resources? Do they receive appropriate supervision? If these questions are addressed, the rank and file will get the word that it is not just about finding an officer to blame. Another participant agreed that the process should become routinized. Everyone must have a common set of understandings at the outset. They should talk about the difficulties — what might work and what might not work. He said it would be best to do something like this in a smaller organization as a pilot project: just within the police agency, for instance. He saw it as problematic across states without federal legislation. He added that other criminal justice players could be added once the police have become comfortable with the process. The pace would vary. Some places, for example, have never worked with a researcher and others have. You have to know how far and how fast you can push an organization, which is what the chief of police in a community would know. As a pilot project, one of the criteria should be to consider the tolerance for change and ambiguity.

One participant said the idea of a post-sentinel event must start with a top-to-bottom review of the local system to develop a baseline of working right in the system. Otherwise, they will be putting the cart before the horse. This initial review would reveal what is being done right and what could be improved on. These are not “either/or” ideas, another participant responded. We cannot wait until there is a perfect universe of best practices before getting started; best practices and monitoring can be done together, he added. The first participant said he did not disagree, but recommended a baseline assessment. In his locale, the review group was initially called an ad hoc committee, but it now has a place in the system and will be maintained. He also wondered about the notion of a “safe space,” since we live in a very transparent society and, if an error has occurred, it will come to light.

With respect to a safe space, another participant noted that even though some efforts like this are ongoing, they are not generally integrated into the main system. People can have discussions in a sentinel-event review and go back to their jobs and refrain from discussing the review committee's work. Within the group, people need to feel that they can disclose their mistakes. This is how we could change the system so that fewer mistakes are made. As a culture, it is hard for us to admit that we did something wrong. Criminal justice professionals should be able to disclose errors without retribution, without encountering a punitive tone.

One participant pointed out the distinction between “problem-focused” and “solution-focused” approaches. NIJ appears to be taking a solution-focused approach. He said a sentinel-event review could be important, but it may not be the whole solution. He agreed that “bad apple” is not a useful term; good people are trying to improve the performance of their department and have made mistakes. All the compounding errors led to other errors. What are the key performance indicators? For police, it is clearance rates. For prosecutors, it is conviction rates. For laboratories, it is clearing backlogs. There is pressure to move on to the next case. All systems are driven by pressure to go with a theory, and you could create a monster that focuses exclusively on convictions. He said that usually, performance improvement is about fixing just one agency — and that a cross-agency model has very specific problems. His instinct was to take a more problem-centered approach to respond to wrongful convictions and near-misses.

Another participant mentioned the 9/11 review. He did not understand the full system from that report, because they only addressed the negative event. He thought that including more events would be better.

Focusing on the sentinel event may be a distraction, one participant noted, since it focuses on the tip of the iceberg. If it is to be a “learning from errors” model, perhaps it should be more open-ended. The name, “Sentinel Event,” did not fully capture this. The main point that was being discussed was identifying errors and investigating and learning from them in a safe space. The Joint Commission defined some events that must be reported in hospitals. Perhaps events could be similarly identified and learned from in the criminal justice system under the rubric of “NIJ learning from error.” The title has to say something specific. Another participant suggested “system error training” as a name. Another stated his preference for the words “sentinel event,” saying it conveyed the idea of standing guard, protecting the Constitution. Another participant disagreed, saying he found the name cumbersome. He also advocated taking on every type of case, not just wrongful convictions. He said the goal is to get accuracy, wherever the errors occur. Some cops will not give you anything, and others will open up. In this big review, they should focus on situations that are challenging yet will yield the biggest gain, and make the process as transparent as possible.

One jurisdiction might have problems that are intrinsic to a particular agency, a participant suggested, and others have problems that cut across agencies. He suggested an approach that would make “seed” money available to encourage experimentation; the sites could come back later with their results.

As to the focus for a “sentinel event,” one participant stated that wrongful convictions are not enough to bring people to the table. Could a sentinel-event initiative program allow each site to decide its own focus event? Criminal justice issues involve multiple players, and it is important to consider what types of multiagency events or errors/near-misses a community would be interested in.

Another participant suggested building a framework within one organization and looking at its value. Then they could iron out the kinks and try to change the mindset to genuinely learn from the process. He said that, only at that point should a jurisdiction involve other agencies, as it is very difficult to get multiple agencies together from the outset. Launching it without having built a framework would be exceptionally difficult. The facilitator asked about the value of seeding different groups and letting the jurisdictions pursue it in their own way. The participant said he did not understand that model, saying a single-organization model would be possible if they get collaboration from some independent outside entity. One organization or agency within the justice system has a lot to lose by starting this kind of effort on their own; they could be left holding the bag.

What was being proposed is exactly the kind of work that some organizations do, including one this participant said he had worked for. There is a nonprofit organization that has projects and reform initiatives that are typically conducted in partnership with national, state and local agencies. He said a sentinel-event review project could take a year to plan and address key questions. This would involve everything from the name of the initiative to the focus. A political and implementation strategy is needed. He gave an example describing a situation in which they spent a year in planning before they ever sought funding support. They also spent a lot of time and political capital securing cooperation from local stakeholders, including judges who were important to the project. After a year, they have 20 percent participation, which is actually good progress in a difficult environment. It worked because it was in the self-interest of the participants. He recommended starting in only one place, disagreeing with the idea of seeding the effort in multiple sites, because it would be unmanageable and need a great deal of attention.

He added that the idea of self-nominated sites was a good one, and that an effort like this would need to start with a commitment from the agencies. He said it was up to NIJ how they wanted to design it. If the funding ultimately becomes available, a sustainability plan will be necessary. On the subject of political leadership, one participant suggested that the federal government should have a light touch. There are some people already doing this, and it should be encouraged. How do you support those who are already doing good work and get the word out? You can make such a project very specific, narrow and discrete. Another participant suggested that when selecting a location, multiple agencies should be involved, including a political stakeholder such as the mayor or another authority or power broker who is outside the criminal justice system, which can help keep a program going over time.

One participant returned to the role of the police, noting that many of these organizational accidents begin at the police level. If the suspect identification process can be improved, errors are reduced throughout the system. He said it makes sense to have others in the room to discuss the implications for policy. However, in reviewing a bad search warrant or a shooting, for example, it is possible that the problem may not extend beyond the police department.

Others felt a broader approach was needed. One participant said DOJ should bring teams together to talk to each other, and that it would be good to have a defense group at the table to also explore that part of the process. If there is a culture of learning from error, he noted, it has to include self-reflection and every agency involved should be committed to learning from error.

Representatives from NIJ ended the day by thanking the group and noting that the discussion thus far was very wide ranging, which was good. Immediate consensus or resolution wasn't NIJ's goal, which was to have exactly this kind of broad-ranging brainstorming.

Overcoming Challenges: Moving Innovation Forward

The facilitator began by noting that, although the discussion aims to consider system changes, people tend to look at it from their own perspectives. When an innovation is introduced, things get uncomfortable. People are creatures of habit and like the status quo, tending to pull back and worry about changes to their own organization. The discussion yesterday was very wide ranging; today, there would be a focusing on individual “screens,” the individual agencies that each has with responsibility to screen out errors in their own agency.

Mr. Doyle asked the group to think about a sentinel-event review process from the standpoint of consumers of a system. As a starting point, some kind of retrospective review would take place with all stakeholders present. The results would be generated by everyone at the table. He asked the group to focus less on the builders of the process and more on the consumers of the process. Would you as learners/consumers find the type of review/“accounting” satisfactory? Would it be useful? Will any process that does not include all the actors be sufficient?

Safety reforms in aviation were aided by black box recorders that helped analyze past events and shape events going forward. There is nothing similar in the criminal justice system that performs the black box function. It was important, he said, to not have a free-ranging solution looking for a problem, but, rather, to develop a way to help the criminal justice system generate narratives about what happened. It would also be valuable, he added, to have a document to pick up and read.

One participant picked up on the idea of black box recorders, saying they were a fantastic asset to aviation. The key problem with the comparison is that black boxes record objectively, while most of the records maintained for criminal justice are not objective or contemporaneous. He agreed with the shared idea of taking all available information and making everything open, including every interview with every witness. He said a recounting after the fact shows that people are not as good at remembering as they think they are. Human memory is accurate for about 80 syllables. For criminal justice purposes, verbatim statements are often needed. However, if they are not taken at the time, the results are not reliable.

Another participant responded by saying that, no matter how much they reconstruct a case, there is a great deal they cannot get to (e.g., what the jurors thought, issues discussed under attorney-client privilege, and so on). Therefore, it is critically important to lay out potential shortcomings of a sentinel-event review process, including a recognition of things that they will never know. They cannot reconstruct 100 percent. There will be many gaps. The facilitator said that facts with respect to the role of police in an event were more easily documented or recorded than, for example, the work of the defense, who, noted one participant, is not going to reveal what they decided to keep confidential or what discovery they may have failed to do. He agreed that Brady violations are a problem, but there is a “reverse Brady on the other side, where the defense withholds information. Not everything will be fully captured.

Another participant said there are obligations built into the defense lawyer's role and there are several paths to go down to get information despite the fact that there can be defense resistance to disclosing information and airing out honestly what happened. There will always be huge gaps. He noted that defense is the fastest-growing area for considering changes in practice. He noted that it was the 50th anniversary of Gideon v. Wainwright (the case that decided that defendants deserve legal representation in state courts, even if they cannot afford it). He said defense practices are varied in terms of what each state and location is doing. As one example, in Orange County, Calif., they are setting up secondary and tertiary defender officers. He also noted the state systems in Louisville, Ky.; Seattle, Wash.; and Portland, Ore. He said there are 14 defender offices in New York City alone and, structurally, it's hard to deal with. You have to ask: Who is the defense? It is not organized or monolithic. The police have units that think about management and budgeting and look at big-picture kinds of things. Defense doesn't have that. He said the Vera Institute of Justice had set up a partial framework. Many defender systems or half-systems are going day-to-day, and they have no time to think about management and risk. You go to court and get through the day.

He continued by noting that complaints to the Bar, including ineffective assistance of counsel, are filed continually. Some supervisors (if there is a supervisor) do not deal with these; and, he noted, many young defenders are handling mandatory jail-time cases their first day on the job. There are no standardized ways that defense offices are run, despite past efforts. There are areas worth exploring, however, and this is an opportunity for change, which will require peeling the “layers of the onion.”

Dr. Feucht said this discussion was useful in understanding the non monolithic nature of the defense bar. He asked the roundtable participants to comment, based on their respective stakeholder experiences, on resources that would best serve them, the deficits what would have to be overcome, and the tools that would equip them to participate in reanalysis or reconstruction.

One participant, familiar with defense work, noted that more and more defender leaders are capable of getting past the zero-sum game that orients their day-to-day lives. They have been brought into discussions about drug courts and other problem-solving courts, so there is more sophistication on the part of the defense in coordinated assigned counsel plans. Private lawyers have a structure. Those that dominate in the big cities also have something, at least nominally. Getting past the zero-sum game approach is a first step. He described this by saying that when he goes to a meeting and learns that a law enforcement officer who was involved in a case or testified in a trial screwed up, he can put that individual on his “bad-cop” list — a list of persons to avoid, to not work with in the future. That is the zero-sum game. That is the biggest hurdle, and some people would not get past it. However, defense lawyers have been brought into discussions about drug courts and other problem-solving courts, which have enhanced their ability to act strategically. Private defense lawyers also may have a role in a sentinel-event review process.

There are, he noted, jurisdictions where there is a culture of working together to address errors and shortcomings. He noted the efforts of New York County District Attorney Cyrus R. Vance, Jr, and the meetings he has held about eyewitness evidence, wrongful convictions and other issues such as Brady. One of the keys is an awareness and acknowledgement that they are only going to get Brady-level reform if they have the support of the district attorney; otherwise it will get blocked. Defense attorneys need to see some value, too — and there is a mutual advantage for prosecutors and defense attorneys to come together in sentinel-event-type review process. Another participant stated that one obstacle might be our adversarial system. Most of the world has an inquisitorial system, with the magistrate trying to get the truth from every source. The adversarial process, however, can be a disadvantage when trying to have open discussions. He wondered whether it would be advantageous to have every witness taped and have open discovery. Would all sides be willing to accept the consequences? He went on to note that there are some jurisdictions in which he gives up his witness list and other information quickly, and receives the same degree of cooperation from the other side. Another participant noted that when it gets down to wrongful convictions or near-miss situations, it is hard, institutionally, to take off the adversarial hat.

Mr. Doyle said wrongful convictions are only one example of a type of sentinel event that could be explored. In a wrongful conviction, there were probably errors on both sides, and everyone's contribution to the error should be considered. For example, a lawyer might have been drunk, asleep or crazy, but the judge may have found that a better lawyer or a bench verdict would not have changed the result, so the source of the lawyer problem would not be examined. He said that in some cases you can find out a lot about the shortcomings of defense counsel, but the court is not that interested in finding out why the defense has problems. He said he was not as interested in large policy questions as in lessons that could be learned from individual cases (i.e., in Brady cases, Why wasn't this specific information brought forth? Was it not asked for? Was it hidden?). We need to get at specific episodes and determine what influenced conduct; why did the person make a decision that turned out to be a bad decision?

One of the participants stated that a great deal can be learned from what is written down and what is not written down, and from both high-profile and low-profile cases. She said NIJ could conduct a pilot or demonstration project around sentinel events, but the issue of replication would be very difficult. There are still situations where an examiner or auditor is unfamiliar with a case, and if you confront the practitioner who worked the case to see where things went wrong, you won't get to the problem. She stated that it is safer for agencies to look at themselves at the staff and line levels than going cross-departmental. There will be a great deal of resistance to having everyone in a room with the mayor. However, work can be done to create a culture of learning from error while learning how to go across departments.

The participant continued by raising again the issue of stopping the line or stopping the surgery. The medical system has opportunities for checks built into it. There is potential in these models for building checks into the criminal justice system. What if a defense lawyer who doesn't know his cases said, I can't go forward? Can the prosecutor say, I don't know this case well enough? Are there opportunities for stopping the line? This is important. Anyone who cares about the system should be allowed to stop if the adversary is asleep at the table.

A participant who had made earlier comments about what is not recorded and the information that is not known was asked about this built-in lack of transparency. Should we take special note of the dark zones? Yes, the participant replied, that's critical. Documents have a tendency to take on a life of their own. People might make conclusions that are not warranted by the information. He noted that his jurisdiction has a great public defender's office that is well funded, but there are still many problems with full discovery. In California, for example, there is a requirement for reverse discovery, but this doesn't always happen or it is turned over the day before the trial. Another participant stated that if a sentinel-events effort were to go forward, participants should be explicit about limitations to avoid undermining the effort and making people think it's not fair. Everybody has to feel they're coming to the table fairly (i.e., that the process is not rigged).

A participant who had commented earlier about what does not get recorded was asked again about the lack of objectivity of written records, noting the discussion about violation of discovery rules. The participant responded by saying that it's best to have better practices regarding discovery on the front end. Is there a way, another participant wondered, to envision a sentinel-event review process that takes these concerns into account? Yes, the participant responded, in part by trying to make information more accurate up front and taping interviews would be the best method. But he noted that it can be 11 to 13 years after a conviction that a case is reviewed; would the recordings still be available that long after the event?

Another participant returned to the idea of stopping the line. Notice what that does to the conversation, he said. If you consider the relative value of an after-action review, you would expect there to be more emphasis on an after-action review the day after an interview rather than years later when a false confession comes to light. The length of time that has passed diminishes the value of the information.

Returning to the idea of approximating aviation's black box, a participant noted implications especially for players at the front-end of the criminal justice system. He said there's a movement toward videotaping interviews and line-ups, but the group hadn't talked about the audiotapes of radio transmissions between officers. The voices out in the field can recreate an event; audiotapes are verbatim records. In addition, in-car mobile dashboard camera videos have not been used to the degree they could be. There are also body-warming cameras that collect data as the officer is engaging in some sort of action. In fact, law enforcement is getting closer and closer to the black box idea. He added that there is only a certain amount of time that these things will be kept in storage. Radio transmissions are recycled. However, accuracy is increased at the sharp end of the system through the use of such technologies, including the lapel camera on police officers. Regarding the idea of stopping the line, one participant said that a DNA tech leader in a crime laboratory can stop the line. This is part of the accreditation process. Someone can come to him and say there is a systemic problem; any individual can stop the process, although it happens only occasionally. “Isn't the point of having reviews to identify the limitations of an investigation?” he asked. By starting the review process, a number of areas might be identified where improvements could be made. This creates hard data.

One participant noted that the discussion seemed to be wandering down a path towards bigger and bigger concepts. We're not going to stop the line in court, he said. We're not going to change the adversarial system. The phases that are manageable to examine and work with are (1) arrest, detention and the decision to prosecute; and (2) the trial process, which is governed at the state level. He said technology has value in two places: the car video (which is not a panacea) and the station house, where most of the problems occur. Videotaping witness interviews would be most useful. He suggested that roundtable participants identify the areas where the greatest change is possible. For example, he discussed how especially during the first year of a review program that he was part of, everyone was in their own niche; the final report made suggestions about what each silo could do. Although significant change could happen, he warned against going to the top of the mountain and starting there.

Suggesting that they step back for a moment, one participant said the discussion covered both the feasibility of a sentinel-event review process and the nitty-gritty of what that it would look like and require. What would it take to look at all relevant information in an event review? There could be information overload. Who would look at all the videos? Who would decode, transcribe and analyze all the data? Also, events are likely to look different retrospectively. Although he recommended that a project such as this be pursued, he warned that overloading it could be its Achilles heel. There could be new demands and problems created by efforts to fix the system. Extensive resources would be needed to do it right, or it might be done superficially or only in a small area.

From reviewing records, one participant noted: They've learned that the truth isn't in the record. It is often found in the things that are not done. It is important to be careful in that the record is what is written down. It is what wasn't thought about that is the problem. She agreed that you can't stop the line in the court, but it's worth looking at the idea of people in the system having the ability to say, “Stop.” If a review is conducted 11 years later, she added, it's still a learning opportunity.

One participant asked if anyone had looked at how systems have changed in the last 11 years. Another participant responded with an emphatic no, not systematically. The questioner said that looking at an event 11 years later would guarantee that the people involved are gone; there are convenient excuses to say, I wasn't here, it's not my problem. He said a disconnect takes place over time; people no longer have ownership. The other participant replied that there is a great deal of anecdotal information about system changes, but no one has gone back case by case. Another participant noted that it is possible to go back; many jurisdictions have done it voluntarily. He had a memo that documented the result of all the Innocence Commission investigations. He said it would be difficult to determine whether changes in policy have had an effect, such as an impact on the death penalty. To gauge system change, could people simply be asked if they are doing things differently?

A participant noted that there were sharp differences of opinion about the stop-the-line idea. The American Bar Association said individual attorneys can stop the line for an individual. Many district attorneys didn't like that, but some District Attorneys' Offices were opposed, because they wanted to make decisions about their attorneys — therefore, he queried, whose job is it to stop the line?

Another participant said that the majority of police departments don't have a policy on arresting a suspect. They might check the address and that's all. Maybe they should ask other questions, such as “Why do you suspect this guy?” Asking such questions as a matter of policy could become a best practice.

He wondered if roundtable participants were thinking that it would be too difficult to include the line officer in a sentinel-event review process. He thought there might be a disagreement about the goal of such a review process. When other participants said, “Just look at the reports,” it sounded to him like they were omitting the officers — but he said police needed to be involved. It would be difficult to determine why police handle evidence the way they do unless police are a part of the review

Another participant noted that at the very beginning of the chain, when a suspect is being booked, one jurisdiction might require a very brief statement; another might require an officer to go to the scene and verify that there was probable cause. He suggested that they might need to go even further back.

Mr. Doyle referred the group to the white paper he had written. His goal was to explore whether there are alternative means of going forward. He was interested in getting a sense of whether a document like this would be useful to the field. He advocated a cross-agency approach that includes all parties. He asked what the safety people would say about that proposition, adding that he wasn't sure the discussion had fully considered a safety-oriented perspective about how things go wrong. He wondered if the group was pulling away from recommending that people step outside their own agencies to make demands of or interact with other people, noting that the safety people would say the screens (i.e., the means by which each stage checks the work up to that point) affect each other, and they're all affected by the environment around them. With respect to best practices, he noted, the safety experts people would warn that, the minute you adopt them, they will be under assault. The new set of best practices will drift, just as the last set was subject to drift. He said he is also interested in ensuring forward-looking accountability using raw material similar to aviation's black box. What is the raw material in criminal justice? Can we approximate it well enough to inform a review? There doesn't seem to be a system-wide move toward creating a sense of forward-looking accountability. Should we examine near-misses, wrongful releases, juvenile suicides in adult detention?

One participant noted that there is a spectrum of possible activities, starting with an after-action report at the police level and then with each organization looking at possible errors. Another asked, Why not bring the different groups together and try a more systemic approach to error analysis?

Outstanding Issues

The group was asked to spend the last part of the meeting raising outstanding issues. One participant responded by pointing out that we could be reinventing the wheel in the U.S., because this has already been done in the U.K., where reform took place under the organizational accident rubric from a policing perspective. Records of the National Policing Improvement Agency (a nationwide think tank that is no longer in existence; functions have been moved to other government agencies) could be valuable in looking at how the U.K. overcame similar obstacles and disseminated products. Although that effort was police-centric, it reached out to other agencies and included research collaboration. He gave an example of a child welfare case that cut across the systems.

Another participant said he initially regarded this sentinel event discussion as being about eyewitness identification. He said law enforcement is responsible for most errors in the system. Attorneys aren't doing lineups. He said they could talk about the legal community, but the root cause of errors is with the police. He advocated taking the consumer approach and using the framework suggested yesterday for building collaborations at the local level and selling it to law enforcement. Because police don't have ample resources to do their jobs, they might want to take advantage of funding that could come with a federally initiated sentinel event project. Using the example of a recent lineup in which an officer wanted to show a victim only one photo, he said that the police need better training. Training standards vary from jurisdiction to jurisdiction. He said the police can adopt changes. He said police already know where the problems lie. They need to look at political, legal, ethical and policy measures. The primary problem is, who is doing lineups, and why do they do them the way they do? If there were training improvements and enforced best practices, it would go a long way toward reducing problems.

One participant responded by noting that every group has its own issues to deal with; it's not just the cops. Another added that there are many other mistakes, but this issue of policing is crucial. If the evidence that comes from an investigation is accurate, it's hard to screw up later. Mistakes become entrenched as a case goes further down the line.

Another participant noted that the whole system is involved, both the police and the attorneys. However, she largely agreed that most failures stem from failed eyewitness identification, adding that this seemed like the first point of entry for change. The variance in police procedures is frustrating. In Canada, they have stakeholders across fields that look at each wrongful conviction, which doesn't happen in the U.S. Here, exonerated individuals are put out on the steps, and a few days later, no one knows the person existed. She said the public is primarily interested in the next media story.

She continued by suggesting that cases be reviewed in the way that a football game is replayed with coaches studying the plays and players, then reporting to the head coaches. She said they should start with the police and have standard, required protocol. She said we're not doing an effective job of training our cops. Every exoneration has an innocent victim, and the way they got there started with the police, even though the court is involved later. The police hone in on someone at the beginning and make the theory fit. There must be systemic changes in our thinking and how we frame it to the community, she said, adding that human nature responds to framing problems in terms of cost, dollars spent. Another way to frame it is that there are guilty people still out on the street. The buy-in from politicians is important, as difficult as that is. Policy changes are needed that require standard protocols.

Another participant stated that, when he first read about this effort, he thought it was a no-brainer. We get all the smartest people together and we agree that we need to do this, and we throw a few things at the wall and see what sticks. He urged NIJ to obtain buy-in from conservatives and Republicans, as well as liberals and Democrats. He looked forward to seeing some draft action plans, and said he would be glad to provide comments on them. He thought that everyone agrees it's needed. This seems to be a leading edge, he said, and things have already started. Do it with integrity; get started as soon as possible.

On the centrality of the police-to-system errors, a participant took issue with an earlier remark. He said there are many reasons for wrongful convictions that are not related to the police. There are other issues such as post-conviction access to DNA, access to post-conviction counsel, the resources of the defense bar, evidence preservation, and prosecutorial use of informants. It is much broader than the police. A sentinel-event review project will require taking a look at the big picture in all systems. He said the organizational accident model is a good one. He personally did not agree that there is a 75 percent rate of eyewitness error, as some have claimed. The facilitator noted that mistaken identification is important and is part of the wrongful-convictions picture, and others in the group verified the high rate of eyewitness error.

The participant who had spoken earlier about police errors stated that, unlike a hospital, the criminal justice system is “involuntary:” The suspect in this involuntary system gets there because of law enforcement. He also mentioned some research NIJ has supported looking at investigative failure.

Another participant noted that the more they actually talk to the front line, including the police, the better they will be, and the more the system will change.

Another participant stated that some things that come out of the discussion will be easy to remedy, such as policy practices, but he asked, “What do you do about insufficient staffing or other resources?”

There are high-profile cases of all kinds, another participant noted, not just wrongful convictions. A locale has to want to do this. NIJ should engage jurisdictions where there is a demand and they are willing to participate, like Harris County, Texas.

Another participant said the idea of a moral imperative should be emphasized as this lies at the core of this type of system review. He said that concept should be included in any publications that are developed. This meeting was a first step, he said, adding that the next step could be more focused, perhaps a series of niche meetings that look at what each part of the system contributes.

He continued by addressing the issue if a federal role. He thought the funding would be minimal. He suggested collecting some cases from around the country and publishing them via the Web or in a monograph so that the discussion could begin. He mentioned a national conference and suggested disseminating recent work on best practices in this area. He was in favor of starting national discussions and finding out the best way to communicate information.

He concluded by saying that, although he did not completely agree with another participant's earlier comments regarding the centrality of the police, they were not far off the mark. Police are the key entry point. Police behavior leads to what is filed by district attorneys, and then it goes on to the court. There is the variable of what the district attorney does with the police's work. He enthusiastically supported a sentinel-events initiative.

Dr. Feucht said that NIJ sometimes has meetings simply to exchange ideas. In this case, they were intrigued enough by the sentinel event idea to consider some next steps. Although the roundtable discussion noted many areas where caution and careful deliberation was advised, NIJ feels that the conversation should continue. Enlarging the conversation and diffusing information would be continuing goals, he said, asking for the participants' input regarding innovation diffusion at this juncture.

One participant replied that, before diffusing the idea, it would be important to define what “it” is, adding that specification about what the proposal is, would be important.

Noting particular concerns that had been raised, Dr. Feucht said the neutrality, a “safe space” and participation by all stakeholders were important and NIJ would continue to be mindful regarding issues such as who should be at the table for such a review process, stopping and starting points, how much reliance should be placed on existing records, including tape recordings, and the investigator and others who were involved in the event being reviewed. The goal of the meeting, he added, was not to nail all the details down, but, rather, to explore a solution to underlying system problems and weaknesses.

NIJ's Dr. Katharine Browning said NIJ may want to fund a demonstration project next year in which an array of stakeholder groups would participate. She wondered whether all stakeholders needed to be there for the pilot or whether, for example, a prosecutor and a police department could conduct a sentinel-event review. Perhaps there could be a two-pronged approach that allows for change in the police department, but also communication with other entities. She said the roundtable discussion had raised interesting nuances and helped NIJ look at the challenges.

One participant said that there are many aspects worth pursuing in a blame-free, multistakeholder review of sentinel events in criminal justice, but he emphasized the challenges and concerns that had been raised. He said he did not think the project, as originally envisioned should go forward, as there are other better framings that were offered in the discussion.

Dr. Feucht said that there certainly was no clear consensus about how to build a multistakeholder team or how to fashion an initiative, but he asked whether a continuing conversation was worth having, including dissemination activities that would be worth doing.

One participant stated his support for a cross-agency, system-wide understanding of errors in a multiagency effort.

Ms. Nancy Ritter of NIJ noted that the challenge is to define the “it,” adding that this roundtable was assembled to help NIJ continue refining a possible sentinel-events review process to examine error in the criminal justice system. Noting the uncertainty, one participant said that backing away from the idea would be a huge mistake. It could be helpful, he said, to look at communities of interest (i.e., niches) where this idea has a good chance to take hold.

Ms. Ritter offered possibilities for the group's consideration: would a white-paper be helpful, or gathering professional organizations to explore the idea, or even try to interest a reporter in writing about the concept in a mainstream publication, such as The New Yorker. NIJ colleague, Ms. Maureen McGough, wondered whether a promising focus would be the culture of truth in an adversarial system.

One participant noted that exploring these kinds of new ideas is what other nonprofit research agencies do. There is a clear methodology; it is not writing reports and it's not niche communities. You begin by being clear about what the thing would look like. To learn something, he said, you have to do something. He was against using a journalist or holding 20 meetings; he said to do something. If NIJ wished, they could involve research agencies with this kind of expertise. Start with an interesting notion and receive some guidance. It takes six to nine months to develop a specific proposal on how an initiative will work (i.e., to define what the "it" is). Working with a “theory of change” means to start with something and then scale it up. He said there is a tested, 50-year methodology for doing this kind of developmental work. If the idea doesn't work, you take it down. In the end, someone needs to plan a thing on the ground somewhere. It's a rigorous process.

Another participant suggested that a parallel thing for NIJ to think about was the IOM's To Err Is Human report. There will always be mistakes; there is error in the system. That is the obstacle for the wrongful convictions cases, for example. There is still a feeling in the community that people shouldn't make mistakes. It's important to educate people about the possibility of error. All you can do is tell people, “Do the best you can, but realize that it's possible you're not getting it right. Across the systems, there is an inevitability of error.” Getting it right as much as possible can be the goal, while understanding that there is always the potential for error.

Another participant noted that as a criminal justice executive, he would be inclined to tinker with these ideas, bend them and twist them to learn about the issue. He noted that one of the biggest complaints from practitioners is that it takes too long to get findings of such research and experiments to the field. He also noted that there is high turnover in the fields. He said that money is not always the solution to solving problems, adding that he knows cops who would be willing to explore the idea of a sentinel-events review. Would law enforcement look outside their system for solutions, he was asked. Yes, he replied. Police have a strong relationship with the key players, and if it doesn't take a lot of resources, he thinks they would run with the idea, and try to get feedback early on. Noting that policing is oriented toward action, he said; you have to jump in and just start doing things. The process is messy, but we learn from that.

With respect to the inevitability of error discussion, another participant noted that some errors happen because people are fallible; some happen because of bad practices. He said the way police question witnesses is wrong. If they don't ask questions correctly, they will miss good information or plant misinformation with subtle cues. No one intends to do this. There are just better and worse ways of conducting interrogation, and we increase the risks of error when the best ways are not used. If many errors can be prevented through eliminating bad procedures, this is a key point that should be considered in any system-wide reform process.

Saying that he liked the sentinel-event review concept, one participant noted that there are roles for organizations such as the National Legal Aid and Defender Association and other organizations to which roundtable participants may belong. He wondered, too, if there was a role for professional societies in helping to change the framework for thinking (i.e., getting beyond individual silos). What is needed, he said, is an emissary to talk about this on a national stage, particularly where smart people get together. He noted that you can get buzz on a topic in certain quarters, and it will end up on training agendas and start to build momentum. He emphasized that NIJ needed to reach smart, influential people.

Mr. Doyle mentioned the large body of literature on diffusion of innovations, which embraces the value of reaching out to professional groups, for example, in an effort to find those willing to first adopt the practice. What other means could be used to identify and reach the early adopters? He noted that, in the medical field, there was a national conference after a Boston Globe reporter died of an overdose of chemotherapy. The underlying issues in that case became an area of focus. In addition, a new organization — the National Patient Safety Foundation — was established after the national conference and the seminal IOM report, To Err is Human, was published. Dr. Feucht said NIJ has used these diffusions of innovation principles in various arenas, including prisoner reentry. A conference was held, which required participating jurisdictions to send representatives from all parts of the justice system and to work as a team. The conference framed a national dialogue about offender reentry, which was propelled by the creation of a jurisdiction-specific plan developed by each team.

One participant noted that they were trying to improve the reliability of fact-finding in the criminal justice system to punish guilty people and not punish innocent ones. At a recent wrongful conviction summit he attended, there were judges and other system representatives, but 90 percent of the approximately 200 people who attended were police. Agreeing that there was a role for the federal government in the development of a sentinel-event review process, he said that the idea of the postmortem review is a component of the process for understanding errors. Through the roundtable discussion, he said he had become a convert: The system should analyze itself, every jurisdiction should do it, and NIJ should show leadership at the federal level.

Saying that he agreed with the idea of a conference, another participant argued that the concept of a sentinel-event review should shift its orientation from a program to a problem. Using the example the previously discussed chemotherapy overdose, he noted that the approach to solutions was problem oriented, and addressed risk-mitigation strategies. He said that NIJ should use the culture-of-safety model and draw from the experiences of other industries. NIJ should consider this as one approach in thinking about problems to investigate. In criminology, he said, there is a tilt toward program-centric thinking, which, for example, lies at the heart of the evidence-based movement — rather than considering an individual problem and developing tools to address it. Many NIJ-funded initiatives are geared toward programs, he said, adding that NIJ should move away from programs that are solution focused and toward those that focus on specific problems. Dr. Feucht noted that NIJ has initiatives in both of those camps. A participant said there are two or three primary areas that contribute to wrong outcomes, including witness misidentification and false confessions; he asked how best practices could be developed to avoid these outcomes. He said that wherever NIJ wanted to take this, it should be framed as a process — including finding people who would want to take on pieces of the process and bouncing it around until there is group consensus. He said that if NIJ addresses two or three key issues, the system would be much better.

Speaking about wrongful convictions as one example of a sentinel event, another participant said that when someone is exonerated, the American public does not think in terms of this meaning that the guilty person is still on the street. What helps change public perceptions is seeing a real victim. She offered the example of how, after 9/11, people started talking about security in airports and, now, everyone is willing to go through security precautions because they recognize the importance. They think it's worth it if it will keep a terrorist off the plane. She voiced the opinion that the way to shift thinking in the general public is through fear — for example, raising awareness about the rapists and murderers who may be living down the street. She also pointed out that all the roundtable participants were white, noting that, for example, communities with a large minority population would be engaging in a different dialogue because their sons or fathers would be more likely to be the victims of criminal justice system errors.

Another participant said that rather than highlighting the fear aspect, it could be more helpful to speak to the general public's natural reflexes of anger when mistakes occur, including a desire to blame and punish, and channel those feelings into a model that emphasizes “learning” from error. Can that be done? For example: What are we doing to prevent this same type of bad outcome from happening in the future? That, he said, is a natural catalyst for learning. He also said that the “truth” often gets lost in the adversarial process, although a trial is supposed to be a vehicle for determining the truth. He urged that a sentinel-event review emphasizes the bottom line of why errors really happen. Put something in place, he said, and test it with one or two early-adopter jurisdictions and see what can be learned from that.

In conclusion, Mr. Doyle said that the roundtable discussion had been very helpful in generating much food for thought. He told participants that if they had additional ideas later, they could submit them to an informal e-mail listserv. He said he agreed with comments that people tend to learn by doing — and that current criminal justice systems do not have a good vehicle for helping us understand why things go wrong, so there's a great deal to learn.

On behalf of NIJ, Dr. Feucht thanked the roundtable participants for a vibrant discussion.

Roundtable Participants

David Borden
Justice (Ret.)
Connecticut Supreme Court

Jim Bueermann
President, Police Foundation

Russell Canan
Judge
Superior Court of DC

John Copacino
Georgetown Law School

Madeline deLone
Executive Director
Innocence Project

James Doyle
Visiting Fellow
National Institute of Justice

George Gascón
District Attorney
San Francisco District Attorney's Office

Michael Gennaco
Los Angeles Office of Independent Review

Doug Ierley
Consultant

Michael Jacobson
President & Director
Vera Institute of Justice

Greg Matheson
Crime Laboratory Director (Ret.)
Los Angeles Police Department

Bernard Melekian
President
The Paratus Group

Dennis Murphy
Legal Aid Society, NYC
Criminal Defense Practice

Jeff Rojek
University of South Carolina
Deptartment of Criminology and Criminal Justice

Gordon Schiff
Research and Practice
Brigham and Women's Hospital

Jon Shane
John Jay College of Criminal Justice

Dan Simon
University of Southern California Gould School of Law
University of Southern California

Sean Smoot
Director & Chief Counsel
Police Benevolent Association of Illinois

Malcolm Sparrow
Harvard University
John F. Kennedy School of Government

Jennifer Thompson

Frank Tona
Charles County Sheriff's Office

Ellen Scrivner (Facilitator)
Police Foundation


Project Staff

Katharine Browning
National Institute of Justice

Thomas Feucht
National Institute of Justice

Maureen McGough
National Institute of Justice

Nancy Ritter
National Institute of Justice

Observers

Nazmia Alqadi
Office of Community Oriented Policing Services

Silvia Dominguez-Reese
Department of Justice — Access to Justice

Jolene Hernon
National Institute of Justice

Scott Hertzberg
National Criminal Justice Reference Service

Mary Lou Leary
Office of Justice Programs

Angela Moore
National Institute of Justice

Toni Morgan
Office of Community Oriented Policing Services

Phyllis Newton
National Institute of Justice

William Petty
Office for Victims of Crime

Date Modified: April 4, 2014