An Inside Look at Creating Standards for Equipment
by Sarah B. Berson
NIJ image, Robert Walker Jr (see
Download and Print
NIJ sat down with Chris Tillery, Office Director for NIJ's Office of Science and Technology, to examine the standards development process at NIJ.
NIJ: Why does NIJ develop equipment performance and testing standards?
Chris Tillery (CT): One of NIJ's goals is to improve criminal justice policy and practice through the application of technology. Identifying the performance requirements of criminal justice practitioners and the equipment that meets those requirements is one way we do that. To that end, we develop performance standards for the unique equipment used by criminal justice agencies.
NIJ's standards development process results in an articulation of the practitioner community's consensus about the minimum performance requirements for a piece of equipment and the test methods needed to assess its performance. NIJ standards improve criminal justice policy and practice by setting the bar that equipment must reach to meet the requirements of criminal justice agencies.
NIJ is not a regulatory agency. Consequently, its performance standards are voluntary. Neither manufacturers nor criminal justice agencies
need to adopt these standards. But there are reasons for both to do so. The manufacturers of the equipment — body armor, license plate readers, dash cams — are incentivized to meet the performance requirements of the standards, because they reflect the consumers' requirements. On the purchasing side, the standards give public safety agencies the ability to compare different types of equipment against a common set of benchmarks.
Standards can also
raise the bar for equipment performance. Here's how that happens: The practitioners say they need a widget that can do X. If there's a manufacturer out there who can do X and X is part of the standard, then agencies are going to want to buy things that do X. They will begin putting X into their requests for price quotes. So even though the NIJ standard is a minimum standard, it's also a best practice in the field. If manufacturers want to stay competitive, the equipment they create must eventually, at a minimum, do X too. That also provides an incentive for manufacturers to do better than X to differentiate their product from others on the market.
NIJ: How does NIJ decide which standards to develop?
CT: This decision arises out of how NIJ prioritizes its technology research investments. We do that by systematically engaging criminal justice practitioners in discussions about their work. This process helps us identify shortfalls in their capabilities that might be addressed by technology. Developing a new technology might be one way to address a shortfall. Developing a performance standard for a technology might be another way.
Wherever possible, NIJ adopts existing standards or adapts them to the needs of the criminal justice community. To this end, NIJ scientists and engineers participate on projects with other standards development organizations. Additionally, we coordinate closely with other federal agencies, such as the National Institute of Standards and Technology and the Department of Homeland Security, to ensure that NIJ is not duplicating their efforts.
NIJ: What's the process for actually creating a standard?
CT: Practitioner-based special technical committees write the standards. NIJ believes that the people who will use the equipment are best suited to understand what the equipment should be able to do.
In addition to expert criminal justice practitioners, the special technical committees include scientists, engineers, test laboratory personnel and experts in conformity assessment. And if there's a representative voice that speaks for the entire manufacturer community, we might also have them at the table. When that's not the case, as with body armor, we engage manufacturers at various points during the development of a standard through workshops so they can provide input.
What Happens After a Standard Is Published?
After publication of the standard, a conformity assessment program must be established to identify equipment that conforms to the standard. Conformity assessment programs can take a number of different forms depending on implementation cost and the risk associated with the failure of a piece of equipment. The simplest type of conformity assessment involves the manufacturer or supplier declaring that a piece of equipment meets the requirements of a standard. Third-party testing for conformity, as exemplified in the NIJ compliance testing program for body armor, is another.
Learn more about NIJ's compliance testing program for body armor.
NIJ reviews its standards and conformity assessment programs on a recurring basis. NIJ's review is informed by its structured engagement with relevant practitioner communities. Reviews can lead to a number of outcomes, including updating a standard, retiring a standard if it is no longer needed or changing the conformity assessment program.
The manufacturer community might also provide input during the public comment period. All NIJ-developed standards provide at least one opportunity for public comment. Depending on the degree of change in the draft standards resulting from the first public comment period, NIJ might open a second public comment period.
The work of a special technical committee is reviewed by an advisory working group, which consists of representatives of the major practitioner stakeholder organizations (such as the International Association of Chiefs of Police) and relevant federal agencies.
There are two major purposes for developing standards this way. One is to get the buy-in of the people who are going to use the equipment. The other is to help ensure that what comes out of this process is a community consensus with respect to the requirements. More than 17,000 law enforcement agencies are out there. How do you achieve consensus? You do it through this sort of process. You use expert practitioners and representatives from the larger community.
The goal is to develop a requirements document that represents the consensus of the community about the performance characteristics of a piece of equipment that are most important and what those characteristics should be.
When the special technical committee and advisory working group have finished, the next stage is to validate the test method included in the standards. The validation process includes asking for existing equipment from the manufacturer community. To do so, NIJ will put a call out: "If you manufacture widgets, we would really love for you to provide one of your widgets for us to test." What we want to do in testing is to demonstrate that there is at least one piece of equipment out there that can meet the standard.
A standard is an exercise in what is doable. For example, the practitioners on a special technical committee might say that their ideal requirement is for a widget that tells them something
instantaneously, but the researchers on the committee might point out that instantaneous communication violates the laws of physics, and the test lab guy on the committee might point out that they can't test for it. So it's this iteration between the ideal requirement and the need for something you can actually create and test. You keep going back and forth until you get something that the user can live with
and that you can test
and that the manufacturer might actually be able to meet.
NIJ: The Justice Department recently approved an NIJ trademark for standards. Why is that something NIJ decided to pursue?
CT: We wanted to explore the potential of using a trademark to reduce representations that a piece of equipment complies with an NIJ standard when it does not. Such representations can happen in several ways, including lack of familiarity by manufacturers or practitioners with NIJ's compliance testing program and, potentially, intentional deception.
Our first planned application of the trademark is on body armor. The problem now is, if a manufacturer says, "This body armor meets the NIJ standard," or, "This body armor is designed to meet the NIJ standard," then how can you gainsay that? Or if a manufacturer says that its body armor is "NIJ certified," do all buyers know that that statement is false, because NIJ doesn't certify products? Part of our effectiveness — in fact, part of the effectiveness of voluntary standards — depends on criminal justice agencies being able to differentiate between participating and nonparticipating equipment. Misrepresentations decrease the effectiveness of the standards.
If we trademark equipment that
does meet the standards, then there's no question. A trademark clearly communicates a product's compliance with NIJ standards to the law enforcement and corrections practitioners who want to buy it. Agencies buying body armor with the trademark can be confident that it meets their requirements. Having a trademark also provides a legal remedy for false representations where there is not one now.
NIJ: Budgets have gotten tighter for federal agencies in recent years. How is the current fiscal environment affecting standards development at NIJ?
CT: The fiscal environment is a challenge. It limits the number of standards we can develop and maintain and the speed with which we can do that. One way we're trying to maximize our resources and still meet the needs of criminal justice practitioners is by partnering with private-sector voluntary consensus standards development bodies, such as, for example, ASTM International and the National Fire Protection Association (NFPA), in the development of standards that address the requirements of criminal justice agencies.
NIJ: The idea being that these bodies, rather than NIJ, would develop standards?
CT: To the extent possible, NIJ will encourage standards development bodies to develop and maintain needed standards. This strategy is rooted in the National Technology Transfer and Advancement Act, which encourages federal agencies to use voluntary consensus standards to the greatest extent practicable and to collaborate and participate with groups that are developing them.
NIJ: How can NIJ be sure that standards developed by private-sector standards development organizations, which don't focus on criminal justice, will meet the needs of criminal justice practitioners?
CT: Implementing this strategy effectively requires the active participation of NIJ scientists and engineers in standards development organizations. The National Technology Transfer and Advancement Act encourages federal representatives to participate on voluntary consensus standards bodies and be as active as possible, and the Department of Justice has found the participation of NIJ scientists and engineers to be in the public's interest, because it saves the federal government money and improves the speed of standards development.
Debra Stoe, an NIJ scientist, is on an ASTM subcommittee for the E54 Homeland Security Applications Technical Committee. Brian Montgomery, an NIJ engineer, leads the NFPA's technical committee that is developing the self-contained breathing apparatus standard. NIJ's participation will ensure that resulting standards adequately consider and meet law enforcement requirements. Because NIJ engages the practitioner community and understands its technology requirements, we can ensure that its requirements are accurately and effectively translated to these organizations and ultimately reflected in the standards.
NIJ: Anyone can download standards developed by NIJ for free. Will that be the case with standards developed by private-sector standards development organizations, or will practitioners have to purchase them?
CT: One drawback to this strategy is that standards developed by voluntary consensus standards bodies are not necessarily free. Although not exorbitant, there are fees, typically less than $100 per license. One of the things we did was to establish an agreement with ASTM in which NIJ will pay an annual stipend of $30,000 to provide any criminal justice or public safety agency with unlimited free access to the complete library of ASTM E54 Standards on Homeland Security Applications, including the body armor fitting standard developed by ASTM.
Read more about NIJ's work with ASTM, and learn how criminal justice professionals can register to access ASTM standards for free.
NIJ: What has NIJ achieved so far with this strategy and what are its plans for the future?
CT: We've already encouraged the development and publication of one standard through this process. Debra Stoe, NIJ's official representative on ASTM's E54 committee, worked with other committee members to help ensure that body armor fits well into the ASTM Work Plan; and in 2013, ASTM published
Standard Practice for Measurement of Body Armor Wearers. As a result of that successful collaboration, ASTM is developing several other standards relevant to criminal justice practitioners, including one on ballistic-resistant shields. Additionally, as I mentioned earlier, the NFPA is developing a respirator standard that will address the requirements of law enforcement agencies, and Brian Montgomery is leading the committee developing that standard.
Although the cost to the federal government is lower when private-sector standards development organizations develop and maintain standards, that's not always going to be feasible. When it's not, NIJ will encourage them to develop and maintain test methods that we can incorporate into NIJ-developed standards, which would help reduce the required federal funding. And that's something we already do to an extent. Many of the test methods in the CBRN protective ensemble standard, for example, were pulled directly or adopted from several standards developed by the NFPA. Our recent standard for protective ensembles worn by bomb disposal technicians also incorporates portions of NFPA standards. So we're expanding and accelerating something that we've been doing all along. We're just trying to do it more.
Our hope is that in the future, NIJ will fund only the development of standards and test methods when a standards development organization in the private sector cannot be encouraged to do so.
For More Information
Learn more about standards at NIJ, including a list of active standards and information on standards under development.
Read about NIJ's body armor research and find the compliant products list for bullet- and stab-resistant body armor.
NIJ Journal No. 275, posted June 2015
About the Author
Sarah B. Berson is a project director at Palladian Partners, Inc.
Back to the top.
Date Modified: September 8, 2015