Fueled in part by Gawande’s manifesto, scholars have, in turn, advocated for greater adoption of checklists in law and public policy. Richard Thaler and Cass Sunstein discussed checklists as part of choice architecture. William Simon urged “[p]rotocols of the sort that Gawande developed,” suggesting provocatively that checklists might have averted major financial and accounting scandals, such as options backdating and Enron’s collapse. In criminal law, scholars have proposed checklists to help police and prosecutors comply with obligations to turn over evidence to defendants, to improve defense lawyering, and to help judges weigh testimony. A Federal Courts Study Committee recommended a legislative drafting checklist, and checklists have been proposed to improve election administration, safety regulation, and education. In the regulatory context, Gawande argued that checklists would enable building inspectors and industry alike to make “the reliable management of complexity a routine.” If so, checklists might address longstanding challenges with decentralized law enforcement and administration, where few proven techniques exist to address dramatic disparities in complex legal decisions among linelevel government officials.
Yet the legal reception to checklists has not been uniform. Checklists, rather than solving the problem of bureaucracy, may create it. Cristie Ford associated checklists with the pathologies of rigid command-and-control regulation, and Lesley McAllister warned that checklists may lead to mechanistic application, so that users fail to detect real risk. Director of the Division of Enforcement at the Securities and Exchange Commission Stephen Cutler, cautioned people to avoid falling victim to a “checklist mentality.” Opening a congressional hearing on emergency preparedness, “Beyond the Checklist,” Rep. James Langevin (D‑RI) admonished, “Our nation’s leaders are not seeing the big picture. Instead, they are driving our departments and agencies to focus so much effort on checking boxes that there is barely time left to actually combat a potential pandemic.” Across areas, checklist skepticism abounds. William Koski critiqued the checklist mentality in school reform, noting that parties in structural litigation “develop[ed] more paperwork, checklists, and bureaucratic oversight, essentially ‘teacherproofing’ the reform process.” In family law, Catherine Ross concluded that “[t]here is … no easy checklist that agencies and courts can rely upon to predict whether a child can be safe with his or her parents.” In securities regulation, Cristie Ford opined that “[c]reating ever-longer lists of prohibited behavior or checklists of compliance-related best practices will not be effective if the basic culture of the firm does not foster law-abiding behavior.” In food safety, Diana Stuart and Michelle Worosz documented one veteran food safety auditor who claimed that “about 70 [percent] of the items on food safety checklists are irrelevant to food safety.”
A core empirical premise of proponents is that checklists, by reminding individuals, should improve the detection of errors, violations, or mistakes for complex tasks. Despite the litany of reform proposals and sharp disagreement over their effectiveness, this empirical premise has never been subject to rigorous empirical scrutiny in law and public policy.
Our article in the Journal of Empirical Legal Studies addresses this lacuna by providing the first evidence in a legal context of the effectiveness of checklists. We study a unique and inadvertent natural experiment in food safety enforcement conducted by the Public Health Department in Seattle and King County, Washington. In 2005, the department revised its health code, subjecting roughly half of violations (classified as critical) to a checklist to be employed by inspectors, but leaving other (noncritical) violations to a freeform recall basis, as was the case before for all violations. Most importantly, while the revision added and revised a range of items, through indepth qualitative research, including engagement with staff responsible for implementing the code revision, we classify the subset of violations that remained identical before and after the revision.
This approach offers several advantages to understanding the causal effect of checklists on code citation. First, a common critique of checklist studies is that the introduction of the checklist is accompanied by additional training, teamwork, and a reorientation of tasks, hence confounding the intervention. By focusing on identical code items—for which there was no additional training and change in instructions— we isolate the effect of a checklist independent of other factors. Second, because the checklist applied only to half of the violations, we account for temporal changes in sanitation and enforcement practices via difference-in-differences. Our design therefore addresses two common limitations to extant checklist studies: before-after designs studies, which cannot rule out temporal changes such as increased managerial commitment to quality improvement, and cross-sectional comparisons, which cannot rule out preexisting quality differences between adopting and nonadopting institutions.
We also use internal administrative data to account for assignments of establishments to inspectors. The checklist effect is hence identified by changes in how the same establishments are inspected by the same inspectors before and after the code revision, for the subset of code items that remained identical except for the checklist format. This has a considerable advantage in the inspection context, where there are well-documented differences in inspection stringency by inspector. Last, because the checklist format was merely incidental to the substantive changes in the health code, our design rules out Hawthorne effects, whereby research subjects may improve performance because they know they are being observed.
Our study examines data from 95,087 inspections, and 15 identical violations (1,426,305 potential violations) scored in each inspection by 37 inspectors serving before and after the intervention for 2001–2009. Applying logistic differencein- differences regressions with inspector and establishment fixed effects, we find that the checklist has no appreciable effect on inspection behavior. Because of the large sample size, our estimates are relatively precise, allowing us to rule out moderately sized effects. On the other hand, elevating the salience of a violation has a pronounced effect, even when the violation remains the same.
These findings have considerable implications for regulatory enforcement and how we conceive of choice architecture for regulatory behavior. Checklists are no panacea, and they cannot resolve core issues of administrative law The views expressed in this paper are those of the author(s) and should not be attributed to the Cato Institute, its trustees, its Sponsors, or any other person or organization. Nothing in this paper should be construed as an attempt to aid or hinder the passage of any bill before Congress. Copyright © 2018 Cato Institute. This work by Cato Institute is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. and the design of regulatory institutions. The benefits of checklists likely emanate from the process of simplifying core responsibilities and focusing on areas of highest risk, not checklists alone. Our findings hence suggest that proper prioritization of risk factors and code item simplification are critical to improving regulatory enforcement.
NOTE
This research brief is based on Daniel E. Ho, Sam Sherman, and Phil Wyman, “Do Checklists Make a Difference? A Natural Experiment from Food Safety Enforcement,” Journal of Empirical Legal Studies 15 (2018): 242–77, https://onlinelibrary.wiley.com/doi/abs/10.1111/jels.12178.
