The Chehalis Basin, the second largest in Washington, has been a site of political rancor over flood control and infrastructure policy for over a century. The use of BCA has failed to alleviate—and has even contributed to—policy gridlock and controversy in the region. The Chehalis Basin is a “typical case” of using BCA to evaluate a river basin’s flood control policies; put simply, the tale we tell is exemplary, not extraordinary.
In his 1995 book Trust in Numbers, historian Theodore M. Porter points out that the “history of cost-benefit analysis in the United States bureaucracy from the 1920s until about 1960 … is not a story of academic research, but of political pressure and administrative conflict.” Porter reveals how a set of “bureaucratic practices” was transformed into a “set of rationalized economic principles.” Yet little attention has been given to the application of BCA in its bureaucratic setting. The bureaucratic lens focuses not only on the value of benefits and costs, but also on how a bureaucrat may utilize BCA to structure or influence the decisionmaking process itself.
The Case: A Problem in the Chehalis Basin
The Chehalis River, which is located in southwestern Washington approximately 80 miles south of the city of Seattle and almost an equal distance north of Portland, Ore., is the second largest river basin in the state. It drains an area reaching from the Olympic Mountains to the north, the Willapa hills to the south, and the Cascade Mountains to the east, before it enters the Pacific Ocean at Grays Harbor. The total population of the basin is approximately 130,000 people.
When the Chehalis River topped its banks on December 3, 2007, it reinvigorated a century-old debate about how to prevent the river from inundating the towns of the basin. The economic cost of the flooding was $166 million, according to one damage estimate. The flood affected the broader state economy as well. Flooding closed part of Interstate 5, the major north-south vehicle corridor on the West Coast, for five days, and closed the primary north-south freight railway corridor as well. Gov. Christine Gregoire declared the basin a disaster area and pledged to provide funding to prevent a flood from occurring again.
However, two years later, before any policy actions had taken place, the river again sprung its banks, flooding the same areas and again closing I‑5. With two floods having closed the highway and damaged the local economy in just a few years, local stakeholders, the Washington State Department of Transportation, and the governor’s office began working toward identifying and funding a policy solution. Thus, they joined the same policy debate that local citizens, regional governments, state and federal agencies, and Indian tribes have engaged in regarding flooding in the Chehalis Basin since the beginning of federal flood control programs in the early 20th century.
The Chehalis River has always posed a vexing flood problem. Since the first settling farmers laid down their plots in the 1800s on what were then camas prairies (named for the plant camassia) inhabited by the local Chehalis tribes, flooding has disrupted agricultural practices and interrupted the major transportation route between the Puget Sound region and the Columbia River.
The first major flood of record occurred in December 1887 and claimed two lives. In the following decades, the Chehalis continued to flood with regularity. To build roads in the region, locals laid boards across the roadway to prevent horses and wagons from sinking into the mud. In winter, roads gave way to canoes as the primary means of transportation. Not until 1944 did the federal government investigate potential projects to address flooding in the basin. This investigation incorporated a new decisionmaking framework known as benefit-cost analysis.
BCA in the Basin
The arrival of BCA in the Chehalis Basis coincided with the expansion and formalization of BCA more broadly by the federal government. The U.S. Army Corps of Engineers had begun using BCA as a formal means to analyze potential flood control projects. The first major works on rivers in the West occurred as a result of a group of legislation, known as the Rivers and Harbors Acts, and were based on Gallatin’s principles for infrastructure decisionmaking. Among those projects were the first government projects on the Chehalis River. When the river was first assessed in 1875, the mouth of the Chehalis was guarded by shoals that prevented boats with a draft greater than 8 feet from entering the river. The Rivers and Harbors Acts facilitated a series of river improvements to enable transportation upriver.
The Corps did little to address flooding, however, as navigation projects were given priority. That changed with the Mississippi Flood of 1927, when the sheer volume and destructive power of flooding prompted congressional action. In 1927 Congress passed a new Rivers and Harbor Act that tasked the Army Corps of Engineers with surveying U.S. rivers for the development of hydropower, irrigation, and flood control infrastructure. This led to the publishing of Section 308 benefit-cost reports. The report on the Chehalis was published by the Corps in 1931 and concluded that “further improvement of the Chehalis River for navigation … was not justified.” Early on, residents learned that the river would flood and they constructed their farmlands so that when the waters came, their houses would generally remain above all but the highest floods.
In 1933, just two years after the Corps’ report concluded that flooding was not a significant problem, the Chehalis River experienced a severe flood. Two years later, in 1935, Congress had 1,600 completed Section 308 reports from around the country and set about to pass the infrastructure projects that the Corps recommended based on positive benefit-cost ratios and regional affiliations. On the floor of Congress, however, numerous projects that the Corps failed to consider were added to the bill. The ensuing controversy resulted in the abandonment of the bill in its entirety.
With no action to construct new infrastructure taken in 1935, Congress took steps to prevent such politicking in the future, and passed the Flood Control Act of 1936. The act established that the federal government should improve or participate in the improvement of navigable waters or their tributaries, including watershed thereof, for flood-control purposes if the benefits to whomsoever they may accrue were in excess of the estimated costs and if the lives and social security of people were otherwise adversely affected.
The 1936 Flood Control Act provided a legislative mandate for benefit-cost studies on flood control projects. The Corps was limited to large capital projects so that other methods of flood control were put outside their jurisdiction, which helped contribute to a national flood control strategy that revolved around capital projects and widely ignored land-use management and stream flow retardation.
In this early use of BCA in the Chehalis Basin, we see how bureaucratic mandates and missions resulted in a policy process that used BCA to analyze a specific alternative rather than to efficiently address policy goals. The Corps examined rivers in terms of hydropower potential, transportation possibilities, and the need for flood control. The Corps was in the business of building large infrastructure projects; this was the bureaucratic lens through which they viewed and employed BCA. Thus, their analysis of the Chehalis Basin was not about how to address flooding efficiently, but instead whether building a dam in the basin would create social benefits that would outweigh construction and maintenance costs of the facility. This distinction has vast implications.
Implication 1: Bureaucratic BCA focuses on the efficiency of a specific alternative, rather than how to achieve a policy goal or outcome efficiently.
Mid-1900s | A 1944 Army Corps report represents the first formal usage of BCA in the Chehalis Basin. The Corps investigated four locations in the upper Chehalis Basin and originally settled on two for potential sites. However, the report showed that project costs were “153 percent of estimated annual benefits, including liberal allowances for the value of such storage for irrigation development and abatement of pollution.”
Here again, the history of BCA in the Chehalis Basin demonstrates an issue that arises in bureaucratic BCA: in an environment of limited resources, benefits and costs will tend to be counted in accordance with easy data availability. It is sensible to avoid the costs of counting everything, but the likelihood that missing data might change the results should be determined as part of the analysis. In the Chehalis example, however, we observe that the Army Corps counted what was readily available—primarily things such as crop prices, which were easily obtained.
Implication 2: The results of bureaucratic BCA reflect costs and benefits that are readily countable, rather than a careful consideration of economic standing or economically significant cost or benefit flows.
BCA in the modern era | In 1966, the city of Centralia and the commissioners of Lewis, Thurston, and Grays Harbor counties petitioned the Army Corps to revisit their study of flood control alternatives. After a broad evaluation, the Corps focused on just two policy alternatives: modifications to the existing Skookumchuck dam and no action. Of the new dam sites considered, none had a benefit-to-cost ratio of greater than 0.4. No dams were built.
The return of flooding in 1990 and 1996 prompted local civic leaders to again call for action. Both 1990 and 1996 saw floods over the 100-year level sweep through the cities of Centralia and Chehalis and the lower watershed. In 1998, Lewis County requested that the Army Corps studies be reopened to see if additional measures could be added to increase flood protection. Concurrently, the Washington State Department of Ecology decided to band the local counties and cities together, creating a Chehalis Basin Partnership to evaluate and advocate possible flood control policies. Most of its efforts focused on dams on the main stem of the Chehalis River.
The Partnership recommended a suite of projects, including wetland creation, riparian re-vegetation and beaver reintroduction, forest conservation, low-impact development, and the blocking of agricultural drainages. Ultimately, very little came of these reports. The programmatic options such as beaver reintroduction or re-vegetation largely vanished from discussion in subsequent years. This was particularly curious given that existing (and subsequent) evidence indicated the cost-effectiveness, flexibility, and robustness of such measures. A primary reason why smaller mitigation measures and various minor retention alternatives fell by the wayside was (and continues to be) the nature of the bureaucratic lens that takes a project-by-project approach to the use of BCA. On a project-by-project basis, wetlands restoration is likely to be an economic “loser” no matter how cheaply it can be done. However, numerous small policy measures can have an additive or synergistic effect that might achieve results that, combined, are comparable to that of a dam, and at less cost.
Implication 3: Bureaucratic BCA takes a project-by-project approach rather than a portfolio approach, rendering small projects that only “work in concert with other alternatives” to be non-viable.
The last decade | In the past decade, the debate over flood control in the Chehalis Basin has significantly increased and BCA has continued to play a pivotal role. Following the renewed call for flood control action in 1998, the U.S. House Committee on Transportation and Infrastructure passed Resolution 2581, asking the Army Corps to reconsider Chehalis flood projects. This led the Army Corps to release a reevaluation report in 2004. Before much could be done, however, the aforementioned 2007 flood events swept through the basin and altered the policy arena. Jaw-dropping photographs and heart-wrenching stories of farmers and business owners became the emotional image of the floods, while the closure of I‑5 became the economic driver of state action, as it cost the Washington economy an estimated $47 million in delayed or diverted freight costs. Similar closures in 1990 and 1996 had also stifled the state economy, prompting Governor Gregoire’s call for a solution to I‑5 flooding along with relief for the property and business owners of Lewis County.
The state’s primary objective for flood projects was that they be “cost effective.” However, the goals of local officials, led by the Lewis County leadership, quickly moved toward a so-called “basin-wide solution”—a dam project that lowered or eliminated flooding throughout the entire basin. The term “basin-wide” implies some element of comprehensive, holistic management, but in policy debates over Chehalis flooding “basin-wide” has almost exclusively come to refer to policies that look to solve the problem of flooding in the basin with a single project, a dam.
In post-2007 deliberations, the idea of a water retention facility in the upper reaches of the Chehalis was first proposed by the Lewis County Public Utility District (PUD), which was interested in the facility because of the potential for generating hydroelectricity. Soon afterward, a citizens activist group called One Voice began to petition local cities and towns, as well as the Chehalis Basin Partnership, for a flood control project that would provide a “basin-wide solution.”
As a result of those efforts, the Chehalis River Basin Flood Authority was established and given two primary objectives: implement a flood mitigation strategy based on taxable flood districts and produce actual project proposals. Flood control districts allow local agents to address flood control issues through the election of a board that has the power to tax within the district. Once the district was created, funds could quickly be directed from the state to the district, which would implement the projects designated by the authority.
By May 2008, 11 local governments, including the Lewis, Grays Harbor, and Thurston county governments and eight city governments, had signed an inter-local agreement officially forming the Chehalis Basin Flood Authority. Each jurisdiction granted the Flood Authority the power to propose flood relief projects to the office of the governor and receive funds to help research and administer approved programs.
The actions of the Flood Authority in the post-2007 flood mitigation debate illustrate another problem with the use of BCA by local bureaucracies. The Chehalis River projects in the Army Corps domain were almost all rejected out of hand, based on their lack of cost-effectiveness. The BCA conducted under the purview of the Lewis County PUD and the Flood Authority, however, reached a different conclusion. Why?
Throughout 2008 and the early part of 2009, the Flood Authority debated the organization of a Flood Control District, listened to project proposals, and conducted a survey on what the public desired in the basin. Chehalis PUD released a concept paper in early 2008 presenting the case for a new approach to the Chehalis flooding. While the paper advocated for continuing levy projects that had long been pursued in the basin, it stated that “levies have been historically deficient in protecting against flooding as they fail or overtop in major floods.” Thus the PUD advocated for “a broader solution for reduced flooding for the entire Chehalis River Basin,” one that Flood Authority members hoped could provide multiple benefits, including domestic water supply, hydroelectric generation, and enhanced fish habitat in addition to flood control capabilities.
In February 2009, the Flood Authority received the results of its first commissioned CBA study, with net benefit calculations that its presenter characterized as “incredible.” The study (known as the “Phase I study”) was conducted for two potential water retention sites, one on the South Fork of the Chehalis and one on the main stem. The combined benefit-cost ratio was put at 2.02 for the 30-year net-present value, and at 1.64 with leveled benefits. With a regional analysis, the ratio rose to 3.58. A careful reading of the report showed a $300 million credit to the dams for eliminating the need to raise the height of I‑5—a credit that basically swung the proposed project from being a net cost to a net benefit. Yet it is unclear that raising the highway would have been undertaken in the absence of a dam. No reliable cost figures for the I‑5 project were given, yet without the credit the dam projects would have failed the CBA. While benefits are widely found to accrue to areas such as agriculture, recreation, and fish stocks, the 2009 report did not incorporate any potential costs of the dam outside of construction and ongoing maintenance costs. Moreover, the cost estimates failed to consider the costs of transporting salmon above and below the dam via truck and assumed without sufficient knowledge that adequate materials would be found onsite.
While a neutral assessment would likely conclude that the Phase I study was as much an effort to engender support for the dam as it was to objectively evaluate the merits of the project, a further relevant issue for our current discussion is the report’s intensive focus on summary metrics, particularly benefit-cost ratios. The Army Corps had long used benefit-cost ratios as their trigger for project approval: if a project had a benefit-cost ratio greater than 1, then it was approved. The unintended effect of this approach, however, is that the use of BCA becomes a source of controversy, in which project supporters count benefits to push the benefit-cost ratio above 1 and detractors count costs to push the benefit-cost ratio below 1. Standard benefit-cost theory speaks to the need for probabilistic simulations, sensitivity analyses, and the presentation of ranges of outcomes in order to demonstrate potential project or policy outcomes. As in the Chehalis Basin, bureaucratic BCA often neglects those approaches, in favor of simple “point-estimate” summary statistics that offer a quick heuristic by which to judge projects.
Implication 4: Bureaucratic BCA focuses almost exclusively on the benefit-cost ratio (or related summary metrics) as a way to justify a desired policy or project, rather than using more detailed tools to understand the risk and uncertainty associated with various alternatives.
Recent developments | A month after it received the first study, the Flood Authority voted to approve the funding of a Phase II study with a continuing contract to show the feasibility of a water retention project. Phase II was released in November 2009, with a period of public comments to follow. While debate over Phase I was limited because of the preliminary nature of the proposal, Phase II sparked much controversy and serves as yet another example of the role BCA can play in fueling, rather than settling, public policy debates.
The continued emphasis on the benefit and costs of a water retention facility within Phase II solidified pro-dam and anti-dam interests within the Flood Authority and other local institutions. Though the initial plan was to allow $480,000 for the investigation, a negotiation with the Confederated Tribes of the Chehalis Reservation reduced the allowable cost to $250,000. Under the compromise, the firm hired to conduct the feasibility study, EES Consulting, could receive the remainder of the $480,000 if no “fatal flaws” were discovered as of the $250,000 point.
Phase IIA, as conducted by EES, contained an engineering and structural study as well as broad economic analysis; this was the $250,000 portion. Unsurprisingly, Phase IIA did not identify any “fatal flaws,” thus enabling EES to proceed with Phase IIB. Rather than play a role in the design process, Phase IIA was classified as a “conceptual stage,” meant to study the possibility of building a dam.
On April 15, 2009, the Flood Authority brokered a written agreement about what the Phase IIB study needed to “prove” in order to prevent opponents from derailing the measure. Within the basin, many policymakers had already made up their minds about what projects were feasible. PUD officials voiced a desire for an objective assessment of the feasibility of hydroelectric power generation on the Chehalis River, but many other local agents approached the water retention feasibility report with the view that the cost-benefit work, if done “correctly,” would prove that water retention was the only and best solution.
While it would be easy to conclude that project supporters simply contracted to get the result they wanted, this speaks to a more endemic issue posed by bureaucratic BCA. BCA is often clothed in the rhetoric and tradition of science, but it is not simply a counting exercise; it is very much an art as well. Public and private analysts are required to exercise informed judgment and discretion concerning what types of benefits and costs to include, to whom such cash flows should be attributed, and how benefits and costs should be counted. Given the role of confirmation bias, framing effects, and other human judgment and decisionmaking heuristics, it is not surprising that the results of bureaucratic BCA tend to correlate strongly with the sentiment of the group conducting or commissioning the analysis.
Implication 5: Bureaucratic BCA tends to find positive net benefits for a given alternative when conducted or commissioned by project supporters, and negative net benefits when conducted or commissioned by project detractors. Both positions may be supported by legitimate bodies of credible evidence.
Discussion
Benefit-cost studies are easily made into simple “yes” or “no” answers as to whether or not a project should be continued, not as an art form meant to provide nuanced information about possible costs and benefits for a policy debate. Analysts and technical experts have long neglected this reality, and the general rhetoric surrounding BCA continues to perpetuate the misclassification of BCA as a purely scientific endeavor. This failure to acknowledge the critical subjective component of BCA often increases, rather than decreases, policy discord. The overarching implication of bureaucratic BCA is that all of the methodological rigor in the world can go to waste when BCA encounters real people and real policy situations.
In what follows, we present three overarching guidelines for executing BCA that are variations of the recommendations that we gave to the Flood Authority. These guidelines are intended to help practitioners and analysts carefully consider the manner in which they conduct and present BCA, in order that it can have a positive effect on policy decisionmaking. In other words, these are empirical guidelines intended to maximize the transfer of information and minimize BCA-centered controversy.
Move beyond the summary metrics. While generating estimates of net benefits for different policy options is the fundamental output of BCA, the lesson of bureaucratic BCA is that analysts should not overly emphasize benefit-cost ratios, net present benefits, and other “point estimate” summaries. Without context, net benefit estimates lack meaning and can result in ill-informed decisionmaking. Each component of the analysis, from data gathering to modeling processes, potentially generates highly relevant policy information that deserves equal footing alongside model results in presentation. Summary metrics tend to crowd out other information in empirical policy debates. Moreover, as is the case in the Chehalis Basin, summary metrics themselves can become the point of policy contestation: stakeholders end up competing over whether the “true” benefit-cost ratio is more or less than 1, rather than using BCA results as a basis for constructive deliberation. A single numerical BCA output can dominate thinking and discussion.
Provide a final product that is tractable and informative. The archival documents we analyzed for the Chehalis Basin demonstrate the bimodal tendency of bureaucratic BCA in that it seems to exist in either of two forms: voluminous, largely undecipherable multi-volume reports or over-simplified shorthand references. This is not a coincidence. Consider the Army Corps’ 2012 closeout report on the Chehalis River levees: the digital .pdf copy was so large that the Ruckelshaus Center at Washington State University, which is a repository for the documents, had to mail us a flash drive containing the document; the print copy checked in at 960 pages! Conversely, the public comment record, Flood Authority meeting transcripts, and other documents are rife with brief references to the closeout report that overly simplify its conclusions and draw no information into deliberation. The Flood Authority members can’t be blamed for this. The bureaucratic process produces a document that is, simply put, unusable. Policymakers, stakeholders, and other participants in decisionmaking do not have time to read such a report and draw out key implications. However, 960 pages of information are not likely to be sufficiently represented by a brief abstract or one-page executive summary (which the closeout report doesn’t even have). While the report might serve to satisfy regulatory requirements or legal mandates, it most certainly does not, on its own, serve to enhance decisionmaking or substantively inform the policy debate.
To counteract this, bureaucratic BCA needs a brief supplement for decisionmakers describing analysis process, results, and implications. Though the actual report will be much longer, especially given technical appendices, a project summary should also be composed that concisely details the analysis process, results, and implications as follows:
- a step-by-step discussion of how the analysis was conducted
- visual or graphic displays of quantitative data such as Monte Carlo simulation results
- explicit statements regarding model assumptions such as the null alternative against which the project is being compared and other analyst-defined aspects
- a clear description of what benefit and cost flows are included in the model
- qualitative analysis of elements not included in the model
- tables or graphs of relevant figures, such as net benefit disaggregation
- discussion of policy alternatives
A related point is that data and model information should be made available to facilitate replication, revision, and refinement of the analysis. Whether provided in a technical appendix, supporting documents, or made available digitally, every effort should be made to create a “living, breathing” model that allows policymakers to explore alternatives and changes so that the BCA is not simply a report to be discarded but a decision tool that can be used throughout the deliberation process. We observed in the Chehalis case that while stakeholders continued to argue about the results of the analyses, the reports themselves were essentially fixed; there was no effort—or ability—to use the information provided to gauge new ideas or examine the implications of questions posed by participants. One method that can help demonstrate the implications of analysis decisions and communicate policy risks is to create interactive models that are accessible to policymakers
Do not provide false certainty. While we discussed the politicized nature of summary metrics above, they also emblemize another problem with bureaucratic BCA: it conveys a false sense of policy certainty. EES’s Phase IIB report simply gives a low, medium, and high estimate of construction costs, which then correspond to a low, medium, and high estimate of total net benefits. The Army Corps’ draft close-out analysis on Chehalis River levees gives only a single benefit and cost estimate. The use of point estimates in BCA conveys a degree of understanding and certainty that is not in keeping with the true nature of our predictive abilities. As described above, this false sense of certainty sparks a great deal of controversy among those who disagree with certain assumptions or figures used in the analysis, thus causing undue focus on the point estimate itself. More importantly, there is a vast body of literature demonstrating that complex socio-ecological systems, such as the Chehalis Basin, are characterized by inherent, irreducible uncertainty. In other words, even with perfect, complete data, system outcomes cannot always be accurately or precisely predicted.
The way in which uncertainty is treated (or perhaps more accurately, not treated) in the Chehalis-related analyses we examined is especially troubling given the high variability associated with flood events. Not only is the year-to-year occurrence of flood events highly variable (as evidenced by the rash of severe floods in recent decades), but the nature of flood events themselves varies greatly as well. For instance, flood damages are dependent not just on the amount of precipitation, but also on the location of the precipitation and other conditions (e.g., existing water levels).
Uncertainty | The analyses we examined use an expected-value framework for valuing uncertain outcomes. The issue with the expected valuation method is that the one outcome we know with absolute certainty will not occur is that the benefits of flood prevention and mitigation will not simply accrue as an even, yearly benefit flow. Instead, there will be zero monetary benefits from flood protection in most years and significant monetary benefits from flood protection in those (likely few) years that floods do occur. Moreover, the expected value framework assumes dependence between years within a project’s lifespan by modeling outcomes such that a 10-year level flood, for instance, can possibly occur only five times in the life of a 50-year project. In reality, an event of this magnitude could occur any number of times or even not at all within a 50-year period. The expected value method seems to play a particularly large role in providing false certainty because it produces one number to represent several potential outcomes. Given that policymakers are not generally able to devote the time and attention to parsing the expected value in terms of these potential outcomes, using expected values simply causes this uncertainty to be lost in translation.
A method to better address uncertainty is to conduct sensitivity analysis. A sensitivity analysis simply varies a given parameter or estimate within the BCA model and evaluates how the overall benefit-cost estimate changes as a function of that value. This is easily accomplished and yet adds a great deal of information value to the analysis. Critically, sensitivity analyses allow stakeholders to alter certain model assumptions or parameters and see how projected results change as a function of this variation.
Conclusion
Implementing each of these recommended guidelines requires significant effort. Moreover, combating the natural tendencies of bureaucratic BCA will not be easy. Nonetheless, we believe that if BCA is to be a force for efficient, effective policymaking, such steps must be taken. Despite its myriad critics, the theory and methodology of BCA is well-founded and “healthy.” As demonstrated by the case of the Chehalis Basin, however, the empirical application of these methods in local and regional policymaking is a far cry from this rigorous standard. Practitioners and technicians would do well to consider the implementation component of BCA just as strongly as they do the theoretical and methodological components. To do otherwise risks having even the most informative, rigorously conducted BCA be a wasted exercise.
