Defense Research and Development (R&D), particularly that which involves the SBIR program, is a useful setting to study these issues for several reasons. First, the Department of Defense (DOD) was historically an important financier and early customer for transformational technologies, including jet engines, cryptography, nuclear power, and the internet. Frontier defense technologies have historically had dual-use components that can lead to large private-sector spillover opportunities. Second, the DOD is the world's largest single R&D investor and comprises about 60 percent of total federal government R&D. Third, the SBIR program is among the world's biggest and most influential government programs to spur innovation in small business, spending $3.11 billion on 11 federal agencies in 2018. Of this, the DOD accounted for $1.32 billion, and the Air Force had the largest single SBIR program at $664 million. Finally, the defense setting enables us to study the government as a customer rather than as a regulator and financier. While there is extensive literature on government regulation and financing, the government's role as customer is quantitatively important in the United States and to an even larger degree in many other countries.
Top policymakers in the United States have suggested that the nation's defense R&D increasingly lags behind the private sector, with one reason being that innovation procurement is narrowly specified and siloed in a small group of defense specialist firms, leaving little room for more radical innovations. We document that the U.S. defense sector has indeed become less innovative compared with the rest of the economy since the early 1990s, a period that coincided with extensive mergers and acquisitions activity that consolidated the defense industrial base. From the DOD's perspective, it is problematic if the best technologies are no longer marketed to the military. From a broader social perspective, there may be significant productivity-growth implications from the department's attenuated role in funding frontier ideas.
To address these issues, the Air Force experimented with open topics in its SBIR contracts starting in 2018. The goals of open topics are to reach nontraditional firms with innovative technology usable for both government and consumer markets and to source ideas that the Air Force may not yet know it needs. The SBIR program is a useful venue for experimentation because it is flexible, with more ability than other procurement methods to adjust contract types and lengths. The program has also been criticized for being dominated by incumbent contractors who repeatedly apply and win many contracts; they allegedly rely on the SBIR program for revenue but fail to produce technology that is useful for military operations. In an open topic, a firm can propose any idea or technology that may be relevant to the Air Force. By contrast, the conventional SBIR topics are more narrowly specified. Like most mission-oriented R&D programs, conventional topics are top-down: research ideas are generated within the Air Force and then firms are invited to complete them.
The open-topic reforms aim to revive the DOD's role as a large, early customer for risky new technologies from startups. Sourcing innovative ideas via open solicitations is not unique to this reform, as other government agencies both in the United States and overseas have developed similar programs. Companies are also increasingly using bottom-up approaches through customer-driven, outsourced, or open innovation, especially in R&D-intensive industries. Whether such a bottom-up approach to innovation can be successful is a long-standing question. In many cases, a research funder cannot spell out exactly what promising projects will look like, making a bottom-up approach more attractive. At the same time, there are potential downsides, especially in the defense context. For example, companies oriented toward private-sector commercialization may not deliver technologies that are useful to the Air Force.
In this paper, we assess how these reforms affected the selection of firms applying for a Small Business Innovation Research (SBIR) award as well as the effects of winning an award. We use administrative data on applications and evaluations of Air Force SBIR proposals over the 2003–2019 period (and outcomes through 2021). We focus on 2017–2019 to facilitate comparison of open and conventional topics, which were run simultaneously in 2018 and 2019. In the baseline 2017–2019 sample, the data include 7,300 proposals from 3,200 firms. The larger sample of applications from 2003 includes 19,500 proposals from 6,500 firms.
We first show that the open topics reached a dramatically different type of firm. Compared with firms applying to conventional topics, open-topic applicants are about half as old, half as large, less likely to have previous Air Force SBIR awards, and more likely to be located in an entrepreneurial hub. Their technologies are also more often software-based rather than hardware-based. In sum, open-topic applicants appear to represent potential high-growth startups in the United States much more than conventional-topic applicants.
Next, we assess the effect that winning an SBIR contract has on two main outcomes: future venture-capital (VC) funding and non-SBIR DOD contracts. These outcomes demonstrate the benefits SBIR contracts give to the wider private sector as well as the DOD, and they correspond to the program administrators' key indicators of success: VC funding or investment represents high-growth innovation potential and leads to spillovers in other aspects of the economy, while defense contracts indicate that the technology may be useful in an operational DOD mission.
We find that winning an open-topic competition increases the probability of subsequent VC investment by 5.4 percentage points, which is 68 percent of the mean among open-topic applicants, but winning a conventional-topic competition has no significant effect on VC investment. Second, we find that winning an open-topic award increases the chances of a subsequent non-SBIR DOD contract by 7.5 percentage points (51 percent of the mean). Again, conventional-topic awards have no effect on subsequent non-SBIR defense contracts. The open-topic program does not seem to have crowded out the effects of the conventional-topic program, because winning a conventional-topic award had no effect on VC investment or non-SBIR DOD contracts before the open-topic program was introduced.
We also consider the quality (as indicated by originality) and quantity of patents as an alternative measure of commercial innovation to VC investment. Winning an open-topic award increases the chances of receiving a patent by about twice the mean, whereas there is no significant effect from the conventional-topic program. Winning an open-topic award is also associated with higher patent originality, while winning a conventional-topic award is not.
The final measure of innovation is future Air Force SBIR awards. We find that winning an initial conventional-topic award increases the chances of winning a future SBIR award, while there is no effect in the open-topic program. This implies that open-topic competitions do not appear to suffer from the same persistent dominance of recurring SBIR winners that seems prevalent in the conventional-topic program.
In sum, the open-topic SBIR program has strong effects on measures of innovation (VC investment and patenting) and on the conversion of these ideas into new non-SBIR military contracts. By contrast, the strongest effect of winning an initial conventional-topic award is that it helps the firm win more SBIR awards in the future (which is not a particularly desirable outcome from the DOD's perspective).
Having shown that the open-topic reforms appear successful, we investigate whether their accompanying effects primarily reflect a different composition of applicants or whether openness—that is, the bottom-up approach—plays an independent role as well.
To more directly assess potential compositional differences between open- and conventional-topic applicants, we use data from two other Air Force SBIR reforms. These reforms had specific topics but had other features, such as faster contracting and outreach to startup hubs, that attracted firms similar to those in the open-topic program. We find that the open-topic program had significantly larger effects than these other programs. We also use machine-learning techniques on application abstract texts to characterize the degree of specificity for each topic (essentially through examining the similarity of text among applications within a given topic). Using measures of patent quality and quantity, we show that when a conventional topic is less specific—and thus closer to the open-topic program's bottom-up approach—winning a conventional-topic award has a significantly larger positive effect on innovation.
The open-topic reforms seem to work because they provide firms with an avenue to identify needs that the Air Force did not previously know it had. The open-topic contracts may represent an entry point to much larger defense contracts, which helps to explain their large effect, especially on VC investment. Open-topic startups with a successful initial application phase can bring evidence to VC investors that large defense customers are interested in their commercially driven development efforts, which appears to improve their odds of raising funds. While high-growth startups appear poorly aligned with the conventional-topic SBIR program, which requires awardees to produce a particular technology previously specified by the Air Force, the open-topic program allows firms to bring something to the SBIR program that is their own idea oriented primarily to the civilian commercial market.
Our results suggest that a more bottom-up approach to innovation that encourages new entrants can have significant payoffs to firms, the military, and ultimately consumer welfare, through enhanced innovation. We cannot be sure that a wholesale switching to open topics would be best for the DOD or that it would be socially optimal, because there may be unobserved benefits from the conventional-topic program. However, we conduct a simple cost-benefit analysis to explore the magnitude of the net benefits to the DOD from running an additional competition, comparing conventional topics with open topics, and find that the net benefits of an additional open-topic competition, at about $1 million, are an order of magnitude larger than those of an equivalent conventional-topic competition, at $1,000.
Note
This research brief is based on Sabrina T. Howell, Jason Rathje, John Van Reenen, and Jun Wong, “Opening Up Military Innovation: Causal Effects of ‘Bottom-Up’ Reforms to U.S. Defense Research,” NBER Working Paper no. 28700, April 2021, http://www.nber.org/papers/w28700.
