Spoiler alert: Yes, there have been a number of well-documented abuses of power under the Patriot Act, as well as subsequent surveillance legislation. I’ll detail some of them in a moment, and the ACLU has a more thorough report. As that report should make abundantly clear, it’s really only a handful of the Patriot Act’s dozens of provisions that civil libertarians have serious problems with, and our concern in most cases is to improve them, not eliminate them. But there are three more general points I think it’s important to clarify before we delve into specifics.

First, intelligence surveillance demands especially robust safeguards precisely because it’s so inherently secretive and therefore lacks many of the checks that exist in criminal investigations. Ordinary wiretaps, for instance, are always eventually disclosed to their targets and typically meant to be used as evidence in a trial, in which defense attorneys have powerful incentives to scrutinize the record closely for misbehavior. Intelligence taps are covert: Targets may never learn they’ve been spied on, and criminal prosecution may not be the goal. Both National Security Letters (NSLs) and so-called Section 215 orders for business records and other “tangible things” come with gag orders that remain in place long after the investigation is complete. The banks and telecommunications providers served with these orders have little incentive to expend time, money and energy challenging demands for information in court, and the few that have done so are prohibited from talking about why they believe the requests are illegitimate. Congressional oversight can help — the Church Committee in the 1970s uncovered a stunning array of abuses stretching back decades — but, absent some major scandal, tends to be limited in scope.

It also doesn’t help that among the already documented abuses is the intelligence community’s failure to fully and accurately inform Congress about their use of those powers. When legislators do become aware of problems, their ability to mobilize support for reform is hampered by their own inability to go public with their concerns. Sen. Russ Feingold (D‑Wis.) has repeatedly said that he believes those Section 215 orders are being misused, but citizens have no real way to evaluate the claim.

Second, “abuse” typically connotes a violation of the law, or at least the internal rules governing surveillance. But there’s good reason to be concerned about some of these powers even when they’re used precisely as intended. In recent hearings, Justice Department officials made quite clear that they vacuum up reams of telephone, Internet and bank records in the preliminary phases of investigations to “close down leads” and spot suspicious patterns. This is another way of saying that the vast majority of people surveilled are innocent — not when this authority is misused, but by design. The FBI issues tens of thousands of National Security Letters each year, and the majority seek information about U.S. citizens. That information isn’t discarded; it goes into a massive FBI database containing literally billions of records. Simple math suggests there just aren’t that many terrorists out there.

So-called sneak-and-peek warrants that were sold as a vital tool for terrorism investigations are now overwhelmingly used in ordinary criminal investigations — contrary to what the public was told, certainly, but in accordance with the letter of the law. If we look back to those abuses uncovered by the Church Committee, we find some cases in which surveillance of journalists, activists and even Supreme Court justices was initiated for manifestly unlawful political purposes. But just as often, information gathered in the course of an initially legitimate national security investigation was later used for an illegitimate political end.

Finally, it’s important to be clear on just what sort of abuse we should be worried about. It is not primarily that investigators will decide to start spying on average Americans — though we have learned that some National Security Agency operators find it hilarious to pass around recordings of U.S. soldiers’ pillow talk. It’s that all this information, even if acquired for legitimate purposes, gives the people who hold it enormous political power. A reporter, activist or senator who crossed J. Edgar Hoover might find his career ruined by an embarrassing leak to the press.

All that said, what do we know about actual abuses so far? We know two successive inspector general reports found endemic misuse of NSLs, including requests for information the FBI wasn’t entitled to obtain and “exigent letters” sent when no real emergency existed. We know that in at least one case, NSLs were used to obtain records after a judge rejected an attempt to obtain them via court order, citing 1st Amendment concerns. We know expanded National Security Agency wiretap powers approved last year have led to substantial “overcollection” of Americans’ purely domestic communications — including Bill Clinton’s e‑mails. Many of these can be put down to sloppiness or ineptitude, though they’re no less troubling.

But history teaches that if there are more deliberate abuses, we probably won’t know about them for decades to come. I don’t know about you, but I’d rather not wait to find out.

This is the second part in a three-part series.

Part I | Part II | Part III