Declaring that “journalists should not be at legal risk for doing their jobs,” President Obama announced Thursday that he had directed Attorney General Eric Holder to review the Justice Department’s guidelines for spying on reporters in the course of leak investigations.

That would be more reassuring if Holder himself hadn’t signed off on a search warrant for the e‑mail correspondence of Fox News reporter James Rosen. The warrant application dubbed Rosen a “co-conspirator” in a violation of the Espionage Act, on the disturbing theory that asking a source to disclose classified information—as national security reporters necessarily do routinely—is itself a crime, even if publication of the same material is constitutionally protected. In other words, the president is asking the fox to investigate mysterious disappearances in the henhouse.

If reporters were looking to take comfort in the press shield bill the President asked Congress to revive in response to the Justice Department’s seizure of Associated Press phone records, they shouldn’t. Because the bill’s protections include a national security exception—and national security leak investigations are precisely when the government is most likely to spy on journalists—it seems unlikely to have made much difference in either the AP or Rosen cases. Indeed, as the Freedom of the Press Foundation points out, it could even make it easier for the government to obtain press records by overriding the common law safeguards some courts have recognized.

If the president really wants to demonstrate his concern for the potential of government spying to chill vital investigative reporting, he needs to take a very different approach, centered on greater transparency and a truly independent audit of Justice Department policy.

Transparency can begin with letting the public know exactly what the guidelines for investigating the press are—and how the Justice Department interprets them. As the FBI’s operational guidelines make clear, the rules requiring the press to be notified when their phone records are obtained only apply to subpoenas—not other secretive tools, such as National Security Letters, which can be issued without court approval. But the rules governing NSL demands for media records remain secret.

The Justice Department should also release any internal memos interpreting the rules governing press investigations. We know, for example, that there exists an informal 2009 opinion in which Justice Department lawyers analyzed how the rules would apply to sweeping demands—such as so-called “community of interest” requests—that can vacuum up a reporter’s records (among many others) even if the reporter is not specifically named as a target. Only brief excerpts of that opinion have been disclosed, thanks to a 2010 Inspector General report, and there is no way of knowing how many others remain secret.

Finally, we need an independent review—conducted by the Office of the Inspector General, not Attorney General Holder—to determine just how much surveillance of reporters has already occurred. It seems clear that the Justice Department does not think the current rules always require the press to be informed when they’ve been spied on: DOJ lawyers convinced a judge that the government never had to notify Rosen they’d read his e‑mails. And because demands for electronic records can be quite broad, it would be all too easy for the government to end up with sensitive information about journalistic investigations even when no reporter was explicitly targeted.

When Congress and the public know what the rules really are, and how they have been applied in practice, we can begin a serious conversation about what reforms are needed to protect press freedom. Asking Eric Holder to investigate Eric Holder, on the other hand, is unlikely to protect much of anything—except, perhaps, Eric Holder.