Saying “I do” and calling someone your spouse who legally isn’t shouldn’t be a crime, but it can be in Utah. While polygamy—being lawfully married to multiple people—isn’t legal in any state, due to its unique history, Utah has some of the strictest anti-bigamy laws in the country. Which probably makes starring in a reality TV show based on your plural marriage not the best idea for Utahns.

Nevertheless, TLC’s Sister Wives revolves around Kodi Brown, his four partners (Meri, Janelle, Christine, and Robyn), and their 17 children. While Kodi is only legally married to one of women, he claims he is in a “spiritual union” with each of the others, and describes all four as his wives—and that puts the Browns on the wrong side of Utah’s bigamy law. The day after the show premiered in 2010, local authorities announced they were investigating the family.

Because the potential sentences are quite severe (five years for each of the women, and up to 20 years for Kodi), the Browns took preemptive action, filing a federal lawsuit challenging the constitutionality of Utah’s law. The district court agreed. In granting the Brown’s motion for summary judgment, the court held that because the law criminalizes “spiritual cohabitation” (arrangements where the participants claim to be part of multiple religious marriages, but make no attempt to obtain state recognition), it violated the First and Fourteenth Amendments, and was “facially unconstitutional.” The state has appealed that ruling.

Together with First Amendment scholar Eugene Volokh, Cato has filed an amicus brief urging the U.S. Court of Appeals for the Tenth Circuit to affirm the district court. Whether or not the Utah law violates the Browns’ religious liberty, it’s a clear affront to the First Amendment’s protection of free speech.

In Utah, it’s legal to have an “open” marriage and any number of unmarried consenting adults can live together, have sex with each other, pool their finances, and describe themselves as being in a long-term polyamorous relationship. They just can’t use the “M” word. Kodi Brown could have lived with all four women, legally. He could have legally lived with his “real” wife, and carried on long-term affairs with the other women, legally. It’s only because the Browns took the symbolic step of solemnizing their relationships with religious ceremonies, and held themselves out to the world as a married quintuple (even if only in a strictly spiritual sense) that they face prosecution.

With the Browns’ case, Utah isn’t really criminalizing bigamy—say, procuring multiple marriage licenses from county clerks while already being legally married—it criminalizes speech. The Supreme Court has made clear that there are only a handful of exceptions to the First Amendment’s protection. Only the most heinous and dangerous kinds of speech—things like child pornography and inciting violence—can be criminalized. Telling people you’re married, even if it isn’t legally true, isn’t the kind of harmful speech any government has the right to censor, let alone criminalize.

In fact, it isn’t harmful at all. The Browns and the TV show that brought them to national (and prosecutorial) attention aren’t hurting anyone—not themselves, not their children, and certainly not the public—by merely claiming to be spiritually wed. Whether plural marriages should be granted legal recognition has nothing to do with this case, which involves speech—not conduct—that the state doesn’t like.

The Tenth Circuit will hear oral argument in Brown v. Buhman this fall.