WASHINGTON — The Supreme Court on Monday wrestled with the scope of the state secrets doctrine, debating whether it allows the government merely to withhold evidence that it says could harm national security or whether it entitles the government to insist that lawsuits against it based on government secrets be dismissed entirely.

The case arose from the surveillance of Muslims in Southern California in 2006 and 2007 by Craig Monteilh, an F.B.I. informant. The surveillance failed to produce any public evidence of wrongdoing. To the contrary, after Mr. Monteilh started talking about jihad and violence at a mosque in Irvine, Calif., a community leader contacted the F.B.I. to report him.

Three of the men Mr. Monteilh spied on sued the F.B.I. and the agents responsible for directing him, claiming, among other things, that their right to exercise their religion had been violated. The government moved to dismiss those claims, invoking the state secrets privilege.

Edwin S. Kneedler, a lawyer for the federal government, told the justices on Monday that dismissal of the case was warranted “because the information concerning the reasons, the subjects, the sources and methods of this foreign intelligence investigation was so central to the case.”

Ahilan T. Arulanantham, a lawyer for the plaintiffs, said the case was not a typical one. His clients sought no secret evidence from the government, he said, and were confident they could prevail using public information.

The Supreme Court has issued just a few decisions on the state secrets privilege, and they seem to draw a distinction.

In a seminal 1953 case, United States v. Reynolds, the widows of men who died when a B-29 bomber crashed in Waycross, Ga., were denied access on state secrets grounds to information that could have proved their wrongful death claims. Under the Reynolds decision, Justice Neil M. Gorsuch said, “the state secret privilege allows the government to keep evidence away from a party but that generally the party is free to prove its case using other evidence.”

The Supreme Court endorsed a broader approach in a 2011 decision, General Dynamics v. United States, which ruled that national security considerations made it impossible for the court to even consider a multibillion-dollar dispute between military contractors and the government.

Writing for a unanimous court in the 2011 decision, Justice Antonin Scalia said that while the Reynolds case was about withholding given items of evidence, some cases — including ones about what he called “contracts to spy” — could not proceed at all because their very subject matter was secret.

Justice Gorsuch said Monday’s case, Federal Bureau of Investigation v. Fazaga, No. 20-828, may well be the first kind, in which the government has a choice: Does it want to disclose secret evidence to defend itself or does it want to risk losing a sum of money in the interest of national security?

Justice Gorsuch said he was wary of letting the government keep both its secrets and its money. “In a world in which the national security state is growing larger every day,” he said, “that’s quite a power.”

The U.S. Court of Appeals for the Ninth Circuit ruled in favor of the plaintiffs in 2019, but on a different ground. The appeals court said a provision of the Foreign Intelligence Surveillance Act of 1978, or FISA, established procedures for considering national security information and that the trial judge should have used them instead of dismissing the case after the government invoked the state secrets doctrine.

Several justices seemed unconvinced by that ruling and indicated that they were reluctant to parse the technical language of FISA to decide when and how it applies.

Justice Elena Kagan sketched out what she said might be “an attractive solution.” She said the case could be returned to the appeals court for a fresh look at “the easiest question in this case” — that of “when dismissal is appropriate” under the state secrets doctrine.

Last month, the Supreme Court heard arguments in yet another state secrets case, that one about whether the government could invoke national security to block testimony by two C.I.A. contractors who were instrumental in the brutal interrogations of the detainee known as Abu Zubaydah, who was waterboarded more than 60 times and is being held without charge at Guantánamo Bay.