MOGADISHU, Somalia — The C.I.A. convoy rolled out of Mogadishu in the dead of night, headed south along a crumbling ocean road that led deep into territory controlled by Al Shabab, one of Africa’s deadliest militant groups.

The vehicles halted at a seaside village where American and Somali paramilitaries poured out, storming a house and killing several militants, Somali officials said. But one man escaped, sprinted to an explosives-filled vehicle primed for a suicide bombing, and hit the detonator.

The blast last November killed three Somalis and grievously wounded an American: Michael Goodboe, 54, a C.I.A. paramilitary specialist and former Navy SEAL, who was airlifted to a U.S. military hospital in Germany. He died 17 days later.

His was a rare American fatality in the decade-old shadow war against Al Shabab, the world’s wealthiest and arguably most dangerous Al Qaeda affiliate. But Mr. Goodboe was also a casualty of an American way of war that has flourished since the terrorist attacks on the United States in 2001, now under greater scrutiny than ever.

The United States’ most ambitious response to the 9/11 attacks was in Afghanistan, where tens of thousands of troops were dispatched to banish extremists and rebuild the country — a mission that recently ended in crushing failure with the chaotic American withdrawal.

But in Somalia, as in countries like Yemen and Syria, the United States turned to a different playbook, eschewing major troop deployments in favor of spies, Special Operations raids and drone strikes. Private contractors and local fighters were recruited for risky tasks. The mission was narrow at first, a hunt for Qaeda fugitives, only later expanding to include fighting Al Shabab and building up Somali security forces.

Now that playbook is also failing. As in Afghanistan, the American mission has been stymied by an alliance with a weak, notoriously corrupt local government, an intractable homegrown insurgency and the United States’ own errors, such as drone strikes that have killed civilians.

As a result, Al Shabab are at their strongest in years. They roam the countryside, bomb cities and run an undercover state, complete with courts, extortion rackets and parallel taxes, that netted at least $120 million last year, by American government estimates.

Al Shabab also appear to have designs on the United States, with the arrest in 2019 of a militant while taking flying lessons in the Philippines, allegedly to commit another 9/11-style attack on the United States. But critics of the American approach in Somalia, including some military officers, say that the threat to the homeland has been exaggerated and that Washington’s own policies only boost the extremists they seek to defeat.

Biden administration officials deny the mission in Somalia has failed, but they say they are cleareyed about its shortcomings. The administration could unveil a new Somalia policy in coming weeks, some officials said.

The U.S. government has been reluctant to commit troops to Somalia since the “Black Hawk Down” episode of 1993, when Somali militia fighters killed 18 American service members in a blazing battle later depicted in books and Hollywood movies. After that fiasco, the U.S. withdrew from Somalia for more than a decade.

Americans eventually returned in small numbers — covert operatives, soldiers and, lastly, diplomats who are bunkered into a windowless, penitentiary-style embassy at the Mogadishu airport that opened in 2018. Fearing another bloody debacle, they rarely venture out.

Nearby lies the C.I.A. compound, where the air crackles with gunfire at night as the Americans train a small Somali paramilitary force that spearheads anti-Shabab operations.

There are now fewer than 100 American troops in Somalia, mostly in intelligence and support roles. In January, former President Donald J. Trump moved most of the 700-member force across the borders to Kenya and Djibouti, though it continues to conduct strikes in Somalia, and train troops.

Outside the wire, Mogadishu has been transformed in recent years with the help of African Union peacekeepers who patrol the streets. There are trendy cafes, gleaming apartment blocks and fast, cheap internet. The city’s Lido beach is packed on weekends. Piracy, a major international preoccupation a decade ago, has largely vanished.

Yet this progress hangs by a fraying thread. Somalia’s fractious political elite is riven by disputes that erupted briefly into violence this year. After the Taliban’s victory in Afghanistan, gleeful Shabab militants distributed sweets in celebration, hoping they too might wait out the foreigners and seize power.

Other Somalis worried that Washington would abandon them next. “It rang frightening alarm bells,” said Abdihakim Ante, a former Somali government adviser.

The fate of Afghanistan “shows how quickly things can change,” said Stephen Schwartz, a former U.S. ambassador to Somalia. “Somalia has no time to waste.”

The arc of the faltering U.S. mission in Somalia can be seen in the stories of two men, an American and a Somali, on opposite sides of the fight.

A Forever Warrior in a Forgotten War

Michael Goodboe was the archetypal elite fighter of the post-9/11 era.

A member of the elite SEAL Team Six, he deployed to Afghanistan within weeks of the Sept. 11 attacks. He worked from the C.I.A.’s temporary station at the Ariana Hotel in Kabul and joined the first “Omega team” — a highly classified unit combining Special Forces operators and C.I.A. paramilitaries that led the hunt for Osama bin Laden and other fugitives.

Colleagues admired Mr. Goodboe, known as “Goody,” for his easy manner, steady temperament and keen sense of purpose — qualities that stood out in the SEALs’ swaggering subculture, and helped him forge close relationships with the Afghan, and later Somali, troops he helped to train, they said.

Many SEALs “do the minimum time, get their trident” — the SEAL symbol, worn on naval uniforms — “and write a book,” said Capt. Christopher Rohrbach, a 24-year SEAL who has served in East Africa, Afghanistan and Iraq.

But Mr. Goodboe “was a team guy,” he said. “He was there for the greater good.”

After retiring from the Navy in 2009 with a clutch of medals, Mr. Goodboe joined the C.I.A.’s paramilitary wing, now called the Special Activities Center — a clandestine group of about 200 fighters, the vanguard of the agency’s far-flung wars. The job eventually took him to Somalia.

The C.I.A. had a checkered history there.

In the mid 2000s, C.I.A. officers based in Nairobi, Kenya, led the American return to Somalia. They regularly flew into a remote airstrip outside Mogadishu, carrying suitcases of money for a coalition of warlords who had promised to help hunt Al Qaeda.

But the operation backfired badly in June 2006 when public hostility toward those warlords galvanized support for an Islamist group, the Islamic Courts Union, that swept to power briefly.

A year later, Al Shabab emerged. The C.I.A. station chief overseeing support for the warlords was transferred.

The C.I.A. returned to Somalia in 2009, establishing a secure base at the Mogadishu airport and teaming up with the National Intelligence Security Agency, Somalia’s fledgling spy agency. The Americans also joined the fight against Al Shabab.

C.I.A. snipers deployed to rooftops around the sprawling Bakara Market, then a Shabab stronghold, picking off Islamist fighters from up to a mile away, said a retired Somali intelligence official who worked with the Americans.

In 2011, Somali security forces killed Fazul Abdullah Mohammed, a Qaeda leader behind the 1998 bombings of U.S. embassies in Kenya and Tanzania, and seized a trove of valuable intelligence, including plots to bomb the elite British school Eton and London’s Ritz Hotel.

The Somalis handed everything to the C.I.A., including a memento — the dead militant’s unusual model of rifle, said Hussein Sheikh-Ali, then a senior Somali intelligence official and later Somalia’s national security adviser. “It was a turning point” in the relationship between the Americans and Somalis, he said.

But as the fruits of cooperation became clear, so did the costs. Human rights groups and United Nations investigators accused Somalia’s spy agency of torturing detainees and using children as spies. Some detainees recently accused the C.I.A. of complicity in torture.

In 2015, the C.I.A. station chief in Mogadishu pressed for the removal of Gen. Abdirahman Turyare, the Somali intelligence chief, accusing him of corruption and mismanagement. General Turyare said he was the victim of American highhandedness and arrogance.

“I refused to bow before the self-made king,” he said in an interview with The Times, referring to the station chief.

The dispute dragged on for a year as State Department leaders appealed to President Hassan Sheikh Mohamud, who comes from then same clan, to take action against General Turyare. Only after Britain’s foreign secretary, Philip Hammond, told Somali leaders that their relationship was also endangered by the dispute was General Turyare removed.

At the heart of that dispute, several Somali officials said, was control of Gaashaan, a paramilitary force officially part of the Somali spy agency, but in reality led by the C.I.A.

Since 2009, the C.I.A. has been training Gaashaan, which means “shield,” and it has grown into an elite force of 300 troops. Among the trainers was Mr. Goodboe. Gaashaan uses cellphone tracking technology to hunt Shabab commanders, mobilizes against militants when they strike Mogadishu and joins with C.I.A. paramilitary specialists for raids.

By late last year, when Mr. Goodboe arrived in Somalia for another monthslong tour, the C.I.A. and Gaashaan had turned their focus to one Shabab leader in particular — a bomb maker with a background in television.

The Master Bomb Maker

Somalis who once knew him say that Abdullahi Osman Mohamed was an unlikely jihadist kingpin.

“A friendly, energetic guy with a baby face,” recalled Mahmood, a former colleague who gave part of his name to speak freely about one of Somalia’s most dangerous men. “Very smart, very handsome. I often wonder how he became terrorist number one.”

In September 2020 Secretary of State Mike Pompeo signed an order designating Mr. Mohamed, also known as “Engineer Ismail,” as a “global terrorist.” According to the United States, he is Al Shabab’s senior explosives expert, head of their Al Kataib propaganda wing and a special adviser to the supreme leader, Ahmed Diriye.

Some Somalis go further, saying that Mr. Mohamed is one of two deputy Shabab leaders.

He was the intended target of the ill-fated November raid in which Mr. Goodboe was fatally injured, according to a retired Somali official and a senior American official who refused to be identified to discuss sensitive intelligence.

The C.I.A. declined to comment. A U.S. official would not say who the target was.

In many ways, Mr. Mohamed typifies the mix of resourcefulness and ruthlessness that has made Al Shabab such a formidable enemy.

He came from a conservative, middle-class Mogadishu family. His father worked for Al Haramain Islamic Foundation, a Saudi charity the United States accused of links to Al Qaeda in 2002.

Mr. Mohamed, then in his early 20s, graduated from university in Sudan in 2006 and began working as a studio technician for Al Jazeera in Mogadishu. His boss, the station’s Mogadishu bureau chief, Fahad Yasin, later went into politics and became Somalia’s spy chief — a striking illustration of the Somali conflict’s complex layers. Mr. Mohamed later spent time at Al Jazeera’s headquarters in Qatar for training.

It was an especially tumultuous time in Somalia. Ethiopia, backed by the United States, invaded in 2006 to oust the Islamic Courts Union. American warplanes bombed Islamist forces.

Like many Somalis, Mr. Mohamed was enraged, said a family friend who spoke on the condition of anonymity to avoid reprisals. Ethiopia and Somalia had fought a major war in 1977-78 and remained bitter rivals.

Mr. Mohamed began moonlighting for Al Shabab.

Al Shabab, or “the youth,” were a faction of the defeated Islamic Courts Union. Ousted from Mogadishu, they fled to southern Somalia and launched a guerrilla war, including bombings and assassinations, against Ethiopian soldiers.

By 2008, Al Shabab had become the most radical and powerful armed faction in Somalia, with thousands of recruits. Their leaders condemned what they called American crimes against Muslims across the globe. The U.S. State Department designated Al Shabab as a terrorist organization in 2008. In 2012, the group pledged allegiance to Al Qaeda.

Al Shabab’s broad goal is to establish their vision of an Islamic state in Somalia. In areas they control they have banned music and movies, and impose harsh punishments like stoning accused adulterers and amputating the limbs of accused thieves.

Mr. Mohamed first helped Al Shabab with propaganda, the friend said. Later, as American airstrikes killed successive Shabab explosives experts, the young militant, whose degree was in electrical engineering, was promoted to take their place.

Al Shabab went on to perpetrate a series of horrific attacks including, in 2017, a truck bombing in central Mogadishu that killed at least 587 people — one of the deadliest terrorist acts in modern world history.

As Shabab leaders were killed off and the Danab, an elite, American-trained Somali commando unit, evolved into a powerful anti-Shabab tool, the militants adapted.

They melted into the countryside, where they were harder to hit, and established a rudimentary parallel state with its own courts, bureaucracy and road tolls.

Al Shabab’s influence also extends into the heart of Mogadishu, where the group and its supporters have infiltrated Parliament, the business community and the security services, officials say. The Western-backed Somali government is ineffectual in comparison, divided by the corrosive clan politics that have crippled international efforts to unify Somalia’s security services. Graft is rampant; Transparency International ranks Somalia, along with South Sudan, as the most corrupt countries in the world.

A Somali intelligence officer in an interview listed the Shabab tax rates at Mogadishu port — $90 to import a regular container; $150 for a large one. He produced a neatly written receipt, provided by a city resident, for a $250 payment to register a recent land sale on the edge of Mogadishu — made out to Al Shabab.

While the militants enforce their writ with violence, many ordinary Somalis grudgingly appreciate their basic services. Even middle-class Mogadishu residents prefer to settle some disputes at Shabab courts that convene under trees in the surrounding countryside.

“If you go to the Somali courts for justice you won’t get it, particularly in property disputes,” said Abdirazak Mohamed, a member of Parliament. “Corruption is pervasive and the judges can’t enforce their decisions. But Al Shabab can do that.”

Somalia’s national army officially has 24,000 troops, but in reality is one-fifth that size, a senior American official said.

American analysts estimate that Al Shabab command anywhere from 5,000 to 10,000 fighters. Under Mr. Mohamed, their bombs have grown more sophisticated and powerful.

The group uses its hold on Mogadishu port to smuggle in large volumes of explosive materials and Chinese-made trigger devices, two U.S. officials said. Last October, Somali authorities intercepted 79 tons of sulfuric acid, an ingredient in roadside bombs.

In January, a bomb struck an armored convoy with American-trained Danab commandos, traveling toward Baledogle, a base 70 miles from Mogadishu.

The blast badly wounded the Danab commander, Maj. Ahmed Abdullahi, who was airlifted to Turkey, and killed a South African employee of Bancroft Global Development, an American contractor that recruits and trains Danab fighters. The South African, Stephen Potgieter, was the seventh Bancroft employee to die in Somalia since 2009, said Michael Stock, the company’s chief executive.

Mr. Mohamed’s growing reputation for chaos and bloodshed has made him a highly respected leader inside Shabab ranks, Somali and Western officials said.

To those pursuing him, he is an elusive figure, always out of reach.

American Mistakes

As in Afghanistan, America’s campaign in Somalia has been undermined by its own deadly misfires.

After an American missile struck a farmhouse near Jilib, southern Somalia, in February 2020, the military said it had killed a “terrorist.” Months later the military admitted that it had, in fact, killed an 18-year-old schoolgirl named Nurto Kusow Omar Abukar.

The attack also injured her sister, Fatima, then 14, who indicated during an interview where a missile fragment pierced her body. She wakes up screaming from nightmares. “I don’t want to say what I see,” she said.

Airstrikes in Somalia surged from 2017, when President Trump eased combat rules intended to protect civilians. The military admits killing several civilians but has not paid compensation — a contrast with Afghanistan and Iraq, where just in 2019 the U.S. made hundreds of payments worth $1.5 million for death, injury or property damage.

In an email, Nicole D. Kirschmann, a spokeswoman for the United States Africa Command, declined to explain why no such payments were made in Somalia. But she said that Somali officials reviewed and approved each compensation decision.

Although Washington is by far the largest foreign donor to Somalia, giving $500 million in 2020, few Somalis see evidence of that assistance because Somali partner organizations hide their American ties to avoid Shabab reprisals. Even bags of American food aid do not carry a U.S. logo.

In contrast, Turkey donates less money but spends it on high-profile projects — new roads, mosques and hospitals — that are promoted with the Turkish flag. Turkey is hugely popular in Somalia.

The American aversion to casualties among U.S. personnel has created an unusually high dependence on private contractors. The best known, Bancroft, hires retired soldiers largely from Eastern Europe, Africa and the French Foreign Legion to recruit and train Somali forces. Bancroft’s property wing built the fortresslike Mogadishu embassy and leases it to the State Department; a senior official said it is among the most expensive to operate in Africa.

Bancroft’s financial practices came under scrutiny this year when the government examined its $33 million contract to train Danab and African Union troops.

In a report published in July, the State Department Inspector General said the department had paid Bancroft $4.1 million for expenses that were not authorized under its contract, including $3.78 million in “incentive compensation” for its personnel — and said the money should be recovered.

In an email Mr. Stock, the Bancroft C.E.O., denied any wrongdoing.

The C.I.A., meanwhile, is struggling to keep its distance from a political storm surrounding a key ally, the Somali spymaster, Fahad Yasin.

President Mohamed Abdullahi Mohamed, backed by Mr. Yasin, postponed an election that was supposed to be held in February, extending his term. Critics accused him of a blatant power grab and rival security factions exchanged gunfire in central Mogadishu, evoking fears that the country’s fragile transition to democracy was collapsing.

American officials proposed slapping sanctions on Mr. Yasin to force him to back down, two Western officials said. But the C.I.A. staunchly opposed the idea, apparently to protect its counterterrorism interests.

It sent the wrong signal to Somali officials about America’s priorities, one of the officials said: “They see the mouth and the body doing two different things. It’s confusing.”

Biden’s choice

Current U.S. officials say missteps by the Trump administration complicated the situation in Somalia. The Biden administration is mulling whether to send back some of the troops Mr. Trump withdrew in January.

Critics of that approach say that Al Shabab are principally focused on East Africa and that their ability to strike in the United States has been overblown.

“If it ever was to pose an existential threat to the U.S. it’s because our presence in Somalia made it so,” said Captain Rohrbach, the active-duty SEAL.

U.S. officials say the experience of Afghanistan shows that success cannot be defined as remaking a government or society, and that the mission in Somalia has paid off by disrupting Al Shabab. Mr. Goodboe, according to friends, judged his work by a similar yardstick: whether terrorists could threaten Americans or the United States.

Still, some analysts say the United States needs to contemplate a totally new approach in Somalia, including a political settlement with Al Shabab, or face the prospect of being trapped in another “forever war” with an inglorious end.

A memorial wall at C.I.A. headquarters in Langley, Va., honors agency employees killed in the line of duty. It has 137 stars — four of them added last May. Though the identity of those four officers remains classified, one was Mr. Goodboe — a final, anonymous tribute.

“Engineer Ismail” is believed to be still at large. In the latest Shabab bomb attack, on Sept. 25, a suicide bomber hit a checkpoint in downtown Mogadishu, a few hundred yards from the presidential villa. Eight people were killed, including a woman and two children.

Reporting was contributed by Christina Goldbaum, John Ismay and Mark Mazzetti.