By Patrick McGroarty in Lagos, Nigeria, and Heidi Vogt in Yola, Nigeria 

Nigerians chose a former military dictator and anticorruption crusader to lead Africa's top economy, after a bruising election that raised hopes for closer relations with the U.S.--especially in the fight against Islamist insurgency Boko Haram.

Muhammadu Buhari won 54% of 29 million votes cast over the weekend, Nigeria's election commission said on Tuesday. President Goodluck Jonathan, the first incumbent to lose an election since democracy returned to the nation in 1999, won 45%. The result reflects the government's failure to tackle the Boko Haram threat and to spread oil wealth more equitably across a deeply divided nation.

A peaceful transition would be a first for Nigeria and a historic step for the continent over which it holds increasing sway. The country's fitful democracy has suffered postelection violence in the past--Human Rights Watch said more than 800 people were killed after Mr. Jonathan defeated Mr. Buhari in 2011--and a presidential race has never been this close in the country.

Mr. Buhari previously ruled for 20 months after a 1983 military coup and has run for president four times. His return to the helm could help Nigeria cement recent gains against Boko Haram. The group's militants have killed more than 20,000 people in the impoverished northeast, and displaced more than a million.

Mr. Buhari's push to battle corruption in Nigeria could lead to more U.S. security assistance, particularly if he investigates alleged civil-rights abuses, said John Campbell, who served as U.S. ambassador to Nigeria from 2004 until 2007.

Still, even with U.S. assistance, the Nigerian military has been so depleted in recent years that major changes could take time to develop.

"Buhari has been out of government for decades and so the question will be how fast he can assemble a team that can actually wrestle with all of the difficult issues that rebuilding the security services involves," Mr. Campbell said.

Many in Nigeria say that as a Muslim and a northerner, Mr. Buhari will be able to forge partnerships that eluded Mr. Jonathan.

"He has the experience and the exposure to pursue a lasting peace," said Gov. Kashim Shettima of Borno State, the epicenter of Boko Haram's onslaught. "I believe Gen. Buhari will bring to bear in the prosecution of the war a more holistic cocktail of options--military as well as economic solutions."

Mr. Buhari faces another tall order in cracking down on corruption and revive Nigeria's sagging economy. Plunging crude prices have shot big holes in the federal budget, and the naira currency is trading at record lows.

Nigerians waited in long lines this week to fill up on gas and withdraw cash from automated-teller machines. Demand for everything from Champagne to small loans has sagged along with the oil price.

"Things are almost at a standstill," said Olutoyin Okeowo, an executive at a group that normally imports 35,000 Toyota vehicles annually. Last year, business fell more than a third as sentiment soured. He hopes the change in government can turn things around.

"Now Nigerians are more politically aware. They know what to expect," Mr. Okeowo said.

But if what they expect is what Mr. Buhari promised in his campaign, they may be disappointed, said Zoran Milojevic, a director at the brokerage firm Auerbach Grayson in New York. "He was talking utopia: new jobs for everyone," Mr. Milojevic said. "That's not going to happen."

Still, from the commercial capital of Lagos to towns in the northeast beset by Boko Haram, Nigerians filled the streets on Tuesday to celebrate Mr. Buhari's victory.

"Jonathan didn't see everything we need--so many things," said Joseph Odeh, a 19-year-old assistant cook listening to radio reports of the president's defeat while perched on a pile of cinder blocks in Lagos. "Buhari will bring change to our country."

In Yola, cars honked as they sped through town and young men shouted and held up Buhari campaign posters. Others burned Jonathan posters.

"We are so happy! We are celebrating!" yelled 23-year-old Aminu Yahya. He said he expected Mr. Buhari to bring employment and security.

"We need Buhari to clear Boko Haram out of Nigeria," he said as dozens of others crowded around him, waving flashlights and shouting "Buhari!"

Fatima Umar, a 20-year-old student, grinned as she sauntered down the street with two friends, all holding Buhari posters in front of them.

"My future is not so bad anymore. We have a change of president, "she said.

Asked what she hoped Mr. Buhari would bring, she said: "Security, schools, water, roads, everything!"

Both candidates pledged to respect the results if they deemed them credible. That became a critical caveat this week as both camps complained about a chaotic voting process in some quarters of this country of 170 million, Africa's most populous.

Transition Monitoring Group, a Nigerian observer organization, said returns appeared to have been manipulated to increase the number of votes from Mr. Jonathan's southern strongholds. But the group said the suspect figures didn't affect the overall outcome.

In 1990, just three African countries were democracies, according to Freedom House, a Washington-based pro-democracy group. By 1994, the count was 18. Now, 19 qualify.

Angola and Zimbabwe are among countries that have staged what civil-rights groups called deeply flawed elections in recent years, extending the rule of presidents in office for three decades.

Other countries including Senegal and Zambia have managed peaceful transitions after opposition candidates won elections. But an opposition victory against an entrenched ruling party in a country with Nigeria's oil wealth is unprecedented, said Muna Ndulo, a professor at Cornell University Law School.

"Peaceful transfer of power through elections is possible in Africa," he said.

The shakeout for Nigeria's political class, which draws its power from the continent's top crude industry, will be widespread, said Stephen Chan, an international-relations professor at London's School of Oriental and African Studies. "Big beasts in the political landscape in Nigeria will be seeing the writing on the wall."

Some warned that Mr. Buhari needs to prove he can live up to the democratic mandate he has received. "Gen. Buhari would be making a mistake if he felt we will all go to sleep again now," said Ayo Obe, a Nigerian human-rights advocate.

Mr. Buhari's backers say he intends to meet that challenge. He came to power after a coup in the 1980s and ordered his police to whip people who were late for work. Now says he embraces democratic rights.

"He's a repentant democrat," said Rotimi Amaechi, governor of the oil-rich Rivers State and a campaign leader for Mr. Buhari.

Damian Paletta in Washington and Gbenga Akingbule in Abuja, Nigeria contributed to this article.

Write to Patrick McGroarty at patrick.mcgroarty@wsj.com and Heidi Vogt at heidi.vogt@wsj.com