The elections are crucial as the new ANC president is also likely to become the country's next president after elections in 2019.
Zuma's presidency has been plagued by series of scandals and corruption allegations since 2009 when he first became the president, which has also caused a rift within the party and tainted the party's image home and overseas.
Deputy President Cyril Ramaphosa and former cabinet minister Nkosazana Dlamini-Zuma were the only candidates nominated for the leadership position during the conference.
The announcement of the result was delayed by a recount of votes after a delegate supporting Mrs Dlamini-Zuma was said to have lodged an objection.
With 4,708 votes cast, Mr Ramaphosa took 51.8% of the vote against 48.2% for his opponent.
Ramaphosa, 65, a former trade union leader, led the historic negotiations in the 1990s to end apartheid before launching a business career that made him one of South Africa's wealthiest men. Ms. Dlamini-Zuma had promised the radical economic transformation her ex-husband never delivered. His reign has been hit with allegations of scandal and corruption which has seen the party of Nelson Mandela's liberation movement plummet in popularity among South Africans.
"I hope you will cooperate with the new leadership.as we move to the 2019 elections", Baleka Mbete, the party chairwoman, told delegates.
The mood at the party conclave, held once every five years, was jubilant on Saturday as delegates arrived at a conference center in luxury buses, clad in the ANC's yellow and green colors.
He has pledged to introduce a minimum wage for workers and will consider privatising some of South Africa's state-owned conglomerates. Keeping the ANC together has been a key talking point at the gathering.