The company said yesterday the appointment was with immediate effect.
It comes amid various changes to the board.
Leonhard Fischer, who had been a director since the company's 2011 IPO, had retired on December 31 also with immediate effect.
Carroll began her career as an exploration geologist and was at the helm of Anglo American between 2007-2013.
She has also been dubbed a role model for women in mining, having pushed for women to have more opportunities in the industry.
Carroll is currently a non-executive director of Hitachi, Baker Hughes Company and Pembina Pipeline Corp.
"She has extensive knowledge of the resources industry as well as strong non-executive director experience," Glencore chairman Tony Hayward said.
Glencore is due to release its 2020 production report later today and its preliminary annual results on February 16.
Shares in the company, which is capitalised about US$45 billion, closed down 1.2% in London yesterday.