Judges whittled down the 13 semifinalists announced in July to six on Tuesday and said the shortlisted books were "moving, pacey, funny, violent, intriguing, horrifying and hopeful".
They were selected from an original 163 submissions.
The Booker Prize celebrates the best works of long-form fiction by writers of any nationality, written in English and published in the UK and/or Ireland between October 2025 and September 2026.
The winner stands to win stg50,000 ($A93,755) and a trophy named Iris, so-called after the 1978 winner Iris Murdoch.
"Most of these books were beginning to feel like old friends after at least two readings," said historian Mary Beard, head of the judging panel.
"And it was painful to have to let some of them go, with their now bent spines and dog-eared pages, their brilliant capacity to make us think, argue and sometimes disagree."
The list includes James' The Disappearers, about a group of gay men in his native Jamaica in what was a hostile environment in the 1980s and Stuart's John of John, a story of father-son secrets on a remote Scottish island.
James won the Booker in 2015 for A Brief History of Seven Killings and Stuart won in 2020 for Shuggie Bain.
If either were to win, they would be the fifth author to have been awarded the Booker Prize twice, joining Australian Peter Carey, JM Coetzee, Hilary Mantel and Margaret Atwood.
Also on the shortlist is Pulitzer Prize laureate Elizabeth Strout, whose heartfelt small-town tale The Things We Never Say, once again celebrates the lives of those in her native New England.
Strout's Oh William! was a finalist in 2022. She won the Pulitzer Prize for fiction in 2009 for Olive Kitteridge.
The others on the list are British authors Rebecca Perry for Medieval-to-modern tale May We Feed the King; 81-year-old M John Harrison, who would become the oldest winner of the award if his postapocalyptic tale The End of Everything wins; and the quirky campus novel Black Bag from Luke Kennard.
It's only the second time in the past decade that men have outnumbered women on the shortlist.
Beard said the judging panel, which included Pulp frontman Jarvis Cocker and American novelist Patricia Lockwood, concluded their deliberations "neatly propped up against one another, almost glowing".
"These novels range from the hilarious to the discomfiting, from the frighteningly strange to the alarmingly domestic, from the powerfully shocking to the intriguingly understated," she said.
Founded in 1969, the Booker Prize has a reputation for transforming writers' careers and is open to novels from any country published in the UK and Ireland. Last year's winner was Flesh by Hungarian-British writer David Szalay.
This year's winner will be announced on November 9.
