In a detailed, equation-laden lecture on the economics of extinction, Assistant Productivity Minister Andrew Leigh will declare rogue AI or an engineered pandemic are the most likely ways the human species could end over the coming 100 years.
"Extinction means the complete loss of our species. No survivors, no recovery, no second act," Dr Leigh will tell the Tasmanian Economic Society on Thursday.
"One estimate, from Australian philosopher Toby Ord, puts the odds of such a catastrophe at one-in-six over the coming century."
Policymakers could consider limits on what AI is allowed to do and how it approaches problems in a bid to reduce the risk, Dr Leigh will say.
"The danger is not only that such a system might 'go rogue' in the science-fiction sense," the former economist will say.
"It is that a government or corporation with access to machine intelligence far beyond human level could gain a strategic advantage so overwhelming that normal forms of competition, geostrategic balancing, political correction and international negotiation cease to work."
The assistant minister will suggest clamping down on "recursive self-improvement", a so-far theoretical concept where artificial intelligence repeatedly upgrades its own code.
He will also suggest a focus on "wise AI": where artificial intelligence is imbued with intellectual humility and sensitivity to context.
"The attraction of this approach is that it targets one of the deeper flaws in current systems," Dr Leigh will say.
"They are often forceful where they should be tentative, narrow where they should weigh competing perspectives, and overconfident where they should reveal uncertainty."
The federal government originally planned to introduce mandatory guardrails for the use of artificial intelligence, but instead backed away in favour of using existing laws to manage the growing industry.
Dr Leigh was among a group of senior Labor figures advocating for a light-touch approach on AI.
The other major risk to the human race is an engineered pandemic, either deliberately or accidentally released by a government or terrorist group, Dr Leigh will say.
"The relevant risks depend on pathogen development technologies that are evolving quickly, on defensive technologies that are rapidly changing, on actors whose intentions are difficult to observe," he will say.
Dr Leigh will suggest greater monitoring, tighter lab practices and better public health systems to help mitigate the risk.
12°C