Education Minister Ben Carroll said changes to the respectful relationships program were being assessed after a teenage boy was arrested in relation to 50 girls' photos from Bacchus Marsh Grammar being manipulated.
"We are having a broader look at it because we think the pornography and other parts of respectful relationships could possibly be updated, and that is something we are having a look at," he told reporters.
"It's something we will continue to be vigilant about."
Federal education minister Jason Clare echoed his Victorian counterpart, saying the manipulation of the girls' photos was "terrifying".
"It's hard to think of something more frightening or more awful than the use of AI technology to perpetrate these false images and circulate them," he said.
"There is no silver bullet here but there is a role to play in the classroom that boys and young men, in particular, (need to) understand how important this is."
Deepfake apps with the ability to impose women's heads on naked bodies are being pushed on social media as experts warn more advanced software could be on the horizon.
Laws cracking down on the sharing of sexually explicit AI-generated images and deepfakes without consent were recently introduced to federal parliament.
However, concerns remain about the availability of online programs to create deepfake images.
Manipulated images were shared of students at Bacchus Marsh Grammar. (Diego Fedele/AAP PHOTOS)
University of Melbourne professor Jeannie Paterson, who specialises in the legal aspects of AI, said while there are laws in place to punish those sharing deepfakes, once they have been shared it may be too late to avoid harm.
"The law is expressed in a way that covers synthetically generated non-consensual intimate images as well as (real) images, to deal with this behaviour," she said.
"The problem is, we're not aware of it happening until it's out there, and once it's out there on the internet, it's just hard to contain."
Professor Paterson said the state government's plan to expand their respectful relationships program did not go far enough.
"This isn't about relationships, this is about exercise of power," she said.
"There is no relationship here – those images have been designed to hurt and humiliate."
Bacchus Marsh Grammar has said it is counselling students after it was made aware of the production and circulation of video content.
Victoria Police officers arrested the teenager over the explicit images circulated online and he was released pending further inquiries.
The investigation remains ongoing.
15°C