Social workers in S’pore raise concerns over deepfake porn

Social workers and online safety advocates in Singapore are bracing themselves for a wave of AI-generated porn targeting victims here, as deepfake-on-demand services are popping up on publicly accessible online channels.

Such programs allow users to generate realistic deepfakes within seconds – and often for free – simply by uploading a picture of someone’s face, which the AI, or artificial intelligence, will combine with a digitally rendered body.

Celebrities have long been targeted by explicit deepfakes created using programs such as Photoshop, but now the accessibility and rapid processing of deepfake-on-demand programs lower the barrier to entry for anyone.

Widely circulated on platforms like Telegram, these apps have fuelled a deepfake pornography crisis in countries like South Korea, where sexually explicit deepfake images of women and young girls – often created based on school photos and social media content – are being widely shared in online chatrooms.

Social workers here have not encountered victims of deepfake harassment, but warn that South Korea’s situation should be a warning to the rest of the world.

The Straits Times found more than six Telegram channels offering deepfake services that allow users to develop “nude renders” using photos of real people within seconds.

These channels, which have up to 95,000 subscribers, are accessible to anyone.

Subscribers can generate deepfakes based on pictures they upload, as long as the target’s face is clearly visible.

In some channels, users upload pictures of real people and request the channels’ administrators to “undress” them, which they do for a fee.

Customers can tweak the footage to their fancy, such as by changing the size of the modified image’s body parts. And the more one pays, the more explicit the generated images can get.

“Deepfakes are a genre of pornography that we are seeing more among addicts,” said We Care Community Services’ counsellor Alvin Seng, who specialises in therapy for those with sex- and porn-related addiction.

The addiction recovery centre has not dealt with victims of deepfakes here as the trend is still new, but the issue has cropped up in an increasing number of talks with school partners, said Mr Seng.

“But it is still early and could be happening right now. I’d imagine that cases are coming soon,” he told ST.

Mr Seng said deepfakes should not be viewed lightly, no matter how unrealistic the renders seem.

“To a victim, seeing their face on a deepfake will feel very personal,” he said. “It is a violation of privacy and a betrayal of their trust.” 

One of the earliest cases of deepfake sexploitation reported here occurred in June, when at least four men were blackmailed by fraudsters with deepfakes using their faces.

One of the victims received an explicit video of himself from an unknown contact, who threatened to circulate it unless he paid $700.

It is not known how many more cases like that have occurred in Singapore. The police said then that they do not track the number of deepfake-related scams.

The deepfake threat has prompted counsellors at non-profit Touch Cyber Wellness to develop a module for parents and young people to address emerging online dangers, including image-based sexual abuse and cyber bullying, according to manager Shem Yao.

Set to launch in 2025, the module will educate young people on the impact of deepfakes, as well as how to spot them and respond if they encounter them.

Deepfakes should not be ignored just because they are fake, said Mr Yao.

“Victims often endure profound emotional distress, feeling ashamed and powerless over their own image,” he said. “This violation can lead to significant stress and lasting harm to their reputation.”

With time, deepfake images are sure to become even more realistic, he added.

Singapore has seen a number of cases involving online sexual harassment.

Notably, more than 44,000 Telegram users were part of a chat group formed in 2018 – SG Nasi Lemak – where lewd images of young girls and revenge pornography were circulated. It was shut down in 2019.

Image-based sexual abuse is among the most common types of online harms faced by clients of SheCares@SCWO, a support centre for victims of online harms, said assistant director of research Natalie Chia from SG Her Empowerment, which runs the centre.

A study by the non-profit organisation in 2023 found that nearly a tenth of internet users have personally experienced image-based sexual abuse, including deepfakes. Women were almost twice as likely to be sexually harassed online than men.

Advocates and experts have long called for new policies that will help online harassment victims seek redress swiftly.

This was echoed by Minister for Law and Home Affairs K. Shanmugam, who said in 2023 that the law needs to be expanded to enable victims of harmful online content to take action and protect themselves.

Since then, the authorities have allocated $50 million for a new Centre for Advanced Technologies in Online Safety which will research tools to detect harmful online content, including misinformation and deepfakes.

New measures to counter deepfakes of candidates during elections were also announced on Sept 11.

More can be done in Singapore to protect those who have limited options to seek redress, as online harassers are usually anonymous, said Cyber Youth Singapore president and chief executive Ben Chua.

The charity conducts cyber-wellness workshops and is revamping its course material to raise awareness of deepfakes.

Tech companies can be slow to remove content too, said Mr Chua.

Messaging app Telegram, for instance, has been slammed for its lack of cooperation with the authorities worldwide, leading to the arrest of its founder Pavel Durov in August over allegedly allowing criminal activity on the platform, including drug trafficking and child sexual abuse images.

Durov’s arrest reflects an increase in pressure to hold platforms responsible for not complying with the authorities in the fight against online harms, said Ms Chia, who called for stronger regulations to make it illegal to create, possess and share image-based sexual abuse materials.

Cyber-security expert Abhishek Singh of Check Point Software Technologies said Telegram is an ideal platform for vices to thrive.

This is because it allows large files to be shared across massive groups of up to 200,000 members, who can choose not to disclose their phone numbers to remain anonymous.

“It has brought dark web content to the masses on a public channel,” he added.

Deepfake software, in particular, has become more widely available since late 2023, said senior threat researcher David Sancho from cyber-security firm Trend Micro.

He said: “Any teenager can (use these apps). It’s not expensive and there are even support forums to guide users on how to do it.”

Mr Sancho added: “The trouble is, if you close one platform, another pops up.”

Advice to guard against deepfakes

Avoid allowing unfamiliar accounts to follow your social media accounts, and be mindful of the content that you post, although this can be hard to control in this day and age, said Mr Seng.

Victims of deepfake porn should make a police report and flag the content to the online platform, said Mr Yao.

They should collect evidence, including images of the online material, time and date, and the usernames of those who have shared it. Mr Yao said: “Evidence is vital, even if the specifics are painful to recall.”

Source: Straits Times © SPH Media Limited. Permission required for reproduction.

Qoo10 under Singapore police probe as vendors lodge reports over delayed payments New anti-money laundering laws in Singapore target kingpins’ profits from environmental crime