Overview
- Introduction – What are Deepfakes?
- Examples of Malicious Deepfakes
- Autoencoders
- Media Literacy and Detecting Deepfakes
- Wrapping Up
Introduction – What are Deepfakes?
The act of photo manipulation is an old one. It was has been used to colorize old WW1 images, but it has also been used for propaganda; Josef Stalin infamously manipulated photos so that his political opponent Leon Trotsky did not appear in important settings. Photo manipulation has been used for over 100 years to both captivate and deceive viewers.
Moving to the current time period, we consume not only images but also video and audio in our daily lives. The internet has facilitated a dramatic increase in video and audio sharing by third party individuals and organizations, in contrast to fixed TV and radio channels in the past. This is great for gaining new perspectives. However, it has together with innovations in artificial intelligence lead us to a new scary concept: deepfakes.
Deepfakes are synthetic media that have altered or generated, with the use of deep neural networks, so that the content is fake. This could be images and speech, but increasingly also videos. There are legitimate use-cases for deepfakes within video production. This could be aging/de-aging an actor for a role that spans a long time within the narrative. It could also be used as an alternative to reshooting close-up scenes where only minor changes are necessary. Yet, deepfakes presents more malicious applications than legitimate ones as of now.
In this blog post, I will give you some context for how deepfakes can be used maliciously. I will discuss autoencoders since this is one of the most common ways deepfakes are generated. Finally, I will talk about how media literacy and common sense is one of the most effective ways that we can combat deepfakes.
Examples of Malicious Deepfakes
Unfortunately, the majority of deepfake applications have a negative effect on individuals, companies, and society at large as of today. Let’s look at three malicious ways deepfakes can be used at each granularity:
Individual – Non-Consensual Pornography
The word deepfake actually originates from a username on Reddit. The user called deepfakes shared pornographic videos of celebrities, where deep neural networks had been used to add their faces to existing pornographic scenes. In recent years the technology has advanced, and it has now become possible to create non-consensual pornographic deepfakes (NCPD) content of everyday people more easily. The majority of NCPD victims are women and the goal is either bullying, revenge porn, or extorsion. Often the victims do not know who has made the content, and it can be almost impossible to remove it from the web. Even though it is "fake", it can still have damaging psychological effect on the victims and impact their ability to obtain jobs.
Company – Social Engineering Scams
For a long time, social engineering has been one of the top methods for gaining unauthorized access to sensitive company information such as employee passwords. Social engineering is, rather than finding technical security holes in software, to manipulate employees to give out information. This often takes the form of impersonating a colleague that needs quick access to an employees user account. Deepfakes brings new fears into this, as one can now create videos that further give the illusion that you are talking to a colleague. In a now famous instance, cybercriminals convinced a finance employee to transfer $25 million after a deepfake video call with the CFO.
Society – Spreading Misinformation
Looking more broadly, deepfakes presents an obstacle to truth in societal discourse. Deepfakes can be used to alter perceptions of political figures. There are many elections taking place in 2024 around the world, and there has been a major uptick during the last years in deepfakes used to sway political opinions. Deepfakes can also be used by interested parties to cast doubt on scientific evidence that are not beneficial to them. Much of our opinions are based on emotions, and there is thus a big advantage to portraying opponents as ill-informed or simply on a crusade.
It is also possible to use more simple methods such as photoshopping or inserting a word in a speech to distort reality; this is sometimes called cheapfakes or shallowfakes in contrast to deepfakes. While this has been an issue for some time, deepfakes lowers the barrier and heightens the impact. Today, almost anyone can create deepfake content without much expertise or knowledge.
Given that deepfakes can affect individuals, companies, and society as a whole, it is vital that we understand how they work and what measures we can take against them. Let’s start with understanding one way deepfakes are created.
Autoencoders
I now want to give an overview of autoencoders and how they are used to create deepfakes. You should know that other architectures such as Generative Adversarial Networks (GANs) can also be used to create deepfakes. I’ve chosen to focus on autoencoders since they illustrate the process of deepfakes most clearly in my opinion. I will only give a high-level overview of autoencoders here – just enough to discuss the detection of deepfakes in the next section.
Autoencoders are a specific type of a deep neural network that emulates compression and decompression. Specifically, the neural network first has an encoder part that compress a given image into a smaller dimensional space. The target space here is often called latent space or latent representation since it tries to extract the latent features (or essence) of an image.
Finally, there is an decoder part that tries to reconstruct the original image from the latent space. The goal is to go through both processes, encoding and decoding, while staying as close as possible to the original image. Hence the measure of success for an autoencoder is the similarity between the input and the output.
The latent space needs to be smaller than the input and output layers in size. In the extreme example, if the latent space is the same size as the input and output layers, then no compression takes place. On the other hand, if the size of the latent space is a single node, then this is clearly not enough information to reconstruct an image.
Below you can see how an autoencoder can reconstruct images from simple clothing items. For more information about this specific example, you can check out the Tensorflow Documentation on Autoencoders.
So what does autoencoders have to do with deepfakes? The trick to create deepfakes is as follows. You have two persons named Alice and Bob. Train two autoencoders, each of them using data consisting of faces from Alice and Bob, respectively. However, make sure that they have the same encoder, but different decoders. Then you do the following:
Take a picture of Alice. Pass it through the common encoder to get the latent features of Alice. Then you pass this through the decoder for Bob. Then the output will try to reconstruct the face of Bob with the latent features of Alice. This gives the deepfake-effect where the facial features of one person is superimposed on another. For more illustrations on how this work I recommend this blog post.
So now that we have an overarching understanding of how deepfakes can be made, what does this say about how we can respond to them?
Media Literacy and Detecting Deepfakes
Recently, several big players in tech such as Adobe, Microsoft, and Google pledged to work together to combat deep fakes in various ways. This is not an easy task. The reason is clear when we reflect on how autoencoders creates deepfakes.
The images are based on something real, and gradually fitted with possibly millions or billions of parameters to the wrong face. This means that unlike photoshopping where rough edges might appear, or taking autoclips out of context where the transitions are not smooth, deepfakes don’t really have a clear characteristic. For some time, deepfakes struggled with hands as in the famous pope Francis image:
You can see that the left hand is not really all there, and that the item Francis is holding is more merged into the hand than properly held. These artifacts will be removed with better models and we will be left with images and videos that could be real from a purely image standpoint.
So is there anything more we could do? Yes! Images, speech, and video do not exist in a vacuum, but is reliant on us to interpret their meaning and their reliability. So we’re back to classic media literacy as a tool that helps with misinformation resulting from deepfakes.
Asking questions such as "Is other sources confirming this?" or "Does this seem plausible?" goes a long way. Is the information radically different from other information that the person has said before? Does anyone benefit from me believing this? Be critical of the content, author, and other interests! The situational dependence can often be enough to detect deepfakes.
A recent example is a deepfake call from U.S. president Biden to Democrat voters in New Hampshire to not vote in the New Hampshire primary, and to save their vote for when it counts; for the election in November. We don’t need deepfake-detecting technology to understand with the tiniest media literacy that this does not make any sense. The sitting president do not call individual residents and tell them to not vote for their party. You would find no official statements from the white house anywhere near this if you looked, and this is radically different from what you would expect. Almost any question you could ask yourself about the situation should make you really skeptical.
Seek out news agencies that have a long history of unbiased reporting and fact-checking. You should not outsource all your critical thinking to a third party, but think of this analogous to an antivirus program. The antivirus program will filter out much, but it is ultimately your responsibility to not download everything you come across on the internet.
When it comes to social engineering fraud, the same amount of critical awareness needs to be present. If someone is asking you through speech or a simple short video to do something drastic (like transfer a large sum of money or to give up account information), then ask for verification from the person on a different forum or in person. This simple rule makes social engineering a lot more difficult to achieve.
For victims of non-consensual pornographic deepfakes, there is unfortunately less measures that can be taken as of now. The legal system has not caught up to the fact that this is happening, or how to properly respond to this. While you should be extra careful to not put out pictures of children on the open internet, it is unrealistic that adults will not have pictures online. The best thing we can do as of now is to be aware that this is a possibility, and to not discount someone in a hiring process because of this. Understanding how uncomfortable being a victim of this can be can make us more empathetic when we learn that it has happened to someone we know.
Wrapping Up
I hope you got an overview of how deepfakes work and how to combat malicious deepfakes. If you are interested in AI, Data Science, or data engineering, then feel free to follow me or connect on LinkedIn.
Like my writing? Check out some of my other posts for more content:
