Detecting Deepfakes in a cheap and effective way [StoryTime Saturdays]
I'm sharing my research in hopes that one of you can take it forward.
To learn more about the newsletter, check our detailed About Page + FAQs
To help me understand you better, please fill out this anonymous, 2-min survey. If you liked this post, make sure you hit the heart icon in this email.
Recommend this publication to Substack over here
I have something special for y’all.
A while back, I had worked on DeepFake detection. I had done the research to lay the groundwork for building a system to detect Deepfakes in a cheap and scalable manner. I have since switched to other projects, and these days because of everything going on, I can’t really dedicate too much time in building my solution. So I’m sharing my foundational research with you, my lovely reader. I hope one of you can take this forward and build a system to detect Deepfakes in a reliable and efficient way. Go make me proud.
Deepfakes are a problem for lots of organizations. The ability to create fake images and video has serious legal and social consequences in our society. Especially when we consider the great strides made in synthetic audio generation. If you think the misinformation surrounding politics is bad now, wait till people can create realistic videos of their political opponents ‘saying’ whatever they want them to say. Fake audio, pictures, and videos could be used to muddy legal trials or slander people in all kinds of ways.
There has been a lot of research into Deepfake detection. Billions of Dollars have been poured into creating, studying, and detecting them. There are some promising solutions created to catch fake videos and pictures. However, they are extremely expensive and would not be feasible as-is in many critical use cases. My research was into alternative ways that we could attain great results without relying on very costy deep-learning methods.
The Basic Idea
When we really analyze the technology used to create DeepFakes/Fake Images, we notice that all methods and architectures rely heavily on processing images, creating representations of various features, and then combining these to create the final image. This had to leave some kind of a ‘fingerprint’ that the image was tampered with. These fingerprints would be features of an image (often invisible to humans) that can be detected by humans.
By creating an ensemble of simple models that look for fingerprints, we would have a system that can reliably detect DeepFakes with minimal cost. Furthermore, by teaching it to find fingerprints- the solution could be adapted to catch newer DeepFakes (that would be built on different technologies) and into related topics (has an image/document been tampered with). An ensemble would have better robustness (different models can cover each other’s weaknesses).
For this idea to be viable we would need the following things to be true-
There is a meaningful difference between generated and real images, even if it is not immediately obvious to the human eye. Furthermore, these differences can be extracted and visualized/recorded through algorithms.
Using these differences will be competitive with state-of-the-art methods for detecting DeepFakes.
This approach of extraction is generalizable over different problems/variants. We can find evidence of artifacts in low/high-quality images, videos, and other variations.
To confirm this, I went through a lot of research papers.
Research Papers Referenced-
I worked on this project (on and off) between Jan-August 2021. I spent a lot of time reading through research papers, refining my idea, and looking into methods. In this article, I will be referencing the following papers-
DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection
FaceForensics++: Learning to Detect Manipulated Facial Images
These will provide you with enough proof of the viability of artifacts in generated images, and the strength of this idea. This will provide you with a base that you can build up from.
About Fake Faces being Generated-
This is the first part we will look at. We can utilize DeepFakes technology to create realistic photos of people, that have never existed. This has multiple security and identity implications. Scammers could create droves of fake personas online. This will allow them to shield themselves from law enforcement much better. If you think bots and fake profiles are a problem now, wait till this becomes mainstream. It’ll make things much worse. Take a look at the image below, created by the following website
Clearly, being able to detect images created by machines is a huge priority. Are there any fingerprints that will let us catch fake images? Turns out there are-
Some authors propose to analyse the internal GAN pipeline in order to detect different artifacts between real and fake images. In [55], the authors hypothesised that the colour is markedly different between real camera images and fake synthesis images. They proposed a detection system based on colour features and a linear Support Vector Machine (SVM) for the final classification, achieving a final 70.0% AUC for the best performance when evaluating with the NIST MFC2018 dataset [62]”
- Deepfakes and Beyond
Notice that the performance was achieved using SVMs and color features. These are just the basics. Adding more on top of this will not be super hard, and will lead to amazing results. However, things don’t end here. Turns out it might be possible to differentiate between the different GANs (and even different instances of the same architecture), using these fingerprints.
Not only is this very promising for the idea, but it also has some exciting implications for people interested in Deep Learning Research. Studying the different fingerprints can help us get a glance at different ML models, and actually see what’s going on behind the curtain. By comparing and contrasting different fingerprints, we will be able to learn more about the BlackBox that is Neural Networks.
On Swapped ID
This is how DeepFakes were introduced. And this has been the aspect of DeepFakes that has attracted the most attention. Using the DeepFake technology, we can morph people’s faces onto other people’s photos and videos.
This technology needs to be checked carefully. Using this, people could spread all kinds of misinformation about others. And the high quality of these fakes will make it impossible for people to differentiate the real from the fake. Imagine someone presenting a DeepFake as evidence in courts, and an innocent person being hurt because of this. We want to prevent this from happening. Fortunately, turns out DeepFakes do leave a couple of artifacts that can be caught.
This is a huge win for our system. Using the features detected from CNNs, we can detect deepfakes. Keep in mind, that this was written before Vision Transformers became popular. We have already covered that Vision Transformers ‘see’ different features in this article. Combining attention-based feature extraction into our pipeline should allow us to see even better results than the ones mentioned here.
As you can see, this solution has the potential to work really well with both the major use cases in Deepfake detection. Obviously building it would require a lot of legwork since you will have to integrate lots of features and figure out the best ensemble. However, to those of you looking for projects to work on (or anyone that is involved in DeepFake detection and need a novel approach), consider implementing this approach. I’ve compiled my research with some notes and points to consider in the following Google Slides. Feel free to use it as a base for your own implementations.
Loved the post? Hate it? Want to talk to me about your crypto portfolio? Get my predictions on the UCL or MMA PPVs? Want an in-depth analysis of the best chocolate milk brands? Reach out to me by replying to this email, in the comments, or using the links below.
Stay Woke. I’ll see you living the dream.
Go kill all,
Devansh <3
Reach out to me on:
Instagram: https://www.instagram.com/iseethings404/
Message me on Twitter: https://twitter.com/Machine01776819
My LinkedIn: https://www.linkedin.com/in/devansh-devansh-516004168/
My content:
Read my articles: https://rb.gy/zn1aiu
My YouTube: https://rb.gy/88iwdd
Get a free stock on Robinhood. No risk to you, so not using the link is losing free money: https://join.robinhood.com/fnud75