Abstract
The rapid advancement of deep learning technologies has enabled the creation of highly realistic deepfake videos and synthetic audio, posing serious threats to digital security and media authenticity. Traditional detection methods that rely solely on visual artifacts are no longer sufficient to identify sophisticated manipulations. This project proposes a hybrid multimodal framework for real-time deepfake detection by analyzing both video and audio streams simultaneously. The system extracts visual features from video frames and acoustic features from speech signals, models their temporal dependencies, and combines them using feature fusion techniques. The integrated model performs four-class classification to distinguish between RealVideo–RealAudio, RealVideo–FakeAudio, FakeVideo–RealAudio, and FakeVideo–FakeAudio. The proposed framework is designed for real-time deployment using webcam and microphone input, providing stable predictions with confidence scores. Experimental results demonstrate that multimodal learning improves detection accuracy and robustness compared to single-modality approaches.
Keywords
Deepfake Detection, Multimodal Learning, Audio-Visual Fusion, Real-Time Detection, MobileNetV2, Bidirectional LSTM, MFCC, Temporal Modeling, Feature Fusion, Deep Learning.







1. Introduction
The rapid advancement of deep learning and generative models has led to the widespread creation of deepfake videos and synthetic audio. These manipulated media contents pose serious threats to digital security, social trust, political stability, and biometric authentication systems. As deepfake generation techniques become more sophisticated, traditional detection methods based solely on visual artifacts are no longer sufficient.
To address this issue, this project proposes a hybrid multimodal deep learning framework capable of detecting deepfake content in real time by analyzing both video and audio streams simultaneously.
Problem Statement
Existing deepfake detection systems primarily focus on binary classification (Real vs Fake) and rely mostly on visual features. However, modern deepfake techniques can manipulate video and audio independently or simultaneously. Therefore, there is a need for a robust system that can analyze both modalities and accurately classify different types of manipulation under real-time conditions.
Proposed System
The proposed system uses a multimodal architecture with two parallel branches:
• Video Branch:
Sequential video frames are processed using MobileNetV2 for spatial feature extraction and Bidirectional LSTM for temporal modeling.
• Audio Branch:
Audio signals are converted into MFCC, delta, and delta-delta features and processed using Bidirectional LSTM to capture speech patterns.
The extracted features from both branches are fused and passed through fully connected layers for classification into four categories:
1. RealVideo – RealAudio
2. RealVideo – FakeAudio
3. FakeVideo – RealAudio
4. FakeVideo – FakeAudio
The system is designed for real-time deployment using webcam and microphone input.
Objectives
• To develop a multimodal deep learning model for deepfake detection.
• To extract spatial and temporal features from video frames.
• To extract acoustic features from speech signals.
• To implement four-class classification instead of simple binary detection.
• To deploy the model in a real-time environment.