Multimodal deepfake detection

Synthetic video is now good enough that a single frame will not give it away, and it is being used against elections, courts, journalism, and identity verification at banks.

The tell is not in any one modality. A generator good enough to fool the eye is usually not simultaneously good enough to keep the voice, the face, the expression, and the file's own metadata consistent with each other. The seams between the parts are where the evidence is.

The first generation established the pipeline end to end, extracting metadata and frames, detecting faces across sampled frames, pulling the audio track and analysing pitch, tempo, energy, and voiced segments, then scoring the consistency between the two and returning a verdict with a confidence figure, behind a drag and drop upload interface.

The second generation replaced the reasoning with a trained multimodal architecture over a public benchmark dataset, in four phases: feature extraction, unimodal training, fusion, and evaluation.

Around the model, the outputs a claim like this has to carry to be believed: modality breakdown charts, ablation studies establishing which modality is actually doing the work, emotion histograms, and full metric reporting.

Four modalities, fused twice.

Video frames contribute facial landmarks around the eyes, nose, and mouth. Audio contributes its spectral coefficients. Emotion is read separately from the face and from the voice, and the mismatch between the two readings is a signal in its own right: a neutral voice under an angry face is a seam. File metadata contributes the fourth.

Early fusion concatenates the low level features so the convolutional layers see them together rather than in isolation. Convolutional layers with pooling extract from there, and late fusion brings metadata back in through an attention mechanism before the fully connected layers and the final classification.

The two fusion points are the design: early fusion catches what only appears when modalities are seen jointly, and late fusion lets metadata weight the result without drowning in it.

Lead developer across both generations. Feature extraction, the model architecture and training, the fusion strategy, the evaluation and visualisation, and the interfaces.

The dataset is enormous and the hardware was not. Training ran on a single consumer graphics card, which made memory management a first class design concern rather than an afterthought, and drove the chunking and batching strategy throughout.

Ablation mattered more than accuracy. A multimodal system that scores well because one modality carries it is a unimodal system with extra steps, and only an ablation study tells you which one you have.