Deepfake Detection · · 7 min read

The Lab-to-Production Accuracy Gap: Why 95% Doesn't Mean 95%

Why deepfake detection tools claiming 95% accuracy collapse on real insurance claims media. Compression, lighting, demographic bias, and the path to production.

A vendor tells you their deepfake detection tool achieves 95.3% accuracy. Impressive. But accuracy on what?

This single question — accuracy on what data, under what conditions — is the difference between a tool that works and one that gives you false confidence. In deepfake detection for insurance, the gap between lab benchmarks and real-world production performance is not a rounding error. It’s a chasm.

The Benchmark Problem

How Detection Models Are Evaluated

The academic and commercial deepfake detection community relies on standardized benchmark datasets to evaluate performance. The most commonly cited include:

FaceForensics++ (FF++). Developed by researchers at the Technical University of Munich, FF++ contains over 1,000 original video sequences with deepfakes generated by four methods (FaceSwap, Face2Face, Deepfakes, NeuralTextures). It’s the most widely used benchmark — and its characteristics define what “accuracy” means for most published results.

Celeb-DF. A dataset of celebrity face-swap videos with improved visual quality compared to earlier datasets.

DFDC (Deepfake Detection Challenge). Created by Facebook/Meta for their 2020 detection competition, featuring 100,000+ videos across diverse scenarios.

WildDeepfake. Collected from the internet to represent more realistic conditions, though still focused on face manipulation in video.

These datasets established the field. They also created a systematic bias: detection models optimized for benchmark performance are optimized for benchmark conditions.

What Benchmarks Look Like

The typical benchmark dataset shares these characteristics:

  • High resolution: 720p-1080p video, high-quality still images
  • Face-centric content: Almost exclusively facial manipulation (face swaps, re-enactment, expression transfer)
  • Controlled conditions: Relatively consistent lighting, clear subjects, limited background complexity
  • Known generation methods: Created using specific, documented deepfake tools
  • Clean capture: Original media captured with professional or high-quality consumer equipment, minimal compression

A model trained on this data learns to identify specific artifacts under specific conditions. When tested on similar data, it performs well. Published accuracy numbers reflect this performance.

What Insurance Claims Media Looks Like

Now compare benchmark conditions with what insurers actually receive:

Heavy compression. Claims photos are submitted through mobile apps that aggressively compress images to reduce upload times. A photo passes through camera processing, app compression, upload re-encoding, and claims system storage — each step degrading quality. By the time an image reaches analysis, it may have been compressed four or five times.

Low and variable resolution. Claims arrive from every device imaginable: current-generation iPhones, five-year-old Android phones, budget tablets, basic feature phones. Resolution, dynamic range, and sensor quality vary enormously.

Non-face content. Insurance deepfake detection isn’t about face swaps. Insurers need to detect fabricated vehicle damage, synthetic property destruction, forged documents, manipulated medical imaging, and altered timestamps. Benchmark datasets contain virtually none of this content.

Uncontrolled conditions. Claims photos are captured in rain, at night, in smoke-filled rooms, in harsh sunlight, in cluttered garages, through dirty windshields. The lighting conditions that academic datasets carefully control are precisely the conditions that real claims fail to meet.

Diverse manipulation methods. Fraudsters don’t limit themselves to the four or five generation methods in benchmark datasets. They use the latest consumer tools — Stable Diffusion variants, Midjourney, DALL-E, FLUX, plus commercial editing tools with AI-assisted features. The specific generation method changes every few months.

Why Accuracy Collapses

The performance gap isn’t mysterious — it has well-documented causes.

Compression Destroys Detection Signals

Many detection methods rely on subtle statistical patterns in pixel values or frequency-domain signatures. JPEG compression — the format virtually all claims photos use — systematically alters both.

When a genuine photo is heavily compressed, the compression process introduces artifacts that can look like manipulation signals to a detection model. When a manipulated photo is heavily compressed, the compression can mask the genuine manipulation signals. The result: more false positives on genuine photos, more false negatives on manipulated ones.

Research presented at IEEE Conference on Computer Vision and Pattern Recognition (CVPR) and similar venues has consistently demonstrated this effect. Models achieving 95%+ accuracy on uncompressed images can drop below 70% when the same images are compressed to quality levels typical of mobile claims submissions.

Domain Shift Breaks Generalisation

Machine learning models learn from their training data. A model trained exclusively on face-swap deepfakes in high-resolution video has learned features specific to that domain: the boundary artifacts of face insertion, the blending patterns at facial edges, the temporal inconsistencies of face tracking.

These features don’t transfer to insurance claims content. The model has never seen manipulated vehicle damage, fabricated property destruction, or forged documents. It literally doesn’t know what to look for.

This is the domain shift problem: performance on the training domain doesn’t predict performance on a different domain. It’s not a bug — it’s how machine learning fundamentally works.

New Generation Methods Evade Old Detectors

Each generation of AI image tools improves on the previous one. A detector trained to identify GAN artifacts (the telltale checkerboard patterns, the frequency spikes) may completely miss content from diffusion models (which have different artifact profiles). A detector trained on Stable Diffusion v1.5 output may miss content from SDXL or FLUX.

The deepfake generation landscape evolves faster than most detection models are updated. Unless detection models are retrained frequently with outputs from the latest generation tools, they develop blind spots for new methods.

Demographic and Content Bias

Benchmark datasets overrepresent certain demographics (typically Western, lighter-skinned faces) and content types. Detection models trained on these datasets may perform differently on:

  • Different skin tones and facial features
  • Different types of content (damage photos vs. faces)
  • Different cultural contexts (document formats, vehicle types, architectural styles)

For insurance — which operates globally and involves diverse claimant populations — this bias directly impacts detection equity and reliability.

Measuring What Matters

The Right Questions for Vendors

Don’t accept benchmark accuracy at face value. Ask:

“What is your accuracy on JPEG-compressed images at quality 70-85?” This is the compression range typical of claims submissions. If the vendor hasn’t tested at these levels, their headline accuracy number is misleading.

“What is your accuracy on non-face content?” Ask specifically about vehicle damage, property damage, documents, and medical imaging. If the model was only validated on facial deepfakes, it’s not an insurance solution.

“What is your false positive rate on genuine claims photos?” In insurance, false positives are costly — they delay legitimate claims, frustrate genuine claimants, and waste investigation resources. A 5% false positive rate on 100,000 claims means 5,000 legitimate claims needlessly flagged.

“When were your models last updated? What generation methods are covered?” If models haven’t been updated in the past six months, they may miss content from the latest generation tools.

“Can you run a proof of concept on our data?” The definitive test. Provide anonymised claims from your own portfolio — including known-genuine claims — and measure accuracy on your actual content at your actual compression levels.

Building Your Own Benchmark

If you’re serious about evaluating detection vendors, create an internal test set:

  1. Genuine claims sample: 500+ real claims photos from your portfolio, across all lines of business, representing the full range of devices, conditions, and compression levels
  2. Known-manipulated sample: Create manipulated versions of genuine claims photos using current generation tools (Stable Diffusion, Midjourney, DALL-E, plus manual editing tools). Match the compression and resolution of your genuine claims
  3. AI-generated sample: Generate entirely synthetic claims evidence — vehicle damage, property damage, documents — using current tools

Run each vendor’s tool against this test set. Measure:

  • True positive rate (correctly identified manipulation)
  • False positive rate (genuine photos incorrectly flagged)
  • True negative rate (genuine photos correctly passed)
  • False negative rate (manipulated photos that passed undetected)

This gives you accuracy numbers that actually predict production performance in your environment.

How deetech Approaches the Gap

We built deetech specifically to address the lab-to-production gap. Our approach differs from benchmark-optimized tools in several ways:

Insurance-native training data. Our models are trained on data that reflects real-world claims conditions: compressed, variable-resolution, diverse content types, captured under the messy conditions that actual claims involve. We don’t train on FaceForensics++ and hope it generalises.

Multi-layer detection. We don’t rely on a single detection signal that compression can destroy. Our architecture combines pixel-level forensics, frequency domain analysis, metadata verification, and semantic consistency checking. Each layer has different sensitivity to compression and different failure modes — the combination is robust where individual layers aren’t.

Content-type-specific models. Vehicle damage, property damage, documents, and medical imaging each have different characteristics and different manipulation profiles. We use specialized models for each content type rather than a single generic detector.

Continuous retraining. We update our models regularly to incorporate detection for new generation methods. Our retraining pipeline ingests output from the latest AI tools as they’re released, closing the detection gap before it becomes exploitable.

Validated on insurance data. We publish accuracy numbers on insurance-relevant data, not academic benchmarks. Our metrics reflect the performance our customers actually experience in production.

The Bottom Line

A 95% accuracy claim from a deepfake detection vendor is not a lie — it’s a measurement in conditions that don’t match yours. The question isn’t whether the vendor’s number is accurate. It’s whether it’s relevant.

For insurance, relevance means: accuracy on compressed, variable-quality, non-face content captured under real-world conditions. That’s the number that determines whether the tool actually helps detect fraud in your claims pipeline.

Ask for it. Test for it. Don’t buy without it.

deetech reports accuracy on insurance-relevant data — compressed, diverse, and captured in production conditions. We offer proof-of-concept evaluation on your anonymised claims data so you can verify performance before committing. Request a demo.

Sources cited in this article: