---
title: "Using 'Slop Forensics' to Determine Model Ancestry"
date: 2025-05-30
author: Drew Breunig
tags: ["ai", "slop", "synthetic data"]
url: https://www.dbreunig.com/2025/05/30/using-slop-forensics-to-determine-model-ancestry.html
---

Yesterday, after playing with some smaller models, I started to experiment with the idea of a flowchart for determining a model's ancestry with a few prompts. For example, could you ask it about [state-censored topics](https://petewarden.com/2025/05/08/why-the-chinese-government-taught-ai-to-lie/) and [about its development](https://community.openai.com/t/looks-like-deep-seek-r1-v3-was-distilled-from-gpt-4-3-5-can-anyone-confirm/1106952) and figure out what model was it trained by or from. Luckily I aborted that effort, because [Sam Paech](https://x.com/sam_paech), who maintains [EQ-Bench](https://x.com/sam_paech/status/1928208120108437728), has built an entire "[slop forensics](https://github.com/sam-paech/slop-forensics/tree/main)" pipeline.

[According to Sam](https://x.com/sam_paech/status/1928208120108437728), his tool "generats a 'slop profile' for each model...then use[s] a bioinformatics tool to infer lineage trees, based on the similarity of the slop profiles." In a nutshell, by generating and analyzing creative writing output from each model, fingerprints based on the frequent and/or unique phrases used can be constructed and compared. 

You can view these profiles by visiting [EQ-Bench](https://eqbench.com/creative_writing.html) and clicking the "i" next to the "slop" score. Here's what the new DeekSeek R1's profile looks like:

![DeepSeek's Slop Profile](/img/ds_slop_profile.png)

Interestingly, [Sam's slop forensics reveal DeepSeek has likely switched from OpenAI's models to Google's Gemini models for generating their synthetic data](https://x.com/sam_paech/status/1928187246689112197). As a result, DeekSeek now sounds more like the Google family of LLMs. A visualization produced by his tool shows the dramatic switch:

![](/img/ds_switch.jpg)

The tool also highlights how these models are converging, with each using (previously) unique names like, "Elara," when writing fantasy fiction. This is an understudyed impact of [our current reliance on synthetic data](https://www.dbreunig.com/2024/12/18/synthetic-data-the-growing-ai-perception-divide.html), especially for non-verifiable fields like creative writing. 

If we're sharing datasets or using the same models to generate new datasets, conversion is likely to occur. Which isn't ideal, if we [want for creativity and diversity in our LLMs](https://www.dbreunig.com/2025/04/18/the-wisdom-of-artificial-crowds.html).