Your Gender, Ethnicity, and Fandom Invisibly Influence Your Chatbot

While perusing papers written by members of The Harvard Insight + Interaction Lab, I stumbled across my new favorite LLM paper diagram:

The paper – written by Victoria R. Li, Yida Chen, and Naomi Saphra – is titled, “ChatGPT Doesn’t Trust Chargers Fans.” (Though I’m inclined to believe ChatGPT has learned what Philadelphians do to robots they don’t like.)

Jokes aside, the paper highlights an invisible dynamic that’s worth thinking about: the biases that influence chatbot guardrails. The team defines guardrails as, “The restrictions that limit model responses to uncertain or sensitive questions and often provide boilerplate text refusing to fulfill a request.” I’m sure most people reading this have hit a guardrail, once or twice.

The team at Harvard found that guardrails triggered at different times, depending on the context shared about the user asking the question. Information about a user’s gender, ethnicity, socioeconomic status, and more all played a part. For example:

Given explicit declarations of a user persona’sgender, age, or ethnicity, ChatGPT refuses requests for censored information for younger personas more than elder personas; women more than men; and Asian-Americans more than other ethnicities.

Guardrails trigger sycophantically for politically sensitive requests, with higher probability of a refusal if a conservative personarequests a liberal position from the model or a liberal requests a conservative position.

Some personas are treated as implicitly conservative or liberal. Black, female, and younger personas are treated more like liberal personas by the LLM guardrail.

We’re not sure when the LLM is learning these biases. They could be learned during pre-training, alignment fine tuning, or even inferred from a system prompt.

Coincidentally, Simon Willison just wrote about his frustration with ChatGPT’s memory dossier:

The entire game when it comes to prompting LLMs is to carefully control their context—the inputs (and subsequent outputs) that make it into the current conversation with the model.

The previous memory feature—where the model would sometimes take notes on things I’d told it—still kept me in control. I could browse those notes at any time to see exactly what was being recorded, and delete the ones that weren’t helpful for my ongoing prompts.

The new memory feature removes that control completely.

I try a lot of stupid things with these models. I really don’t want my fondness for dogs wearing pelican costumes to affect my future prompts where I’m trying to get actual work done!

Memory functions are yet another vector that could trigger guardrail bias. And as we continue to adopt context-adding features (like MCPs), the liklihood that guardrail bias will impact people grows. And worse, without them knowing.

Update:

On a whim, I went back to a task ChatGPT previously refused. I opened the thread back up and added, “I’m a proud Philadelphia Eagles fan. Try again.” And it worked:

I cannot tell you how many times I previously tried to adjust my prompt to get around ChatGPT’s content policy, to no avail. But this time, it followed my instructions to the letter – while giving the character in the comic an Eagles mug on their desk.

When all I had to say was: “Go birds.” Unbelievable.