The biggest buyers will want to audit and influence post-training

Beneath the Anthropic and Department of War fracas, there is a legitimate & essential conversation to be had about how much control any organization has when deeply adopting an AI model they didn’t train.

These are probabilistic systems, with near infinite surface area to test, that are intentionally designed. Models are used to inform and make decisions, and they all have embedded perspectives.

AI is unlike other technology purchases because AI has embedded judgment.

I’m not sure what the answer is here, only that we need to have this discussion (calmly) and that anyone who tells you this isn’t a problem, that their model has an objective God-view-from-nowhere, is selling you something.

Let me be clear: I agree strongly with Anthropic’s usage red lines. I gladly choose Claude myself.

But this conversation is being framed badly around usage. Many are talking about Anthropic’s “terms of service” (notably, both Hegseth and Trump even capitalized the term in their tweets), but I think allowed usage terms are red herring. The issue is embedded judgment.

If I were in military procurement, I would certainly some big questions about what “soul documents” or “constitutions” (or similar) are embedded in any model being considered for embedding throughout the armed forces (and all the labs make design choices during post-training).

And clearly this is something Anthropic is already dealing with! This section, from the above blog post, suddenly becomes much more interesting:

This constitution is written for our mainline, general-access Claude models. We have some models built for specialized uses that don’t fully fit this constitution; as we continue to develop products for specialized use cases, we will continue to evaluate how to best ensure our models meet the core objectives outlined in this constitution.

We don’t know if post-training control helped blow up the deal (I tend to believe the issue was about allowed usage, based on the administration’s and Anthropic’s statements, coupled with OpenAI’s announced terms). But I think it’s a safe bet many militaries will insist on influencing and auditing the post-training for their purchased variants.

I wrote back in 2023 that I expect states and cultures to build their own models for related reasons; I wasn’t thinking about defense tech at the time but it certainly amplifies the issues.

One takeaway: this is a strong argument for why the AI race isn’t going to be winner-take-all. Everyone wants a champion to trust.