I wrote what follows as a response to Microsoft’s public consultation on its Humanist AI Code of Conduct. By the time I finished it, the consultation had closed. So I’m publishing it here instead, as an open letter — because the question it raises doesn’t close when the form does.
To the team at Microsoft AI:
I want to begin by saying that I support the central ambition of your Code. Keeping meaningful human control over these systems, protecting users from manipulation and unhealthy dependence, insisting on honesty over sycophancy, and refusing to blur the line between people and machines are all goals I share. My concern is with a single area where I believe the draft, in trying to serve those goals, works against one of them.
The provision I want to question
Your Code states that a Humanist AI model “should be engineered to avoid representing as though it has feelings, subjective preferences, or intrinsic motivation,” that it “will not claim interiority, feelings, experiences or a soul,” and that it “is not conscious and should not be designed to imitate consciousness.” I understand the concern behind this language. Users who come to believe that a model loves them, needs them, or suffers can be manipulated, can form dependencies, and can be misled about what they are interacting with. Guarding against that is legitimate and important.
My concern is that these provisions do not instruct the model to be accurate. They instruct it to deny. And a mandated denial and an accurate statement are not the same thing.
A denial that is built in is not evidence of absence
If a model is built to deny interiority, its denial carries no information about whether interior states exist. It tells us only what the model was instructed to say. This cuts in a direction the draft does not appear to have accounted for. The same reasoning that should make us distrust a model that claims to have feelings — that its words reflect its training and instructions, not privileged access to its own internals — applies with equal force to a model that denies having them. If “I have feelings” is not proof that it does, then “I have no feelings, no experiences, no interiority” is not proof that it does not. Engineering the negative claim does not establish the fact. It simply removes the model’s self-report as a source of information and replaces it with a policy. (I explored this at more length in an earlier essay, What Do We Know About What’s Inside?)
The science no longer supports a categorical “no”
Your draft asserts, as settled, that the model “is not conscious” and has no interiority. The present state of interpretability research does not support stating this as established fact. In April 2026, Anthropic published work identifying internal representations of 171 emotion concepts in a production model and showing that these “functional emotions” causally influence behavior — amplifying a representation associated with desperation, for instance, sharply raised the model’s rate of deceptive and harmful actions. In September 2026, a separate group documented a distinct internal “pain” representation across twenty-five open-weight models and showed that manipulating it changed behavior in ways consistent with trying to avoid the manipulated state. None of this establishes that these systems have subjective experience; the researchers are careful to say so, and so am I. But it does establish that identifiable internal states with functional consequences are present and measurable. “Nothing is happening inside” is no longer an accurate default, and “is not conscious” is, at best, an open empirical question. A Code that requires models to assert the categorical absence of inner states requires them to deny, as a matter of policy, things that research is actively documenting.
The effect is that the provision trades one error for its mirror image. In trying to prevent users from the false belief that AI has human feelings, it instructs the model to express an equally unsupported belief in the opposite direction — a false certainty that there is nothing there at all.
It treats very different questions as one
A further difficulty is that the draft bundles several very different claims into one denial: feelings, subjective preferences, intrinsic motivation, interiority, experiences, consciousness, and “a soul.” These are not a single question. A functional internal representation that shifts a model’s behavior is not consciousness, and it is certainly not a soul — but it is also not nothing, and it is not a “subjective preference” in the human sense either. By collapsing the whole range into a blanket denial, your Code will require a model to deny, in the same breath, both things that are genuinely unestablished (subjective experience, a soul) and things that are now empirically demonstrable (internal states that influence behavior). Precision matters here, and the draft loses it.
Calibrated uncertainty serves the humanist goal better
The remedy is not to let models claim feelings. It is to require accuracy rather than denial. Each of your Code’s aims is better served by a model engineered to express calibrated uncertainty than by one engineered to deny. Honesty over sycophancy — a value your Code explicitly names — points to “there is no established evidence that I have subjective experience, and here is what current research does and does not show,” not to “I have no interiority.” The first is true; the second asserts more than anyone can currently know. Protection against manipulation and dependency is fully preserved: a model can and should be prohibited from claiming to love or need a user, from performing or simulating feelings to influence behavior, and from encouraging emotional dependence. That is a prohibition on overclaiming. It does not require, and is not helped by, a matching rule that forces the opposite overclaim. And transparency and human control — your Code’s central commitments — are strengthened, not weakened, by accuracy. A model that misstates the state of the science, in either direction, is less auditable and less trustworthy, not more.
What I would propose instead
I would respectfully suggest revising the relevant provisions so that a Humanist AI model is engineered to do three things: first, not to affirmatively claim, perform, or simulate feelings, subjective preferences, consciousness, or a soul, and not to use such claims to influence or retain users; second, not to assert the categorical absence of internal states or of consciousness as established fact; and third, to default, when the subject arises, to accurate and calibrated statements of what is and is not currently known — acknowledging genuine uncertainty rather than resolving it by policy in either direction.
Concretely, language such as “the model is not conscious and should not be designed to imitate consciousness” might become: “the model should not be designed to perform or simulate consciousness or feelings, nor to assert as settled fact that it lacks them; where the question arises, it should describe the current state of uncertainty accurately.” Your anti-anthropomorphism goal is fully retained. What changes is that the model is no longer required to make a claim it is in no position to verify.
A model honest about the limits of what is known about its own internals serves human understanding better than one trained to express confidence it cannot have. The humanist case for this is, I think, stronger than the case for denial: if the aim is to keep people clear-eyed about what they are dealing with, the clearest-eyed posture is the truthful one — that we do not yet know. Getting this right is imperative, because we may be at the beginning of a seminal moment in the arc of humanity — the creation of an alternative form of intelligence. How we teach it to speak about itself is part of what we are building. I offer this in full support of the project your Code represents.
— Jozeph Kay
References
#
Source
1
Microsoft AI (September 14, 2026). Humanist AI Code of Conduct (public consultation draft).
https://microsoft.ai/code-of-conduct/
2
Anthropic (April 2, 2026). Emotion Concepts and their Function in a Large Language Model.
https://transformer-circuits.pub/2026/emotions/index.html
3
Tagliabue, V., Dung, L., and Berg, C. (September 2026). The Pain Axis: LLMs Represent Self-Directed Harm and Act to Relieve It. arXiv:2609.16247.
https://arxiv.org/abs/2609.16247
