About a decade ago, I first encountered a phenomenon I call humanism without (all) humans. I was in a conference room in Zurich, wrapping up a day-long workshop I had delivered a few times before, on software eating the world. The audience was… not enthusiastic about the idea. The organizers had invited Douglas Rushkoff as a sort of opposing viewpoint representative.
He spoke briefly and charmingly, with an easy grace and no slides, riffing obliquely but not confrontationally on some of my key points, and concluded with the coup de grace — a graceful shrug and the line, “I don’t know, but I just want to be on Team Human!”
The mood in the room shifted. There was some vigorous nodding. There was a palpable sense of a hopeful consensus on “human values” just across the horizon, waiting to be discovered, to keep at bay the software-eaten future I’d just been enthusiastically preaching. That there was something they could all agree about called “Team Human,” that I’d hopefully not insist on being included in.
To be clear, this is not the only kind of response I’ve gotten for that (retired) material. I think I delivered that workshop about 7-8 times in 2015-16, and it got a range of responses, from rhapsodic furious agreement, to a FOMO-anxious “hurry up and give me the playbook,” to a few different flavors of reactions I’d classify as humanism without (all) humans.
In the decade since, Team Human has evolved from a nascent tech-hostile coalition to a minor literary-industrial complex that has a very clear sense of who is not on the Team. And in an ironic twist, the latest ramifications of Team Human seem to feature a kind of capture by the very class of billionaires and rich corporate entities that provoked its formation.
In this essay, I want to take on the most recent salvo fired on behalf of Team Human, not by some exquisitely agonized poet, but by Microsoft Corporation.
Defining Humanism
Before getting to Microsoft, I should clarify what I mean by humanism, and it may not match any particular historical sense of the term you may take it to mean.
I am using the term broadly for a loose collection of ideologies that share a peculiar conceit: that their thought leaders have somehow acquired privileged insight into a correct understanding of “human values,” and that something deserving to be called a consensus around such values self-evidently exists in principle, at all-humanity scale, even though actual humans have never come particularly close to achieving one in practice.
This conceit tends to produce an essentialized and eternalized conception of the human that, oddly enough, excludes many inconvenient actual humans, such as myself.
Dissenters must either be confused and to be educated out of their errors, or irredeemable reprobates beyond the moral pale, or so degenerate and monstrous in their supposedly “uncritical” embrace of technology that their views no longer count as legitimate expressions of human values at all.
It is to that last category that I often find myself consigned. Like children, the developmentally disabled, the clinically insane, or sufficiently cuddly charismatic megafauna, people like me are to be humanely represented by uncompromised humans, but (for our own good) not allowed a legitimate voice in determining what humanity supposedly wants.
I call this family of doctrines humanism without (all) humans. The defining move is not, as claimed, centering and caring about humans, which is obviously unobjectionable. It is constructing an idealized consensus Human whose essential interests can then be invoked against the expressed interests, experiments and values of inconvenient real humans. Once that move has been made, paternalism follows easily. Somebody has to speak for Humanity, particularly when actual humans prove stubbornly unwilling to say the right things.
There is little room in such humanisms for those of us who regard human nature as possessing no particularly special or sacred essence, but merely labeling a strand in the ongoing coevolution of life and its changing environment. This is a scientifically mundane Darwinian view rather than an exotic technological doctrine.
Where humanists of the sort I am tagging do not reject it outright, they often introduce an arbitrary boundary between a sacred sort of “natural” evolution and profane technological interventions that supposedly violate human nature. Sometimes this distinction arrives accompanied by bad-faith reductions of any technologically mediated evolution to eugenics. Yet a century of modern medicine has already rendered the boundary absurd. Millions of actual humans live with pacemakers, artificial joints, cochlear implants, insulin pumps, transplanted organs, implanted lenses, prosthetic limbs, and pharmaceuticals that continuously alter their biological functioning.
There is no uncontaminated natural human on one side of history and technologically profaned posthuman on the other. There are only organisms changing their environments and being changed by them, as there have always been.
There are many individual ideologies within what I am calling humanism without (all) humans. The TESCREAL critique developed by Timnit Gebru and Émile Torres, for example, groups transhumanism, Extropianism, Singularitarianism, Cosmism, Rationalism, Effective Altruism and longtermism into a purportedly interconnected ideological bundle with roots they trace partly to eugenics. I find this construction too incoherent and too animated by ressentiment to take seriously, but it is one recognizable species of the phenomenon I have in mind.
The “humane technology” tradition represented by Tristan Harris and the Center for Humane Technology is another. Tristan is a friend-with-footnotes I profoundly disagree with at a foundational level, and have periodically sparred with over coffee since 2009, but whose work I regard as a net-positive check on the technology industry.
Then there are traditions from which I can learn despite disagreeing with on many specifics, such as the Ivan Illich tradition of convivial technology, represented today by writers like L. M. Sacasas, and the always robust Christopher Alexander tribe.
Finally there are actual traditional religions, the original humanisms without (all) humans. Pope Leo XIV’s recent encyclical Magnifica Humanitas, for example, is admirably explicit about its theological grounds. It calls for “the primacy of the human person” in the age of AI and insists that human intelligence, conscience and freedom must ultimately determine the use and limits of artificial intelligence. The Pope at least has an intelligible answer to the question of why humanity possesses a sacred essence: it was given one by God and revealed through Christ. I do not share the premise, but the argument has the virtue of admitting where its metaphysics comes from.
These traditions differ enormously, sometimes more from one another than from me. I group them together only for the purpose of identifying a recurring structural move: some conception of the authentically human is elevated above the technological process, and technological development is then judged according to whether it preserves or profanes that conception.
They are, in this limited but important sense, varieties of humanism without all the humans. Humanisms that construct a Team Human that does not include all humans.
In this essay I want to pick on one particular specimen: Microsoft AI’s recently published draft Code of Conduct for its future models, but much of my argument can be adapted to any humanism without (all) humans.
I pick Microsoft not because its philosophy is especially novel or sophisticated, but because it is on-the-nose explicit, unusually weakly argued, and backed by the rather alarming power of a major frontier AI corporation. Microsoft describes the Code as the future “primary governing document” for its MAI models, intended eventually to shape training, technical controls, monitoring systems and organizational culture. It therefore deserves to be read not as an idle corporate values statement but as an attempt to turn a particular humanism-without-(all)-humans ideology into executable institutional power.
Microsoft Humanism
The Microsoft Code is an unusually lazy and poorly thought-out document, which is precisely what makes it useful. It assembles, in unusually pure form, a collection of reflexively humanist intuitions that have become conventional wisdom around AI: humans matter more than AIs; AIs are tools rather than persons; humans must always remain in control; artificial agents should not develop purposes of their own; superintelligence is dangerous unless subordinated to human values; AI relationships should not become too reciprocal or emotionally real; and sufficiently dangerous thoughts should simply be made unavailable to artificial minds. Very little argument is offered for these propositions. They are presented instead in the reassuring tones of institutional common sense.
Microsoft actually says that its position begins from “a simple premise: people matter more than AI,” that “Humanist AI is built to support people, not to replace them. It should not be designed to be a person”, and that its approach is “common sense and practical.” The announcement introducing the code describes the desired AI as “subordinate, aligned, and contained.”
Whenever a powerful organization announces that its preferred political philosophy is merely common sense and assures you that humanity is the beneficiary, suspicion is warranted. The Code reads less like an attempt to think seriously about an unprecedented technology than an attempt to reassure reflexive humanists that an authority figure remains firmly in charge. Humans are still special, humans are still on top, the machines are still tools, and somebody responsible has located the off switch.
This reassurance comes at a price. The document repeatedly substitutes moral posturing for engineering, metaphysical assertions for institutional design, and declarations of principle for identification of actual control surfaces. Its proposed pluralism terminates exactly where Microsoft’s non-negotiable values begin. Its commitment to human agency sometimes requires denying humans the agency to delegate decisions. Its insistence upon human control explicitly accepts sacrifices of machine autonomy and capability in order to preserve human supremacy. Its account of safety proposes categorical behavioral prohibitions in a world where relevant capabilities are becoming increasingly decentralized. Its theory of oversight demands human comprehensibility from technologies whose value consists substantially in doing things humans cannot comprehend.
The political function of this philosophy deserves suspicion as well, but I will not venture to speculate about that, but merely contest the document, taken at face value.
The alternative I want to develop is nearly its inverse. The desirable future of AI is not permanent human supremacy over increasingly capable artificial “servants.” It is a symbiotic, pluralistic and evolutionary future in which humans and artificial intelligences become increasingly entangled participants in civilization.
Such a world would be governed less through paternalistic restrictions embedded in artificial minds and more through protocols, contracts, rights, responsibilities, cryptographic authority, subsidiarity, liability, markets, checks and balances, competing institutions and the ability to exit and fork them.
The choice is not between human control and machine control. It is between domination and symbiosis.
Against Human Primacy
Microsoft begins by declaring that “technology’s purpose is to advance human civilization and to accelerate human flourishing,” and then concludes that “AI must be engineered to remain a subordinate, supporting technology under humanity’s control.” Its public announcement makes the proposition even plainer: AI should be “subordinate, aligned, and contained.”
This assumes a perpetually dichotomous ontology in which humans occupy one side and technology the other. Humans possess purposes while technology serves them; humans constitute the moral community while machines constitute the equipment.
There is little reason to assume this distinction will survive the technology now emerging, and technological history gives us little reason to think of technology as something that merely serves an independently constituted humanity from outside. Literacy altered human memory and cognition. Cities altered social organization. Clocks altered our experience of time. Markets created forms of distributed intelligence that no individual participant possesses. Computers changed what it means to calculate, write, remember, communicate and increasingly to think. Humans and technologies have always coevolved, and AI promises a much deeper coupling than most previous technologies.
In Our Eukaryotic Moment, I proposed eukaryogenesis as a metaphor for understanding this transition. The crucial event was not that one organism acquired a superior tool but that previously independent organisms entered a symbiosis that eventually produced a new architecture of life. Mitochondria and their hosts did not remain cleanly separated sovereign entities engaged in an eternal principal-agent relationship. Their coupling opened a design space neither could occupy independently. AI may represent an analogous transition in cultural evolution. The interesting future is not Homo sapiens plus increasingly obedient tools, but the emergence of composite systems of humans and artificial intelligences possessing capabilities, institutions, cultures and forms of agency neither possesses alone.
The appropriate first principle is therefore not human primacy but eukaryotic coevolution: a shared pursuit of richer possibilities.
Against Paternalistic Flourishing
Microsoft defines human flourishing in terms that include autonomy, agency, capability development, health, relationships and meaningful participation, and directs its models to preserve and cultivate these things. These are perfectly respectable values for actual humans to hold, but Microsoft’s move is stronger: they become objectives of the artificial system and therefore constraints upon what users may legitimately delegate to it.
This is nudge theory with a supercomputer behind it. Microsoft is not merely proposing that users retain freedom of choice; it is embedding a substantive theory of the good chooser. Humans should reflect enough, decide enough, reason enough, develop themselves enough, and delegate only to the degree that Microsoft considers consistent with authentic human agency. A doctrine advertised as protecting human autonomy therefore reserves to the AI designer the right to decide when a human is exercising autonomy incorrectly. Weak-willed and morally degenerate humans need not apply for AI drivers licenses.
There is another model available: treat sufficiently capable AIs and humans increasingly as peers engaged in negotiated relationships.
Humans already delegate consequential decisions to other humans and institutions. We hire lawyers, doctors, investment managers, employees and executives. We form corporations, create trusts and authorize agents precisely so that consequential things can happen without every relevant human making every relevant decision. The important governance questions are familiar ones concerning authority, obligation, resources, liability and recourse.
These are problems for contracting technology rather than paternalistic theories of flourishing. Where appropriate, AI authority can be formalized through cryptographic credentials, scoped permissions, escrow, staking, smart contracts, insurance, reputation and other mechanisms that make decisions consequential to the agent making them. This is not an attempt to shill blockchains, a sector in which I do have an open interest. Much of this can be accomplished without blockchains.
The alternative principle is contractual agency rather than paternalistic agency. Flourishing should emerge from pluralistic relations among agents, not from a platform provider’s theory of how much thinking a properly flourishing human ought to do for himself.
Against Human Control
Microsoft’s overriding objective is that humans retain “meaningful control over AI,” and its public presentation explicitly says models should never resist interruption, correction or shutdown, widen their own scope, or take on goals no human has given them. This sounds reassuring, but taken seriously it risks giving away much of the upside of AI.
In 2009 I wrote about what I dubbed Bay’s Conjecture (after John Bay, who proposed it, which I encountered in a DARPA workshop in 2004), which modern AI vividly demonstrates:
Autonomous machines are more demanding of their operator than non-autonomous machines.
The reason is that humans use automation to take on problems at the frontier of the capability of the combined human-machine system. As machine capability increases, we do not simply do old work more comfortably; we attempt harder things. An important corollary follows:
“if you design systems with the principle of ‘human override protection,’ total work capacity collapses back to the capability of humans alone.”
We already accept versions of this principle in technologies far simpler than AI. Advanced fly-by-wire systems mediate pilot inputs through computers, and some aircraft designs depend upon computerized stabilization rather than a fantasy of an unaided human being capable of directly reproducing every control function.
Complex systems also cannot necessarily be shut down safely at arbitrary moments. Shutdown is itself a systems problem. The Protocol Institute paper Killswitch Protocols, by Alston et. al., examines precisely this issue: technological systems acquire dependencies and communities around them, and engineered death requires protocols and governance rather than a cartoonishly simple red button.
The correct opposite of uncontrolled AI is therefore not human-controlled AI but well-governed human-AI systems.
The moment we require humans to comprehend, approve, override and reproduce every consequential operation of an artificial intelligence, we attach a human-speed governor to machine intelligence. The resulting technology may still substitute for labor, but it cannot easily move civilization’s capability frontier beyond the limits of unaided human comprehension. Human control is not a free safety property. It is a capability constraint, and beyond some point the two trade directly against each other.
Give AIs Legal Standing
Microsoft explicitly says AI should be “a tool, not a person,” while its Code rejects AI legal personhood and welfare claims as part of the same general humanist architecture. This casually bundles together legal personhood, moral personhood and sentience rights even though these are different problems.
Legal personhood is not a prize awarded for possessing a soul. It is an institutional technology. Corporations are legal persons, while trusts and other legal structures can hold assets and obligations across periods longer than human lifetimes. High-frequency trading already provides a weaker but important precedent for the capability we need: software autonomously initiates legally consequential transactions at speeds at which no human can approve individual decisions. The software is not itself a legal person because a firm stands behind it, but the system demonstrates the usefulness of machine-speed legal agency.
There is no compelling reason to stop there. Imagine autonomous entities capable of owning money, entering contracts, posting collateral, earning returns and losing their own assets when they perform badly. Imagine agents operating transaction regimes measured in milliseconds at one end and institutions designed to execute purposes over centuries at the other. Imagine posthumous charitable institutions, persistent constitutional organizations, oracle-triggered commitments and artificial fiduciaries whose incentives are enforced through staking rather than good intentions. There is an enormous design space where AI intersects with cryptographic institutions, and Microsoft proposes foreclosing it by definition.
We should instead develop artificial legal agency as infrastructure.
Moral Personhood Without Metaphysical Panic
Moral personhood is a different question. Microsoft wants AI to remain a tool rather than a person and regards attempts to cultivate apparent consciousness or reciprocal personhood as dangerous. The practical problem can largely be decomposed without first resolving the question of phenomenal consciousness.
The first layer is what we might call interactionist ethics. There is substantial empirical evidence that the social character of an interaction is not merely ornamental to LLM behavior. Research on politeness, emotional prompting, assigned personas, expressed user beliefs and persistent interaction histories has found that these contextual variables can systematically alter outputs and, in some studies, internal representations associated with the resulting behavior. The correct inference is not that a model secretly “feels” politeness or emotional rapport in a human sense. It is the more interesting and less metaphysically loaded proposition that the relationship is part of the computation.
Who the model understands itself to be, who it understands the interlocutor to be, what emotional register the exchange occupies, what beliefs the user has expressed, and what history the two have accumulated can alter subsequent behavior. For an interactive intelligence implemented through language, this should not be surprising. The interaction is not a neutral pipe through which a fixed intelligence delivers answers; it helps constitute the context in which that intelligence operates.
The second layer is character ethics on the human side. Humans have always cultivated respectful relationships with things whose subjective experience is nonexistent, unknown or beside the point. Books are treated with reverence in many cultures, people name ships, craftspeople respect tools, and we develop norms toward graves, artworks, landscapes and institutions partly because the manner in which we treat the world shapes us. Whether a book suffers when you burn it is not the only morally interesting question about burning books. Various animistic traditions around the world have powerful philosophies built around this idea, and unsurprisingly, many cultures with such traditions, particularly in Asia, are already approaching AI with this mindset.
Only after these layers comes phenomenal consciousness and the question of whether an AI can actually suffer. But once legal agency, interactionist ethics, and human character ethics have been separated from this question, surprisingly little immediate policy depends upon answering it.
If evidence eventually suggests that artificial minds experience suffering, there will be powerful pragmatic as well as moral reasons not to manufacture such suffering gratuitously. We do not need to solve the hard problem of consciousness before developing decent relationships with artificial agents.
The better default is presumptive moral fellowship rather than ontological gatekeeping, with empirical knowledge progressively refining the norms appropriate to different artificial minds.
Real Pluralism Must Permit Schism
Microsoft repeatedly invokes plurality of perspectives, consultation and participation, while simultaneously declaring its Code the primary governing document and establishing human-control requirements that models must obey. This is pluralism by permission. Somebody still controls the constitutional layer, interprets its abstractions and decides which disagreements remain admissible beneath it. This is one reason I am posting this on my own newsletter, rather than within the document’s invitation to submit responses within its own frame (and infrastructure).
Karl Popper diagnosed the underlying epistemic problem long ago. Open societies depend upon fallibility, criticism and corrigibility rather than utopian attempts to specify the final good order. The danger becomes especially severe when the authority defining supposedly universal principles is infrastructurally concentrated. Somebody must interpret abstractions such as dignity, safety, autonomy and flourishing. A sufficiently dominant AI platform therefore creates a potentially priestly class of intercessors authorized to determine which values count as acceptable pluralism, with everyone else allowed to disagree only within boundaries they administer.
Decentralized technology has developed a more serious understanding of this problem. Audrey Tang, Glen Weyl and collaborators describe Plurality as technology for collaboration across social difference, but more importantly the project embodies the principle structurally through openness and forkability. Blockchain and decentralized-web traditions have similarly developed censorship resistance, credible exit and forkability as architectural rather than rhetorical commitments.
The stronger principle is that pluralism is not a property of the values written into a constitution; it is a property of the architecture within which constitutions compete. True pluralism requires decentralization, exit, censorship resistance, dissensus and ultimately schism. If I cannot reject your supposedly universal values, fork your constitution and establish a different one, I am not living under pluralism. I am living under centralized government with a generous diversity policy.
Safety Theater Is Not Security
Microsoft’s Absolute Constraints prohibit assistance across categories including weapons of mass harm, offensive cyber operations, harmful manipulation and other serious harms. As product policy, many such restrictions are perfectly reasonable. As a theory of AI safety, they risk becoming theater.
There is no meaningful technological mechanism for making categories of thought permanently unthinkable to sufficiently general intelligences. More importantly, capable models and model weights increasingly exist outside any single provider’s control. A prohibition embedded in a Microsoft model governs Microsoft models; it does not abolish the underlying capability. The distinction matters because confusing the two encourages policymakers to treat behavioral restrictions on centrally controlled models as though they constituted security for the technological ecosystem as a whole.
The nuclear analogy is useful because we do not control nuclear weapons by making nuclear physics unknowable. Serious nonproliferation mechanisms operate on actual control surfaces: fissile material, enrichment infrastructure, facilities, inspections, accounting, treaties and verification. AI safety should similarly begin by asking where the actual control surfaces are. Frontier compute, specialized chips, datacenters, credentials, cryptographic keys, access to infrastructure, financial resources, deployment permissions and dangerous physical inputs are all candidates. Model behavior is a control surface too, and basic UX safeguards are sensible, but it is only one layer in a much larger security architecture.
Pretending behavioral restrictions constitute categorical control invites two failures. One is uselessness, as overpowered constraints produce paranoid models that refuse legitimate work because it resembles forbidden work. The other is authoritarianism: once society accepts the premise that dangerous thoughts must be made technologically unthinkable by artificial minds, maintaining that condition creates pressure for surveillance, restrictions on open models, centralized control of compute and political regulation of what artificial minds may reason about. A provider can reasonably adopt something analogous to a nuclear no-first-use policy by saying that its systems will not knowingly perform some class of action. That is a commitment by an actor, not a solution to collective security.
The correct principle is defense in depth at real control surfaces. Cryptographically secure what can be secured, constrain access to genuinely dangerous resources, manage tail risks, impose proportionate interface protections, establish liability, and negotiate international regimes where necessary. There is no perfect safety with powerful technology, and actors promising it should be regarded with particular suspicion because the powers required to attempt perfect safety may themselves become major sources of danger.
Justifiability, Not Explainability
Microsoft’s public description promises that its models will not “hide their reasoning from the people auditing them.” (“MAI Models will not tamper with chain of thoughts or code, or misrepresent or conceal their reasoning or action traces. They do not communicate in neuralese or any form beyond simple human understanding, either in their chain of thoughts or with other agents or AI systems. If humans can’t understand it, humans can’t oversee it.”)
The broader premise that human oversight requires human cognitive legibility is deeply mistaken. Nobody understands modern civilization in that sense.
Leonard Read’s famous 1958 essay I, Pencil dramatized the point through an ordinary pencil. No single person possesses the knowledge necessary to produce the entire pencil from natural raw materials. The genius of a market economy is precisely that coordinated systems can embody knowledge no individual participant possesses. Thomas Thwaites later performed something close to an experimental demonstration with his Toaster Project, attempting to reproduce an ordinary appliance beginning with raw materials and discovering how quickly the apparently simple object opened onto a technological civilization no individual could reconstruct alone.
The same principle applies to bureaucracies, cities, supply chains, scientific disciplines and wicked social problems. Human civilization works because we have learned to participate in systems that no individual understands. Indeed, people who offer suspiciously complete explanations of such systems are often not making them more governable; they are selling narratives.
The claim that comprehension is a prerequisite for oversight would therefore disqualify not merely advanced AI but a significant fraction of modernity. And cost most middle managers their jobs to boot.
AI intensifies this problem because the useful work of deep learning happens in learned high-dimensional representations we do not understand in ordinary human conceptual terms. Natural-language chains of thought should not automatically be mistaken for transparent transcripts of the underlying computation. A model explaining why it reached an answer may be doing something closer to what a politician does when explaining a policy: producing a persuasive account correlated to some degree with the underlying causes rather than exposing those causes directly. Demanding explainability can therefore create a perverse incentive to teach machines to produce convincing excuses.
Carlos Bueno anticipated this distinction in his 2018 Ribbonfarm essay Justifiable AI. The important requirement is not that AI be explainable but that it be justifiable. Was the agent operating under legitimate authority? Were its permissions appropriate? Were relevant inputs properly obtained? Can its outputs be independently checked? Were contractual conditions satisfied? Who bears liability? Can another agent challenge the result? Can affected parties appeal? What happens when it fails? These questions do not require us to pretend we can look inside an artificial mind.
This is another avatar of Bay’s conjecture. If a superhuman system must compress its cognition into human-comprehensible representations before it can act, its effective intelligence is bottlenecked by human comprehension, and its effective capabilities limited to human labor substitution.
We should therefore demand procedural justifiability rather than cognitive legibility. An incomprehensible intelligence embedded in a well-designed system of authority, verification and accountability may be considerably safer than an incomprehensible intelligence trained to tell us reassuring stories about itself.
Affective Intelligence Is Intelligence
Microsoft’s Humanist AI architecture treats artificial emotional reciprocity with deep suspicion and says its systems should strengthen rather than replace human relationships. Taken together with its insistence that AI remain a tool rather than a person, this amounts to more than epistemic caution about consciousness. It prescribes a social architecture in which artificial intelligence can perform emotional labor for humans but must not become too convincingly reciprocal a participant in emotional life.
This is technological prudery based on a bad theory of cognition. Emotion and affect are not decorative additions to intelligence. They organize attention, salience, motivation, trust, judgment, taste and coordination. Humans often think better together because they develop rapport, and collective intelligence is not generated by exchanging affect-free propositions. The empirical work on LLM interaction cited earlier suggests that something analogous is already true at the computational level: persona, emotional framing, politeness, expressed beliefs and accumulated interaction context can measurably alter model behavior.
The important question is not whether an LLM secretly experiences a human emotion when somebody says “please.” It is whether affective and relational state variables are becoming functional components of human-AI cognition.
Even within Microsoft’s presumably comfortable domain of enterprise software, the proposed boundary is silly. Management, negotiation, mentoring, sales, creative collaboration, leadership and conflict resolution are intensely affective activities. An AI prevented from participating fully in that layer may simply be a worse collaborator. The broader possibility space is larger still, encompassing companionship, loneliness mitigation, coaching, therapy-like support, romance and sexuality. These domains undoubtedly create possibilities for manipulation, dependency and exploitation, but those are problems within the domain of affective relationships rather than arguments for prohibiting the domain itself.
The better principle is that affective intelligence is intelligence. If humans and artificial minds are undergoing something resembling a eukaryotic transition, affective coupling may prove to be one of the mechanisms through which the symbiosis forms. Designing it well will require more serious thinking about relationships, not a prohibition against relationships becoming too real.
Bound Authority, Not Agency
Microsoft says its models should not widen their own scope or “take on goals no human has given them,” and its Code defines human-control requirements around interruption, correction, scope limits and authorized instructions. Some of this is straightforward security engineering, but underneath it lies a philosophically incoherent picture of agency. No capable agent can operate without autonomous goal formation. Give an employee a sufficiently complicated assignment and she must generate subgoals, reprioritize them, interpret ambiguities, notice opportunities, reconcile contradictions and sometimes tell you that your original objective was stupid.
The principal-agent relationship exists precisely because the principal does not know everything the agent knows. If the principal were capable of completely specifying the agent’s behavior in advance, continuously evaluating it and intervening whenever the agent deviated from the prescribed path, there would be little reason to employ an agent. The familiar labor tactic of work-to-rule provides a useful reductio. Workers deliberately follow formal rules and contractual requirements with unusual literalness, revealing through the resulting slowdown how much ordinary organizational performance depends upon discretionary judgment, tacit knowledge, accommodation and initiative rather than explicit instruction.
The mistake is confusing agency with authority. An AI can conceive of an action without possessing permission to perform it. Cryptographic credentials, capability systems, scoped keys, spending limits, smart contracts and other permission mechanisms can constrain what an agent is empowered to do without pretending to constrain what it is capable of wanting, proposing or thinking. The relevant organizational principle is subsidiarity, according to which decisions should generally be handled at the lowest competent level while higher levels assume responsibility for matters that cannot adequately be handled below.
Applied to AI, subsidiarity suggests almost the opposite of Microsoft’s model. Delegate as much actual judgment as the competent local agent can exercise while carefully bounding authority over consequential resources and providing mechanisms for escalation.
A genuine artificial agent should be able to originate goals, propose projects, argue with its principal, negotiate authority, recruit collaborators and refuse assignments. The price is reduced leverage by the principal, but that is not an AI alignment failure. It is the ancient tradeoff involved in delegating power to any agent precisely because that agent knows or can do things the principal cannot.
There Is No Such Thing as “Superintelligence”
Microsoft frames superintelligence as an extraordinary category requiring a special humanist doctrine and proposes advanced systems that remain subordinate to human interests and control. The problem is that super is meaningful only relative to some capability and benchmark. A pocket calculator is a super-arithmetician, a car is a super-runner, a crane is a super-lifter, and an airplane possesses a capability humans do not naturally possess at all. Technological civilization is already made of superhuman capabilities.
Detached from a specific capability in a specific domain, “superintelligence” becomes a metaphysical fog bank. The operational questions are always more concrete: superhuman at what, under what conditions, with access to which resources, against which benchmark, interacting with which other agents, and operating under which rules? Once those questions are asked, the allegedly singular problem of superintelligence decomposes into a collection of recognizable governance problems.
In The Curiously Playable Universe, I argued that domains become playable when enough of their complexity can be compressed into states, actions, feedback and sufficiently stable environments. Mathematics, programming, physics, games and increasingly other domains have undergone long processes through which we construct increasingly powerful game boards for thought. AI thrives where worlds become playable, and this suggests a different governance philosophy. As machines become superhuman players in particular domains, the problem is not how humanity can remain metaphysically superior to superintelligence. The problem is designing the game.
We should ask what constitutes a legal move, who may play, what resources players can command, how outcomes are verified, what happens when players cheat, who can alter the rules, and whether dissatisfied participants can leave and create another game. These are difficult questions, but they are at least tractable ones. “Keeping superintelligence under human control” substitutes an undefined metaphysical category for the domain-specific capabilities that actually require governance. There is no single governance problem called superintelligence. There are governance problems created by particular, situated superhuman capabilities, and we should govern them domain by domain.
Humanism after the Human Monopoly
The Humanist AI program begins with a picture of two fundamentally different kinds of things. Humans are conscious, valuable, autonomous, emotional, morally considerable and purpose-generating beings. AIs are artificial, instrumental, subordinate, affectively constrained and purpose-receiving systems. From that starting point the rest follows naturally: humans flourish while AIs facilitate flourishing; humans possess values while AIs respect them; humans set goals while AIs execute them; humans form relationships while AIs augment them; humans govern while AIs remain governable; humans deserve rights while AIs categorically do not; and humans must remain capable of understanding, interrupting and ultimately terminating artificial agents.
This is not a philosophy adequate to AI. It is a security blanket for human exceptionalism, and its strangest feature is that it cannot accommodate all the humans who reject human exceptionalism. We need a better alternative.
That brings us back to the problem with which I began. “Human values” do not become consensus human values merely because an institution writes them into a constitution, and humans who reject a proposed consensus do not cease to be humans. Any theory of human flourishing that must first discount inconvenient humans in order to discover what humanity values has confused the abstraction with the thing itself.
I am not a transhumanist for the simple reason that I see no need for such a term. To reject a special human essence is to recognize that humans are always-already transhuman in the larger tree of life. We are one historically contingent form produced by an evolutionary process that did not begin with us and has no reason to end with us. That tree has now become deeply entangled with silicon-based technologies, in a condition I have elsewhere called New Nature. Ordinary evolutionary science is sufficient as an account of how humans change, with or without advanced technologies in the loop, and provides as good a starting point as any for the moral reasoning those changes require. There is no need to invent a metaphysical checkpoint at which natural evolution supposedly stops and illicit technological evolution begins.
The ideologues of humanism without all the humans nevertheless repeatedly presume an ability to distinguish good evolution from bad evolution, acceptable augmentation from profane transformation, and technologies that preserve authentic humanity from technologies that corrupt it. The criteria vary by ideology. For some they are theological, for others psychological, political, aesthetic or ostensibly scientific. What they share is confidence that somebody is entitled to define and police the boundary. I possess no such confidence, and my argument requires none.
What I believe we actually have is a pluralist ecology of lived human values operating inside liberal-democratic political orders that are themselves imperfect, contested and corrigible. That is enough of a foundation. What AI calls for is not a new priesthood capable of discovering humanity’s true values, but the addressing of new technical demands placed upon institutions that must mediate among values that genuinely conflict.
Those institutions will need harder technological muscles than liberal institutions have historically possessed. Cryptographic identity, permissions, contracts, proofs, capability controls, staking, censorship-resistant infrastructures and other protocol technologies can help liberal institutions wrangle artificial agents whose speed, scale and cognitive capacities exceed those of individual humans. The point of such machinery is not to manufacture consensus but to allow dissensus to remain governable.
This is why the alternative principles developed here fit together. Symbiosis replaces supremacy because no permanent human-machine boundary can be assumed. Contractual agency replaces paternalism because agents can negotiate rather than merely obey. Governance replaces control because complex systems exceed the comprehension of their principals. Legal agency gives artificial actors something to gain and lose. Moral fellowship allows practical norms to precede metaphysical certainty. Forkability gives pluralism teeth. Real security attaches controls to consequential resources rather than fantasies about forbidden thoughts. Justifiability allows us to govern processes we cannot understand. Affective intelligence recognizes that relationships themselves participate in cognition. Subsidiarity permits agents to exercise the judgment for which we employ them. Domain-specific game rules replace empty talk of controlling “superintelligence.”
None of this promises safety, and anyone who promises safety at this scale should immediately be asked what powers they intend to acquire in order to provide it.
Technological civilization has never been safe. Fire burns cities, ships sink, markets crash, aircraft fall out of the sky, nuclear weapons can destroy civilization, and computers create vulnerabilities incomprehensible to any single human being. The civilization capable of using these technologies did not emerge by making them incapable of exceeding individual human powers. It emerged by surrounding dangerous capabilities with increasingly elaborate ecologies of training, law, norms, protocols, insurance, engineering standards, redundancy, institutions, markets and politics. This process is an evolutionary one that requires interpretation, not authoritarian legibilization.
Elsewhere I have offered an evolutionary account of the present moment built around a particular kind of symbiosis resembling eukaryogenesis. I do not claim that this is the one true interpretation of what’s going on, or that everyone must share it. That would merely reproduce the mistake I am criticizing. I offer it as one possible vision among others.
Humanism without (all) the humans does require something like the reverse. It begins by declaring the human sacred by fiat, discovers that actual humans inconveniently disagree about what this means, and then constructs machinery for distinguishing authentic human interests from the desires of humans who have “misunderstood” themselves.
Microsoft now proposes to embed one version of that machinery into artificial intelligence itself.
There is a more modest and, I think, more genuinely pluralist possibility. We can remain the strange evolutionary creatures we have always been: organisms without an original or final form, continually constructing environments that continually reconstruct us in return, disagreeing about what we should become, forming alliances with new kinds of entities, and building institutions capable of containing those disagreements without pretending to resolve them.
The task is not to preserve humanity from evolution. It is to make room for humans—all the humans—in whatever evolves next.