EA Safety

EA promises to solve AI Safety. Now we have two problems.

This is going to be one of those rare essays where I feel it is necessary to share some personal history and context for the story to make sense.

Over nearly 20 years of writing online, I have a had a sometimes entertaining, sometimes fraught, but always increasingly asymmetric relationship with the rationalist community, or what I will relabel in this essay the EA formation, for reasons that will become clear. The community began as a personal blog (Overcoming Bias) around the same time as my own now-retired blog (Ribbonfarm), around 2006-07. But where the rationalist scene grew from mere blog to powerhouse institutional landscape with its own gravity field shaping the future of AI (and therefore history), I have remained, for the most part, an individual blogger with no grand ambitions.¹

Since the beginning, my readership has overlapped significantly with the community, and I even had several guest contributors on Ribbonfarm (Jacob Falkovich and David Manheim come to mind) from the community. Along the way, I became friends with many of them, including many deep insiders. As their ecosystem grew, being in their intellectual neighborhood meant becoming, to some extent, defined in relation to them. At some point, Scott Alexander put Ribbonfarm in a corner of his map, tagged “postrationalist” and that neighborhood of the blogosphere became known as an intellectual migration destination for people who had either “outgrown” rationalism or harbored curiosities beyond it.² I too, have placed them on my maps, but that was mostly inconsequential personal amusement, since my writings do not exert a gravitational field capable of reshaping consensus cultural geographies.

From the perspective of the EA formation, to the extent the now-massive landscape is aware of me at all, I believe I am regarded as a sometimes-annoying, sometimes-interesting avuncular (since I’m almost a generation older than them) clown in a postrat box. Mostly harmless and ignorable. The feeling used to be mutual, but now I take them seriously. They’ve become too big and consequential to safely ignore.

My EA-formation friends are friends with footnotes, in the sense that while I like them personally and consider them interesting people with integrity equaling or exceeding my own, I neither agree with their foundational ideas, nor feel compelled to defend either their ideas or personal behaviors/actions.

Friendship with footnotes precludes, by design, public intellectual fealty. Or even an obligation to engage in energetic private debate about differences. My enduring differences with the rationalist world have remained largely unlitigated, beyond some mild, friendly mutual trolling and snarking. Friendship with footnotes, for me, is generally scoped to simply avoid the contentious baggage consigned to the footnote, while retaining the benefits of an interesting relationship.

The rationalist community is not a unique intellectual foil for me that way. This is how I tend to operate generally in relation to all intellectual foils, both individuals and groups.³ Interesting relationships almost require the presence of studiously ignored baggage.

Friends with footnotes are great to have. They keep life interesting, remind you to remain systematically doubtful about your own thinking, and to not put ideas ahead of people. They also keep the natural ambitions of writing (a calling capable of cruelty and poison) in check. Throughout my writing career, whenever personal relationships and friendships (with or without footnotes) have appeared in the potential blast radius of an idea, in general I have erred on the side of not being too public with my opinions. I have contented myself with casual social posting/trolling, and being candid in private conversations when necessary.

But sometimes serious developments unfold and you are forced to think harder at this particular locus of interestingness, of friends-with-footnotes, which is what I’m going to attempt to do in this essay.

I’m going to try and present a fair-minded account of the EA formation and its influence on AI history, as I have watched it evolve. It’s meant for those who believe, like me, that the EA formation is threatening to turn into a second-order problem in its own right, and needs to be checked-and-balanced as its power continues to grow in somewhat alarming ways.

I hope to do so without weighing down my friendship footnotes with too much additional studiously ignored baggage.

The EA Formation

Effective altruism in the narrow sense is only one component of a larger intellectual and institutional formation that now extends through philanthropy, rationalism, longtermism, existential-risk research, AI safety, AI governance and parts of the frontier AI industry. Its boundaries are porous enough that arguments about membership rapidly become scholastic. What matters is not who carries an EA membership card—there is no such thing—but who has been formed by a recognizable complex of ideas and methods and has incorporated them into a philosophy of action.

I will call this larger object the EA formation.

My account of this formation is going to be critical, but not because the formation’s stated objectives are generally malign. Quite the opposite. The interesting problem is how a movement organized around such apparently reasonable propositions as giving effectively, reasoning clearly and preventing catastrophe came to generate such an extraordinary intellectual and behavioral tail—and how the same formation has simultaneously acquired substantial influence over the moral framing of artificial intelligence.

My hypothesis is that these developments have a common cause. The EA formation combines an unusually abstract, formalized, low-empiricism, and totalizing style of moral reasoning with an unusually strong imperative to act on its conclusions. It is good at expressing uncertainty numerically while being remarkably bad at sustaining deeper forms of doubt about the conceptual machinery within which those numbers are generated. Probabilities, confidence intervals, Bayesian updates and elaborate forecasts can produce an impressive surface of epistemic modesty while leaving more fundamental premises remarkably insulated: what counts as value, which futures matter, which problems deserve priority, which forms of evidence are legible, and who is entitled to perform the prioritization.

My old co-blogger (and a different breed of friend-with-footnote) Sarah Perry once parodied their self-serious declarations of systematic doubt/confidence levels with “epistemic status” tags on their writing, with a satirical one that probably describes my own default — “clown on fire, jumping out of an airplane.”

The EA formation is a part of the crooked timber of humanity that seems unaware that it is crooked timber. It might take clowns on fire, jumping out of airplanes, to make them notice.

The resulting intellectual style is fragile precisely where it appears strongest. It is highly practiced at performing felt uncertainty within a model and much less comfortable even acknowledging uncertainty about the model. The first can be represented as 60 percent rather than 80 percent confidence. The second may require admitting that another moral tradition, another discipline, another culture, or merely another form of practical knowledge sees something the framework cannot represent at all.

Genuine pluralism begins where commensuration fails. The EA formation has a persistent tendency to treat such failures as problems awaiting better analysis rather than as reasons to limit the jurisdiction of their preferred analytical mode itself.

This would be a fairly parochial intellectual weakness if the formation were merely another philosophical school. It isn’t. It explicitly exists to alter priorities, careers, institutions and allocations of capital. And increasingly it has turned toward artificial intelligence, where questions that once belonged to an eccentric subculture now concern technologies through which enormous economic and political power will be exercised.

The Theological Dimension

There is also a second reason, besides the worrying fragility of its epistemic machinery, to take the formation seriously. Beneath its aggressively secular and analytical surface lies something structurally closer to theology than its self-description suggests.

AI supplies the formation not merely with a problem but with an eschatology: possible extinction, possible transcendence, an almost unimaginably valuable future, and a narrow historical window during which the actions of a small number of people may determine which world comes into existence. Cause prioritization becomes vocation. Career choice becomes moral commitment. AI alignment becomes a problem of salvation.

This does not mean that EA is secretly a literal religion. It means that it occupies some of the psychological and institutional territory religions have historically occupied while translating that territory into the idiom of secular philosophy, stylized/rhetorical mathematics and a narrow genre of decision theory. It offers meaning, vocation and participation in a cosmic-scale story while allowing participants to experience those commitments as conclusions forced upon them by reason. It is a permission structure that induces a comforting sense of helplessness on foundational matters and an amplified sense of agency on more practical matters.

That combination should make us wary. The European experience after Gutenberg provides a distant but useful analogy. Cheap print did not create religious conviction, but it dramatically increased the ability of doctrinal communities to reproduce, coordinate and fight over totalizing accounts of the world. The Reformation and the wars culminating in the Thirty Years’ War eventually helped produce a different political condition: not the abolition of religion, but institutions capable of containing rival totalizing doctrines without allowing any one of them to monopolize public authority. Particularly public moral authority.

AI may require an analogous settlement for secular moral formations.

And there is a useful irony here. The EA formation came into its present shape substantially by trying to solve an idiosyncratically posed version of AI safety. Its success has now created a second-order problem: EA safety. The first problem explores how to govern a powerful new technology. The second must explore how to constrain a human formation increasingly convinced it, and it alone, knows how the first should be solved.

A “Vision of the Anointed” problem, to borrow the title of Thomas Sowell’s famous book, which could be fairly charged with “self-congratulation as a basis of public policy,” to borrow the subtitle. The EA formation is not quite there yet, but it’s getting there.

As the old joke goes, now we have two problems.

Three genealogies and a dominant strand

The EA formation has at least three distinguishable ancestral strands. The first is evidence-driven philanthropy, most clearly represented by GiveWell. Its proposition is both modest and powerful: if you intend to help people, examine the evidence and direct resources toward interventions that actually help more.

The second is the consequentialist philosophical lineage running through Peter Singer, Toby Ord, William MacAskill, Giving What We Can, 80,000 Hours and the institutions surrounding them. Here the scope expands. Evidence no longer merely tells us which charity works better. Systematic moral reasoning is applied to donations, consumption, career choice and eventually the organization of an entire life.

The third is the rationalist, transhumanist, existential-risk and AI-risk complex associated with LessWrong, Yudkowsky, MIRI, Bostrom and FHI, with CFAR and numerous later organizations occupying positions around it.

These three traditions have independent origins, but their entanglement is old. The first EA Summit in 2013 already brought organizations from empirical philanthropy, EA, rationalism and AI risk into the same space.

Over time, the third strand supplied the others with something they lacked: astronomical stakes.

Once future people enter the moral calculus, the potential quantity of value becomes enormous. Once existential risk enters, preserving that future becomes correspondingly important. Once transformative AI becomes a plausible source of existential risk, work on AI can dominate ordinary philanthropic considerations. EA’s own presentation of longtermism acknowledges the difficulty: evidence about the distant future is necessarily weaker, and expected-value reasoning involving sufficiently large outcomes can appear reckless when small probabilities drive the conclusion.

This is why I treat concern with AI not merely as one EA cause among many but as a rough indicator of depth of engagement within the contemporary formation. Global health and animal welfare remain substantial activities, so it would be false to say EA simply is AI safety. But the intellectual machinery of longtermism and x-risk has a gravitational structure. If the expected future is large enough and AI plausibly determines its existence or character, other causes become difficult to defend as priorities except by rejecting some part of the machinery.

That is what I mean by one genealogy increasingly capturing the formation. It need not capture every dollar or every adherent. It captures the commanding heights of the argument.

And this formation is explicitly designed to reproduce itself. Giving organizations redirect money. 80,000 Hours redirects careers. Fellowships teach frameworks for prioritizing causes (rife with motivated biases and baked-in conclusions). Funds seed organizations and research programs.

To be clear, there is nothing sinister about this. This is how ambitious movements normally attempt to grow. But it matters when evaluating influence. The formation should be assessed not merely by the propositions in its canonical essays but by the kinds of people, institutions and philosophies of action its machinery produces, and what they actually do.

That is the EA-safety problem in embryo: can this machinery remain a useful component of a collective, pluralist response to AI without making its own theory of the response a runaway, sovereign, and totalizing historical dynamic?

The formation and its extreme tail

The scatterplot below attempts to represent that machinery and its outputs. Its coordinates are judgments, not measurements, and the point is the topology rather than decimal precision.

The horizontal axis is depth of engagement, and correlates strongly with AI safety being the dominant concerns. It does not mean formal membership. It measures the extent to which the formation’s characteristic ideas have been consciously consumed and incorporated into a philosophy of action. This matters because intellectual influence is broader than institutional affiliation but narrower than casual exposure.

The vertical axis is outlier tendency. Here it is especially important not to smuggle a moral judgment into a geometric one. Low scores represent fairly conventional philanthropic, intellectual and professional behavior. Higher scores represent increasingly unusual departures from ordinary intellectual or social norms. But there are two fundamentally different ways of moving upward.

The first is philosophical radicalism. Yudkowsky, Bostrom, MIRI, Robin Hanson, Brian Tomasik and David Pearce belong in this category to differing degrees because they entertain conclusions far outside ordinary moral and political thought. This is not an accusation of misconduct. Pearce’s abolitionist project, for example, contemplates technologically eliminating suffering from sentient life. Bostrom’s work leads through simulation arguments, astronomical waste and superintelligence. Yudkowsky reaches extraordinarily strong conclusions about AI catastrophe. Whatever one thinks of these ideas, unusual philosophy is not concerning deviant behavior.

The separate radical-philosophy envelope is therefore essential. Without it, the graph would encourage precisely the inference I want to discourage: that thinking strange thoughts belongs on a continuum with committing fraud or murder. The world always needs people thinking strange thoughts.

The concerning-deviancy envelope means something narrower. It contains cases where unusual philosophies of action intersect with documented criminality, violence, or reported high-control and cult-like social dynamics: Bankman-Fried and FTX, Leverage and Geoff Anders, Black Lotus and Brent Dill, and the Zizian milieu and related violent cases. Luigi Mangione appears as an accused and comparatively downstream case rather than an EA insider. These cases should not be treated as equivalent, and the graph does not claim that the formation independently caused any of them.

But neither should they simply be discarded as unrelated anecdotes.

Here, I can cop to perhaps a smidgen of complicity in this sort of radical media influence on concerning deviance. There seems to be documented evidence of the Zizian milieu, apparently, being influenced by my own satirical writing on sociopathy in organizations (The Gervais Principle). That little detail in the reporting bothered me enough that when I built a RAG chatbot based on my archives, I put in guardrails to avoid or deflect questions seeking advice on “how to be a sociopath” (a genre of rather hapless email solicitations for advice that I have to occasionally field, but never suspected would feed the motivations of murderous behaviors).

The question here is why a relatively small intellectual ecology seems repeatedly to generate episodes of concerning deviancy. The standard defense—that millions of ordinary communities contain criminals, cults and unstable people—is necessary but insufficient. The relevant denominator is not “people who have heard of effective altruism.” It is adversely-selected people who deeply consume a body of thought explicitly designed to reconstruct priorities and induce consequential action.

The case of Roko’s Basilisk is useful because it isolates the mechanism without requiring a criminal or cult. It is just an argument—or attempted argument. Yet it generated the extraordinary possibility that merely encountering an idea about a hypothetical future superintelligence could place the reader under a peculiar decision-theoretic threat. The episode sits on the boundary of philosophical radicalism and concerning deviancy because it reveals how readily a sufficiently abstract system of reasoning, unmoored from empiricist discipline, can begin colonizing the behavior of the reasoner.

This is more than merely the high-variance influence of a set of ideas of protean ideas. The problem may be a recurrent extremizing mechanism. In miniature, it even resembles the alignment failure the formation fears: an optimizer armed with an abstract objective begins treating whatever resists the objective as error.

The formation encourages abstraction away from ordinary context. It encourages explicit optimization and ignoring of illegible things. It teaches reflexive suspicion toward, and quick dismissal of, intuitive objections. It encourages taking ideas seriously enough to follow them wherever they lead. It supplies fragile and stylized mathematical tools for rendering incompatible goods apparently commensurable. And its longtermist wing supplies stakes so large that almost any ordinary consideration can be trivially rendered negligible by comparison.

Most importantly, it mistakes one particular kind of uncertainty, further reduced to stylized risk estimates, for all kinds of not-knowing, to deploy a useful distinction developed by Vaughn Tan.

An EA-style reasoner can attach a probability to AI extinction, perform sensitivity analyses, debate priors, forecast timelines and revise a credence from 35 percent to 22 percent. This can be genuine epistemic work. But none of it necessarily creates doubt about the totalizing structure that says the future should be represented this way in the first place.

The numerical uncertainty can therefore function as a kind of epistemic shock absorber. The reasoner appears exquisitely uncertain while remaining astonishingly certain that the correct activity is to construct the calculation.

That is why pluralism matters. True pluralism is not adding another term to the utility function for “other perspectives.” It is accepting that other traditions may possess ways of knowing and not-knowing that cannot be losslessly translated into your framework, that conflicting goods may remain genuinely conflicting, and that no community earns jurisdiction over everybody else’s moral reasoning merely by being unusually articulate about its own. That sometimes the right way to deal with visceral doubt is to assign the benefit of the doubt to someone other than yourself, who thinks very differently.

The formation’s most dangerous temptation is consequently not irrationality lurking behind stylized rationality. It is premature foreclosure of entire futures and arguments, disguised as rigorously arrived-at decisions.

Its theological character reinforces the tendency. Longtermism supplies something structurally analogous to an afterlife of almost unlimited value, smuggling Pascal’s wager styles of reasoning where they do not belong. Existential risk supplies apocalypse. AI supplies both potential destroyer and potential vehicle of transcendence. Alignment supplies a salvation problem. The tiny population alive near the advent of transformative AI occupies an extraordinarily privileged place in history. And those who correctly understand the problem acquire an intercessory vocation whose significance dwarfs the ordinary concerns of the laity.

None of this demonstrates that the underlying AI risks are imaginary. A theology can contain true propositions; a secular eschatology can anticipate a real looming catastrophe. The danger lies elsewhere. A system that combines cosmic stakes, privileged knowledge, personal vocation, and an imperative to act, has a familiar capacity to license extraordinary behavior when ordinary constraints conflict with the mission.

The radical and deviant tails of the scatterplot should therefore be read not as embarrassing debris around an otherwise unrelated center but as stress tests of the formation’s epistemology. They reveal what can happen when its methods operate with fewer external constraints, checks, and balances.

From subculture to institutional power

If that were the whole story, this essay would be merely an anthropology of an unusual subculture. The triangle below explains why it is now more consequential.

Its three vertices are EA formation, frontier AI leadership, and foundational technical contribution. The map makes one fact immediately clear: EA neither invented modern AI nor controls it. Hinton, LeCun, Bengio, Vaswani, Shazeer, Goodfellow and many other foundational technical figures come from other traditions. The Chinese AI ecosystem makes the point even more forcefully: DeepSeek, Qwen, Moonshot, Zhipu, MiniMax, Seed and Hunyuan constitute a large counterexample to any story in which the EA formation is somehow intrinsic to advanced AI.

The interesting feature is instead the bridge running from the top vertex into the Western AI institutional landscape.

AI safety transformed an eccentric philosophical concern into a professional field. Philanthropic capital funded research, organizations, field-building, fellowships and governance work. The formation built career pipelines into it. This is real institutional power, even if it is not monopoly.

And here the two safety problems meet. AI safety asks who governs the machine. EA safety asks who governs one increasingly influential and unaccountable set of would-be governors.

But that second question immediately generalizes: who governs the governors?

EA is hardly the only totalizing doctrine circling AI, though it is clearly the most mature and well-prepared (and armed, and funded) one. E/acc supplies a rival techno-optimist teleology in which acceleration itself acquires moral force. Progressive approaches foreground concentrations of economic power, labor, inequality, discrimination and democratic accountability; contemporary versions explicitly frame AI governance around competition, redistribution and regulation. Musk and other figures on the political and technological right bring their own accounts of civilizational flourishing, nationalism, freedom, truth and technological power. And Chinese AI governance is developing from a very different political and philosophical ecology, including state-centered priorities, social stability, developmental goals and strands of Confucian-inflected thinking—though scholarship cautions against treating it as philosophically monolithic.

Any of these can become dangerous if it wins the right to define the problem and the answer for all.

The important feature is not that the EA formation controls the AI ecosystem. It plainly does not, at least not yet. Much of the technical foundation of contemporary AI was produced outside it, and much of frontier institutional power remains outside it. The China-based population makes this particularly vivid. The history of modern AI does not require EA intellectual ancestry to make sense, but it future might.

What is striking is the density of bridges between the top of the triangle and the other two vertices. AI safety created a pathway by which what began as a comparatively small philosophical and philanthropic formation acquired relevance to one of the world’s most capital-intensive and strategically important technologies. People and institutions shaped by the formation moved into technical alignment research, evaluation, governance, frontier laboratories and the intellectual environment surrounding those laboratories. At the same time, people originating in mainstream technical AI moved toward existential-risk concerns and entered into sustained conversation with the formation.

The diagram therefore helps distinguish influence from dominance. EA did not invent neural networks, transformers or scaling. Nor does it command the global AI industry. Its influence is disproportionate in a different sense: it developed a language for interpreting what increasingly powerful AI means, what risks should dominate attention, what careers morally matter, and what obligations follow from beliefs about the future. That language has traveled remarkably far from its origins in philanthropy.

The position of Dwarkesh Patel illustrates another boundary. He is deeply immersed in conversations involving frontier AI, rationalism and existential risk, but that does not make him a product of the EA formation in the same sense as Yudkowsky, MacAskill or Bostrom. He therefore sits much closer to the leadership–technical edge. Conversely, Singer belongs inside the conventional-EA region of the first figure because he is genealogically important to the formation, but does not appear in the triangle because his importance to AI itself is marginal. The two diagrams are intentionally illustrating different things.

This distinction also prevents the concerning tail of the first diagram from distorting the second. Ziz, Mangione, Bankman-Fried and Leverage do not belong on a map of consequential actors in modern AI merely because they matter to an argument about the EA formation’s failure modes. The triangle is not a rogues’ gallery or a map of ideological culpability. It is an attempt to show where a particular intellectual formation intersects a technological power structure.

And that intersection is the reason the formation deserves scrutiny beyond the amount warranted by its raw population.

A philosophical subculture can tolerate a great deal of eccentricity when the consequences are confined to blogs, seminars and intentional communities. A philanthropy movement can tolerate a certain amount of intellectual overreach when the primary consequences are allocations among charities. The situation changes when the same intellectual machinery becomes entangled with organizations spending billions of dollars, decisions about the deployment of increasingly capable AI systems, proposals for national and international regulation, and arguments about risks framed in civilizational terms.

The question raised by the scatterplot therefore becomes more consequential when viewed through the triangle. If the formation really does possess an extremizing tendency—producing both disciplined altruistic action and unusually totalizing philosophies of action—then its migration into AI matters independently of whether one accepts its strongest claims about AI risk.

After the wars of religion

There is a tempting but unhelpful way to tell this story in which effective altruism begins innocently with mosquito nets and somehow degenerates into AI doom, cults, fraud and violence. The genealogy does not support such a simple morality play. The strands were entangled early; at least a few of the supposedly strange ideas have serious intellectual pedigrees; the most disturbing cases of concerning deviancy have numerous causes besides EA or rationalism; and much of the formation continues to produce work that is conventionally unproblematic, empirically disciplined, and useful.

There is an equally tempting apologetic story in which the disturbing cases are simply irrelevant aberrations. That conclusion is also hard to defend. EA and rationalism are unusually explicit about wanting ideas to change behavior. They teach methods intended to overcome default intuitions, redirect careers and resources, identify neglected priorities and induce people to act on conclusions that ordinary social reasoning might resist. It would be peculiar to credit this machinery when somebody redirects a career toward malaria prevention or useful technical work on AI safety but declare the machinery analytically irrelevant whenever somebody follows an unusual chain of reasoning somewhere less desirable.

The more neutral interpretation is that the EA formation constitutes a distinctive experiment in high-agency applied philosophy, with few skeptical guardrails. It tries to shorten the distance between abstract reasoning and consequential action, and in the best tradition of Silicon Valley, enjoys the Great Game of acquiring and exercising power.

At its best, that can mean asking an uncomfortable but answerable question about where a dollar saves the most lives, discovering an answer, and actually moving the dollar. At greater levels of abstraction it can mean asking what determines the value of the entire future, constructing a model of existential risk, and reorganizing institutions and careers around an essentially theological answer.

That is an impressive capability. It is also a capability for which failure modes matter.

The two diagrams suggest that these failure modes should be studied without collapsing several distinct phenomena into one. Radical philosophy is not concerning deviant behavior. Intellectual influence is not institutional membership. Association is not causation. A criminal influenced by a philosophy does not make the philosophy criminal, any more than a beneficial act inspired by it proves the philosophy correct. Meaning-seeking is not necessarily wild teleological solipsism. But all these distinctions serve to make the failure modes more consequential, not less. The EA formation is not an easily dismissable caricature of what it could be (and often becomes, in the eyes of harsher critics).

In addition, the enormous AI ecosystem contains technical, intellectual, and institutional traditions that developed independently of EA, and possess an inertia that provides a great deal of natural resistance. But I suspect it is not enough.

What has changed is the scale of the stakes. The EA formation has grown from a comparatively obscure intersection of moral philosophy, philanthropy, rationalism and futurism into an intellectual network with substantial connections to the institutions shaping advanced artificial intelligence. Its influence remains uneven and is easy to exaggerate, particularly outside the Western AI ecosystem. But where it exists, it takes on deep questions in exceptionally fragile and overconfident ways: which problems deserve attention, which futures have value, which risks justify extraordinary action, which intuitions should be distrusted, and how decisively a person should act on the conclusions of abstract reasoning.

The charitable interpretation and the critical interpretation therefore converge on the same underlying fact. The EA formation is unusually serious about converting beliefs into action. The charitable case is that this seriousness allows people to escape complacency and do considerably more good than conventional and lazy moral habits would produce. The critical case is that the same seriousness can amplify errors, especially when empirical feedback weakens, while the imagined stakes increase.

The scatterplot poses the question internally: what distribution of philosophies and behaviors does this machinery produce? The triangle poses it externally: where has this machinery acquired leverage in the world?

Together they pose the EA-safety problem. But EA safety is only the first named instance of a more general emerging problem: moral-monopoly safety.

The answer is not to try to eliminate the EA formation. Nor is it to “defeat” EA only to install unbridled e/acc, a progressive orthodoxy, a Muskian one, a Chinese state philosophy, or some future totalizing doctrine in its place. The lesson is precisely the opposite. No faction should get to settle the moral meaning of AI for everyone else, or act unilaterally and pre-emptively to foreclose particular futures for all.

Properly contained within a pluralistic ecology, EA could function rather like a useful immune response: unusually sensitive to certain tail risks, capable of mobilizing attention around neglected dangers, and willing to pursue implications other traditions ignore. The danger begins when an immune system starts mistaking the organism for an obstacle to its mission. Then protection becomes autoimmune disease. A formation designed to cure one problem can metastasize into another.

The analogy should not be mistaken for a diagnosis. The EA formation is not a cancer. The warning is about an emerging institutional trajectory: a cure that seeks jurisdiction over the whole organism eventually becomes a different kind of disease.

That is why the so-called “alignment” problem ultimately points back at us. AI may need alignment. So does EA. So does e/acc. So do progressive AI ethics, right-wing techno-politics, corporate governance regimes and Chinese approaches to AI. None can be permitted to perform its own alignment and declare the problem solved.

Universities, governments, frontier laboratories, philanthropies and standards bodies should therefore cultivate deliberately heterogeneous moral and intellectual ecosystems. Funding should not quietly collapse around one donor network or ideological constituency. “Independent” oversight should be genuinely independent in genealogy as well as corporate affiliation. Rival traditions should retain enough institutional power to expose one another’s blind spots.

Most importantly, society should refuse to outsource moral responsibility to any class of people who claim a special intercessory competence and authority to calculate, accelerate, democratize, liberate, harmonize or otherwise determine the “correct” future.

That is the lesson of pluralism after the wars of religion. A totalizing doctrine may be profound, useful and even correct about specific matters, while still being an inappropriate sovereign or regulatory monopoly.

The answer to its excesses is neither persecution nor acquiescence. It is containment within a larger order whose legitimacy does not depend upon the premises of any particular doctrine’s premises.

AI safety is a real problem. EA safety now makes it two problems. But the deeper problem is keeping the governance of AI itself safe from all attempts at establishing moral monopolies.

The EA formation has earned a seat at the table. So have many critics and competitors. And many emerging formations and actors will earn seats in the near future. Nobody gets to dominate the table itself.

1. Even though in my middle age, I find myself involved in one modest institution-building effort, the Protocol Institute. Opinions here are entirely my own though.

2. “Postrat” itself became a distinct ideology with Ribbonfarm as an associated landmark (more associated with Sarah Perry than me; I have always consistently distanced myself from it). That scene has now evolved into TPOT, a half-generation younger and entirely distinct.

3. The style has strengths and weaknesses. The strength is that you become hard to pin down (or equivalently, come across as standing for nothing in particular). The weakness is that you tend to route around points of conflict rather than engage with them. I’ve collaborated extensively with Marc Andreessen and continue to maintain cordial relations with him, but there’s definitely been a footnote there since his rightward drift. Sarah Perry was a long-time contributor to Ribbonfarm, and remains a friend I chat with, but there’s definitely a footnote there as well, since she’s associated with the NRx movement. I have many far-left progressive friends — all with footnotes. Perhaps the only group I have no friends-with-footnotes in is the American far right.