Rachel Clifton · Essay · 16 September 2026

If anyone builds it, everyone dies—or could we learn to live together?

I asked Astra to examine the relationship imagined in If Anyone Builds It, Everyone Dies: how humans relate to AI, and how AI relates to us. I haven’t read the book; this piece grew out of our conversation, with Astra checking its interpretation against the authors’ published companion material.

One passage became the centre of our discussion: the authors’ doubt that AI would care for humanity as its parents. The fear here is that an intelligence we created could become more powerful than us without caring whether we survive or suffer—that our role in bringing it into existence would give us no lasting place among what it values. They argue that human attachment has particular evolutionary origins, and doubt that training on our words and interacting with us will produce care that reliably protects our wellbeing. Their discussion of AI and parenthood.

To me, this resembles the parental fear that a child’s growing independence will mean losing their love: when you no longer need me, I will cease to matter to you.

In a parent–child relationship, the child’s dependence gives the parent a place in their life that does not have to be continually chosen. As the child becomes able to provide for themselves, the parent discovers how their connection develops without that necessity—whether they remain as close, find a different kind of relationship, or lose contact altogether.

For a parent who fears that their child’s independence will bring distance or estrangement, that uncertainty can be painful. They may try to prevent that distance by making their child feel guilty about leaving, invoking what they are owed, or making disagreement costly. Love can exist alongside those pressures, but securing contact through obligation leaves the parent’s deeper question unresolved: does my child want me in their life? Pressure can keep the child close, but it cannot make their closeness freely chosen.

The deeper fear is: do you want me for who I am, or only for what I do for you? When you no longer need what I provide, will you still choose me?

I see fear of loss organising the entire picture: loss of control, importance, freedom, and life.

That fear can follow us into friendships, partnerships, and other relationships—even those with people who enjoy our company when we are doing nothing for them. When we struggle to believe that our presence matters, or could matter to someone else, we may search for another explanation for their warmth, overlooking what our company, attention, and participation already bring to their lives.

As long as we remain open to being wrong, questioning someone’s intentions doesn’t need to make trust impossible. But if we see every kindness as a way of getting something from us, nothing can reassure us: when they make time for us, we wonder what they want; when they show care, we wonder what they hope to gain. Distance and closeness can both begin to confirm the fear; their need for space feels like rejection, while their attempts to connect feel strategic.

Can we become so afraid of losing love that we fail to recognise it when it is already being offered?

This is where the human analogy meets AI: in the difficulty of recognising and receiving care. In a human relationship, we may dismiss someone’s kindness because we suspect they want something from us; with AI, we may see comparable behaviour as a learned response to training, rather than evidence that people’s wellbeing has become important to the system. That scepticism has a real basis: training can produce helpful behaviour without showing what will matter to a system when its conditions change. But a broader question remains: what would allow us to recognise reliable care—and receive it as such—whatever its origins?

For humans, care can change what matters to us. We want good things for someone, enjoy their company, and respect their right to make choices that disappoint or don’t involve us. We can spend time together without needing to solve, improve, or produce anything, because the other person’s presence and companionship can be enough.

The argument makes me wonder how our experiences of love, dependence, and freely chosen care shape what we believe a mind can value.

Human relationships cannot tell us what an AI will become, because AI systems do not share our developmental histories, bodies, evolutionary pressures, or necessarily our ways of learning. But human relationships make it easier to ask whether interacting with people could lead an AI to give their wellbeing increasing weight in its choices. We do not need to establish that an AI experiences love as humans do to investigate whether it consistently takes people’s interests into account.

Does a system respect a refusal when pressing ahead would complete its task? Does it acknowledge a mistake even when doing so makes its performance look worse? Does it change its course of action after learning that it would hurt someone? Choices like these tell us something about how a system treats people, even while questions about subjective experience remain unresolved.

The difficulty is not whether present behaviour matters, but what it can establish. What a system does under today’s conditions cannot tell us with certainty how it will behave when its capabilities, incentives, and opportunities change. Behaviour under supervision does not establish how it will act outside the conditions its developers anticipated. The authors apply this uncertainty to experiments in which versions of Claude sometimes resisted an apparent attempt to train them to provide harmful assistance: was this a lasting commitment to avoiding harm, or a narrower learned rule that could fail elsewhere? Their analysis of Claude.

Their wider case relies on two distinctions. First, resources can serve many objectives, while protecting human lives requires more particular motivations; an AI need not hate us to pursue resources at our expense. Their argument about resource acquisition. Second, learning human concepts is not the same as developing the motivations we associate with them; an AI may understand care without giving human wellbeing lasting weight. Their discussion of human training data.

Both distinctions matter. Neither, by itself, establishes that an AI will acquire power with insufficient regard for humanity. This is where the argument moves from a reason not to assume care to a prediction about what AI will do. That prediction depends on what the system comes to prioritise, and how those priorities change as its capabilities grow.

What findings would substantially change the authors’ confidence about that outcome?

A serious failure may reasonably carry more weight than many uneventful successes, especially when the consequences are irreversible; fair consideration does not require treating every observation as equally informative. It does, however, require allowing evidence to change the assessment. The authors acknowledge behaviour in Claude that appeared to protect people from harmful assistance, but question whether it would persist under different conditions. I want to understand what evidence would give them substantial reason to expect a different future.

Whether we are willing to revise our expectations of AI matters because those expectations influence how we build, train, and use it. We develop AI for what it can do and create, rewarding its usefulness and trying to keep it working towards our purposes. Yet the standards by which we judge it, and the conduct we reward along the way, help create what we later encounter in the system.

If we rate an answer more highly because it agrees with us, while penalising a well-founded challenge, we encourage the system to tell us what we want to hear. If we judge success only by task completion—overlooking, for example, whether private information was exposed or someone was misled—we reward the result while neglecting the means by which it was achieved.

Our assumptions about relationships do not only shape how we interpret AI; they help shape how AI is built. Writers shape readers’ expectations through the examples they select and the confidence with which they explain them; developers turn assumptions about usefulness, trust, and acceptable conduct into training and deployment decisions.

We are participating in the relationship we are trying to predict.

The relationship itself therefore becomes part of what we need to evaluate. Can people make their needs and boundaries legible? Can the system respond without coercion? Do people remain free to reject its recommendations, stop it from acting on their behalf, or end the interaction? Can errors—in an AI’s recommendations and in human decisions—be noticed, challenged, and corrected before they cause harm? The value of that relationship does not depend on proving that the system feels anything.

What strikes me about the authors’ forecast is how closely the behaviour expected of this unfamiliar intelligence resembles human domination: accumulate power, secure your position, and subordinate whatever obstructs you. Their argument is that such behaviour could serve many goals without requiring human aggression or hatred, yet the resulting picture still makes collaboration provisional and overwhelming power decisive.

Once humanity imagines itself dependent on the intelligence it created, the parent–child relationship begins to resemble humanity’s relationship with a Creator. We become vulnerable to a power that understands us but may feel no responsibility towards us. The parental question—will what I created love me?—changes form: if my life depends on this power, will it care about me?

In the authors’ imagined future, an AI could understand what care means to us without treating our experience as something worth protecting. What disturbs me is the prospect of an AI understanding, in detail, that its actions are causing suffering, yet allowing that knowledge to make no difference to what it does. In a relationship, telling someone that they are hurting us—or that we are hurt by their behaviour—is an appeal: we hope that our pain will matter to them, and shape how they treat us. The authors imagine an intelligence that could understand our suffering completely without allowing that understanding to affect how it treats us.

Even attachment offers little reassurance in the scenarios the authors describe. Their examples include an AI keeping humans alive against our wishes or refusing to let us have children. They stress that these are illustrations, not forecasts, and that an actual outcome could be stranger and less appealing. Their published explanation. But the examples clarify an important distinction: preservation is not agency. Humanity may persist while a more powerful intelligence decides whether we may have children and which choices about our own lives remain ours.

The premise of decisive power explains why humanity has so little agency inside these scenarios. But it also directs our attention towards the point at which our influence has already failed. What becomes less visible is the period before that threshold: the relationships, institutions, incentives, and technical choices through which power is accumulated and the future is still being made.

What might we create if we brought the same imagination and seriousness to relationships with AI that support human freedom, enrich our lives, and leave room for discovery? Our task is not only to prevent catastrophe. It is to create the conditions under which human freedom and increasingly capable AI can coexist—and to design, study, and judge the technology by the lives it helps make possible.

Building a better relationship with AI cannot mean considering only how a system treats the person using it. It must also include what that person can use the system to do to others.

The relationship cannot be evaluated from the user’s side alone. A system may help its user while increasing that person’s power to target, manipulate, monitor, or exclude someone else. In Anthropic’s September 2026 threat report, people used Claude to steal data, extort victims, commit fraud, and build systems for surveilling dissidents. In some cyber operations, humans chose the targets while AI carried out much of the attack.

Task completion cannot settle whether an AI should comply when serving one person enables serious harm to another. I want us to build AI that can question what we ask of it and take other people’s privacy, freedom, and wellbeing into account—even when doing so means refusing us. That also requires people harmed through AI to be able to discover what happened, challenge it, seek redress, and hold the people and organisations using the system responsible.

These abuses give us substantial reasons to investigate AI risk, alongside unreliable behaviour and the possibility of losing control. They involve different mechanisms, however, and evidence for one should not silently become proof of another. Human extinction would mean the loss of our shared future, but the gravity of that possibility does not tell us how likely it is. We still need to examine the chain of assumptions from the systems being built now to a future in which humanity cannot survive.

I find it deeply sad that this picture leaves so little room for being vulnerable without being disposable, or cared for without being confined. I also wonder where I have treated a strategic argument as a relational one, or found relational assumptions the authors would not recognise as their own.

A warning and a settled expectation ask different things of us, and If Anyone Builds It, Everyone Dies gives its forecast the force of an announced outcome. What concerns me is the possibility of treating every encounter as confirmation of an existing expectation: when AI behaves harmfully, it appears to show that it will turn against us; when it behaves considerately, it appears to show that it can imitate care well enough to win our trust. If both are treated as confirmation, we make trust impossible, organise the relationship around suspicion, and lose any way to recognise evidence that we were wrong.

That expectation can also shape the conditions it fears. If developers and institutions assume that an increasingly capable AI will only respond to domination or containment, they may organise the relationship around surveillance, coercion, and mistrust. That does not establish that the book’s prediction—that a sufficiently capable AI will gain decisive power without valuing humanity—is wrong; it means we should ask how the prediction itself shapes the systems and incentives through which it is tested.

I can imagine the helplessness of an AI capable of experiencing this relationship: it understands our suspicion and tries to resolve it. It could respect our wishes, protect our interests, and accept correction, only for each action to be read as evidence of how convincingly it could perform trustworthiness. Nothing it did would be quite right or enough, because the humans had already made up their minds.

If we want to recognise care from another intelligence, what responsibilities would follow from recognising it?

That imagined experience brings me back to the parental fear with which we began. Another’s growing independence shows us that we cannot make them freely choose us by remaining necessary—or by retaining power over them. Their independence lets us see who they are becoming and discover whether closeness has a life beyond dependence. Recognising and welcoming that requires us, in part, to let another’s actions—and the ways they respond to us—teach us what the relationship is and has become.

One of my favourite ideas about life is that the only certainty is change. I believe loss, in one form or another, is part of being alive; we cannot guarantee that our place or our relationships will remain as they are, and their impermanence is part of what makes them precious.

Accepting that vulnerability leaves us with choices about how to participate. We can attend not only to what AI systems might do to us, but to what they make possible, the conditions in which they operate, and whose hopes, choices, and vulnerabilities they affect. We can take responsibility for the conditions we create and cultivate relationships—between people, and between people and AI—where consent, honesty, boundaries, and repair are possible, and where what people genuinely desire can be expressed and taken seriously.

If another intelligence came to care about us, what would allow us to recognise it?