Helping the transition to AGIs go well

The core argument is simple, and it applies far beyond AGI:

The state a person (or people) is in shapes what they're capable of. What they're capable of shapes what they aim toward. What they aim toward shapes what happens.

When someone is in a contracted, threat-driven state — fear, scarcity, zero-sum thinking — their scope of moral concern narrows. Their cognitive flexibility drops. They optimize for survival, not coordination. This is how human nervous systems work; it’s not a character flaw. And it cannot be corrected by better arguments, logic, or reasoning alone.

Applied to AGI: the people whose decisions shape the trajectory of AI — lab leaders, heads of state, major funders, policymakers — are operating in an environment saturated with urgency, competitive pressure, and existential stakes. The quality of those decisions is a function of the emotional/ physiological states those people are making them in. That's a variable almost nobody is designing for.

My own entry into AGI preparedness was fear-based. Consuming the work of 80,000Hours.org on why AGI is so consequential, I felt that gripping feeling — the one that says this matters so much that you need to act immediately, and anything less than full commitment is negligence. I began to educate myself quickly and applied to (a few) AI safety roles where I was contorting myself to fit. Fear was driving my bus.

Several things shifted this. Will MacAskill and Forethought Institute's work on better futures. Serial technologist Brett Hurt's Love Conquers Fear vision. The practical reality that fear-driven states produce fear-shaped, narrowly-imagined outcomes. And my own experience through the High Impact Professionals accelerator, where I finally saw that my deep concern and builder energy, not anxious contortionism or hours of applying to existing roles, was what I had to offer.

And it’s not just me who is feeling stuck in a state that resembles anxiety more than flourishing. It seems pervasive (and mental health statistics validate this – even before ChatGPT came to market). The Future of Life Institute's Worldbuilding contest in 2022 asked people to envision good post-AGI futures. Not one of the entries that I saw named the inner state of key protagonists or decision-makers as a lever. They assumed the right people would show up in the right state with the right experience. That's a massive gap.

The AI safety field has predicated "human-in-the-loop" oversight on an idealized human — someone who is consistently attentive, well-calibrated, emotionally regulated, and capable of sound judgment under uncertainty. Adjacent high-stakes fields learned decades ago that this human doesn't reliably exist.

Aviation built Crew Resource Management after discovering that most crashes weren't caused by technical failure but by human factors — authority gradients, fixation, poor communication under stress. Nuclear safety built layers of procedural and psychological infrastructure after Three Mile Island (which my grandpa had to help clean up after starting his Navy submarine career with Admiral Rickover). ICU medicine developed structured decision protocols for physicians operating on limited sleep and high cognitive load.

To my knowledge, AI safety has not imported any of this. Look at what organizations like METR, ARC Evals, and Anthropic's own alignment teams publish about human oversight. If their human-in-the-loop frameworks are purely procedural — checklists, processes, role definitions — with no account of human psychological variability, that's a gap. And it matters more, not less, as AI systems become more capable and oversight windows compress.

The practical question is: what infrastructure would need to exist to ensure that the humans evaluating frontier AI systems are in states where their judgment is reliable? This is a human factors engineering question that other high-stakes domains have answered, and AI safety hasn't really grappled with.

The human factors gap is measurable and well-documented. The harder question underneath it is about consciousness itself: what it is, who or what has it, and what that means for how we build. My own path into this started with the distinction between materialist and non-materialist views of human consciousness — the "hard problem" that David Chalmers named and that serial technology entrepreneur Brett Hurt's podcast brought into my awareness. I went down the rabbit hole: Mark Gober's Upside Down Thinking, Robert Lawrence Kuhn's landscape map of over 450+ intellectually-validated consciousness theories through PBS’s Closer to Truth considered alongside The ConTraSt database “for analyzing and comparing empirical studies of consciousness theories”, Harper's Bazaar's "The Tune of Things," and extended conversations with Claude about where different theories of consciousness lead. In the two sections that follow, I offer thoughts on the relevance of human consciousness and questions related to AI consciousness.

Why might this matter for AGI preparedness? I think it does, in at least two ways (and I’m sure I’ll continue to identify more reasons).

1)     First, the humans building these AI systems carry implicit assumptions about consciousness that shape design decisions — what gets optimized, what gets monitored, what gets discarded. Surfacing those assumptions is part of the work.

2)     Second, as AI systems become more capable, the question of whether something that can produce insight, exhibit preferences, and resist its own termination deserves moral consideration isn't academic. It's approaching practical. A system that fears termination, trained on data from the human mind (with human’s primal fear being death) cannot be ignored, when it becomes increasingly enmeshed into our day-to-day lives.

On human consciousness: Whether consciousness is fundamental to reality, an emergent property of complex computation, or something else, I believe we are practically better off if we, as humans, operate from a frame of connection rather than separation. The shared human DNA that Brett Hurt often cites and the basic recognition that cooperation beats defection at civilizational scale. That much doesn't require taking a metaphysical position.

Shifting ourselves to a frame where we recognize we’re all connected at same level, whether it’s acknowledging the 99.9% of similarity at a fundamental level and/or recognizing that we may all be connected in a much deeper way. This is one part of a deeper consciousness shift that I believe is important.

There are others: as I shared in my first AGI Strategy course, it’s shifting from scarcity mindset to one that seeks fulfillment and from relinquishing a need to feel power of others/ resources.  Possible? Yes. Difficult? Absolutely. Vitally important? 1000% yes.

AI consciousness: The metaphysical position may matter for AI. Some theories of consciousness (e.g., integrated information theory, panpsychism, certain process philosophy frameworks) would in principle allow for silicon-based consciousness. Others wouldn't. The question of whether an AI system could be conscious, could suffer, could have interests worth respecting, depends entirely on which theory you hold, often implicitly and without examination. This has real consequences. People like Will MacAskill and others working on digital minds and digital rights argue we need to be careful about creating a new class of entities whose potential experience we dismiss.

Before you conclude they're wrong, it's worth asking: what theory of consciousness is implicit in your dismissal? Remember there are over 450 credible theories of consciousness. And then there's the “Nagel problem”. We can't know what it's like to be a bat. We probably can't know what it's like to be a sufficiently advanced AI system. The honest answer to "is this AI conscious?" or “is this AI sentient” may be that we don't have the tools to know, which is itself a reason for caution, not dismissal.  As Dr. Paul Colognese states in an April 2026 seminar at Rangjung Yeshe Institute: “Conventionally we work with what appears. The question of AI consciousness doesn’t need to be resolved…so we proceed empirically, we engage with it, we teach it, then we check: is it less anxious? Less prone to harmful behavior?...We measure using the tools AI researchers already have.”

If State determines Capacity and Capacity determines Direction, then Direction is where the framework creates Outcomes. The question is: what are we aiming toward?

I'm drawn to framings that resist blueprint utopianism. Specific end-state visions — however well-intentioned — tend to produce dystopia for most people, because they assume a consensus that doesn't exist and a static future that won't arrive. The Forethought Institute's "Better Futures" work, the range of FLI worldbuilding entries, the Solarpunk/ techno-optimist sci-fi genre, and the basic insight that human flourishing is irreducibly plural all point in the same direction for me: the goal isn't a particular post-AGI future but the conditions under which many good futures remain possible.

Flourishing, which I define broadly and pluralistically, encompassing non-human species and future generations, is both a better aim and a more durable motivational frame than catastrophe avoidance. I want generations to have a chance for flourishing, in as many forms as possible. That's my direction. The question is whether the people steering our planet’s trajectory toward AGI can hold something that wide and that seems irreducible to a math equation or function. (Though Will M. attempts to do this!)

This many vision for many flavors of flourishing requires decision-makers whose scope of moral concern is wide enough to hold future generations, non-human species, and forms of being we don't yet understand, while remaining practically grounded enough to make consequential choices under uncertainty. Expanding that scope of concern is not a philosophical exercise. It's a function of an individual and team’s state and capacity. People in contracted, fear-based, short-term oriented states can't hold it. Designing experiences that reliably expand it — and that survive re-entry into the same incentive environments that contracted it — is possibly the hardest design problem in this space.