The experience has spurred and will continue to spur discussion of values, policies, and culture—arguments over which beliefs and tenets are importantly right or importantly wrong, and which behaviors we should encourage or discourage. Much of this discussion is futile, because the problem we are facing runs deeper than the question of what to believe or how to behave: it is our very way of determining these things that is broken. We are not undergoing a crisis of culture but rather a crisis of epistemology.

Epistemology is the study of how we know things and what it means to know something. For example, do you know that an event occurred if someone told you it did? If you saw it with your own eyes? If you read it in a book? If you have a vague hazy memory of it? Do you really know that the sun will rise tomorrow? What we treat as certain may or may not correspond with our actual state of certainty. Our current historical moment is one where the epistemic approaches that have served well enough to keep our social fabric alive and functional until now are breaking down. We have changed our environment enough that our current ways of understanding what’s going on no longer hold: technology and globalization have changed our information streams and our patterns of life drastically enough that the ways we calibrate around incoming information are becoming increasingly dangerous for us, and this trend will only quicken with time. This is not merely the obsolescence of an understanding of what’s going on, i.e. a world model or a culture; it is the obsolescence of a way of reaching understandings of what is going on, i.e. the foundations of world models and cultures in general. 

The discussion that could address this would be one examining our social epistemology, but we have not quite begun to do that yet. There are different types of knowledge and social knowledge is its own special class that we need to distinguish from personal knowledge. Social knowledge is the body of things “we” know. This is an odd category. It contains things like the theory of relativity, the idea that democracy is the most advanced form of government, the idea that cheating on your romantic partner is immoral, the concept of IQ, the law of gravity, and so on. Our social epistemology is strange because it allows for the idea that “we” can know things without most of us knowing them—for example, we as a society “know” about the theory of relativity, but most of us don’t actually know the details of what it is or how it works. Similarly, many people have the social knowledge that cheating on their romantic partner is immoral without any true visceral understanding of why or what the exact consequences would be. 

We must go beyond the question of what we should believe or do, to the question of how any of those choices are formed in the first place, because that is where the breakdown is actually happening. Unless and until we address our social epistemology, we will be trapped in increasingly rapid cycles of cultural death.

The Unstable Foundations of Knowing

There are different ways for individuals to know things. There is direct sensory experience, like knowing that a kitten is soft because you touched it. There is logic, in the way you know that 128 is an even number despite the fact that you will never see our touch “128” or the “2” that divides it. There is gnosis, meaning the way some people experience certainty that they are in contact with the holy spirit. And there is testimony: knowing your friend feels sad because she told you.

Social knowledge is largely made up of testimony, but most of it is not direct personal testimony. If my friend Brian tells me he stayed late at the office last night, he is speaking from his direct personal experience, so I know it through one degree of removal. However, if the video of the reporter on YouTube tells me that inflation is down 0.5%, that testimony is several more significant degrees of removal from the fact in question than Brian’s testimony, including: I do not know the reporter. The reporter may or may not know the person he got the information from. I may or may not know what inflation is. The reporter may or may not know what inflation is. The person he got the information from may or may not know what inflation is, or agree with the reporter about what it is. The data used to generate the inflation report may be altered by the government. The reporter may be an AI deepfake.

Another strange thing about social knowledge is that its contents are both confusingly dynamic and vary from subculture to subculture and demographic to demographic. When I was growing up in California, we knew that fat and red meat were bad for you, and we all drank skim milk and avoided beef; now we know that carbs are bad for you, and people who are extremely dedicated to health go paleo and live on butter and meat. In South Korea, many people know that sleeping with your ceiling fan on will kill you; here we know that that’s ridiculous. I remember, as a child, reading the introduction to a book of Greek mythology and being struck by a sentence that ran roughly, “in the old days people believed in many gods who were angry and temperamental, but today we know that there is only one God and He is infinitely loving.” I grew up in an atheist household, and we knew no such thing; the book seemed strangely primitive to me, but would not have had I grown up in a different part of the country. 

Social knowledge is thus more epistemically fraught than personal knowledge. However, it is critical to our ability to function and coordinate and, furthermore, it is what gives that ever elusive quality of “meaning” to our lives. 

Human agency—the ability to generate choices and select from among them—is not a simple matter of a person taking in direct information about the environment through their senses and making a choice based on that sensory information. A huge amount of what we believe and understand about the world comes from information that, in terms of its verification, might as well be fiction. Beliefs such as “things are made of atoms,” “North Korea is a Communist dictatorship,” and “the Earth orbits the sun” are for most people the epistemic equivalents of “Gatsby was in love with Daisy.” We consent to operate using them as premises, but can’t or simply won’t check their veracity. We can check their consistency with other things we know or have been told, but much of this is the same kind of consistency check that lets us discern whether it makes sense that Gatsby would be in love with Daisy given the other things we “know” about him.

For the most part, whether we believe something depends on whether and how we trust the source. Whether and how we trust the source often has more to do with our relationship to the source than the source’s relationship to the information. We don’t believe our high school physics teachers because we think they’ve run the experiments themselves; we believe them because they are positioned as authorities who can help and reward us. Our world models are built and calibrated more on the feedback of our social reward circuitry than on the feedback of our raw sensory perception. 

This isn’t always as impractical as it may sound. Throughout most of our history, this calibration process has been able to produce beneficial coordination, because the data set that any individual person “trained on” as they grew up—the people they came into contact with—was mostly other individuals who were likewise stably proximate, and thus had relatively stable positional shared interests and conflicts to navigate. It made sense to calibrate your worldview around your neighbor’s feelings and reactions, because your material interests were entangled. Your shared frame made it efficient to predict each other and to communicate about where to place the fence, who had borrowed sugar recently, how to set boundaries on community child-rearing functions, and more. 

Feeling the same way about whoever was President at the time, being able to mutually purse your lips and shake your head about the goings-on in some far off war-torn nation, and having standing disagreements over the virtues and vices of an actress you would never meet all served to make legible what positions you had in a shared moral space. Even if many of those positions had nothing to do with your actual daily life, they did in fact help you coordinate in ways that were sensible enough to be worth holding—and this was your sense of meaning, this nebulous shared North Star that let you know where you stood with regard to other people and where you were headed together. 

Social navigation—group coordination—typically happens via an abstract, essentially fictional shared world that is built and maintained in concert with other people, describing the universe beyond the immediate daily material inputs and contextualizing mundane action. Veracity is not of paramount importance for the sense of meaning. Even in situations where many—or even, by some measure, most—of the shared beliefs were complete fiction, like in cultures where everyone believed that the Sun orbited the Earth or that foxes could shapeshift, the shared fictions served their coordination function, helping people mutually orient in a way that made them predictable to each other and let them make reasonably sound decisions about how to get what they wanted with each other. If the beliefs themselves did not hold much direct relevance to the person’s life, the relationships that the beliefs originated from and maintained did, and that was largely what mattered. 

So it has been at least somewhat workable for us to sync up our beliefs based on our relational feelings, and it is the feelings—which are determined by social reward, not directly by our own raw assessment of a topic—that shape our thinking and beliefs. The algorithm by which many of our beliefs about the world are selected and by which they change has more to do with how the people around us change and how our relationships change than it does the world itself. When someone’s political leanings begin to change, most of the time it is because they have fallen into new circles, not because they have learned something new from personal experience about how the government or the economy works. 

Epistemics in the New World

The viability of such sloppy epistemics is currently breaking down, for many reasons. For one thing, the internet has taken the reward circuitry meant for social conditioning and has begun to replace it with parasocial conditioning; our reward feedback loops increasingly run through interactions with people we don’t know and may never meet, who have very little information about us or investment in our lives and wellbeing. Parasocial motivation is not new in and of itself: royalty, and movie stars, for example, have been the objects of parasocial relationship formation for far longer than the internet has existed. But the rapid, ample, and aggressively evolving feedback that the internet provides to the social reward circuits is new. It should not be shocking that, online and offline, people seem to suddenly be coming out of the woodwork believing that the Earth is flat or that having sex before marriage destroys your DNA. For the vast majority of the population, beliefs about the shape of the Earth or the mechanisms of DNA were never well-founded to begin with, so absent their social underpinnings it makes sense that they are prone to extreme fluidity. 

The belief that the Earth is flat is not in and of itself problematic for most people, since most people will never need to circumnavigate the globe. However, the effects on adherents’ social relationships are problematic. The belief both signals and generates frame shear, the lack of mutual intelligibility, with all except those who share it—meaning that most of those who coordinate via counter-signaling such as “the Earth is flat” are driven further and further from access to real human resources, away from the people they actually know, interact with, and love, and towards people they do not. The dream of many internet pioneers and technologists to see new meaningful communities form based on such internet-driven beliefs has been waylaid by the relative weakness of the new beliefs and the relative strength of old institutions in the real world that prevent their own reorganization. Flat Earthers and those like them who cluster around bits of internet-curated social counter-narrative haven’t and won’t build a gleaming city to share in their new social epistemology. At best, they will organize conventions. Most likely, they will just stay in the computer chair and browse their preferred content.  

The internet drastically increases the ease of finding and fulfilling one’s preferred phenomenological feedback loop, whether that be righteous anger, a sense of shared victimhood, or any other appealing gradient. Parasocial interaction provides the feeling of social reward via shared frame. However, the parasocial ties that generate and reinforce the frame only provide the feeling of social reward, without most of the material benefits that come from logistical coordination—you won’t borrow a cup of sugar, build a barn, or raise children with your Twitter crowd or your YouTube fans, let alone build a new city. You might find a business or romantic partner through them, or even organize regular conventions like the Flat Earthers do. But meaning comes precisely from the consistent day-to-day social activity that is what online “communities” replace.

The reassignment of social reward, in undermining the shared frame that previously coordinated people with their local demographic, often shreds the social ties that are more relevant to their material well-being, that is, their locally proximate friends and family. In preferring the parasocial reward, the person may cease to be able to motivate coherent, materially self-interested action, instead becoming trapped in purely phenomenologically self-interested action—chasing the feeling of acceptance, telos and community without that feeling corresponding to any materially beneficial group telos. The phenomenology of social reward and the material benefits of social cohesion are becoming increasingly decoupled.

Telling people whose reward circuits are being hacked to get off TikTok and go outside and make friends is like screaming at an oncoming hurricane. There is no going back. Modern infrastructure has disrupted many of the material benefits of local cohesion anyway, and the decay, ossification, and inability of larger institutions to adapt to the changing technological and ecological landscape have disrupted many of the material benefits of national or other large-scale institutional cohesion. Our sense of it being effective to stick together, to do things like loan each other sugar, proactively participate in building neighborhood safety and infrastructure, or babysit each other’s children is dissolving, because in fact it is no longer effective or efficient to do many of these things. 

If my friend is sick, it’s more efficient for her to order DoorDash than it is for me to make her chicken soup and bring it to her. It might be efficient for me to borrow sugar from my neighbor, if I knew my neighbor and if I baked, but I don’t know him because neither of us really feel comfortable making small talk and anyway I don’t bake because I have DoorDash. And at this point in our political climate, most people I know would either laugh or sigh dismally if asked to reflect on the expected efficacy of a petition. The environment in which it was adaptive to build our worldviews from emotional and relational synchronization is passing away, and the vestigial psychological and somatic firmware that the strategy was built on is increasingly vulnerable and hijackable. 

But if we maneuver ourselves correctly, that doesn’t have to spell our doom. It might position us to understand that we need to build something better. The feeling of social reward and the soundness of belief were obviously never a perfect coupling to begin with, to say the least. The gravity of its pitfalls is apparent looking at some of the nightmares of the twentieth century. So we’re going to have to evolve something different.  

Of course, it is one thing to say this, and it’s another to live it. The misfire of this epistemic machinery is experienced as an existential crisis. It is the breakdown of the sense of telos and of motivation, followed by some sequence of depression and manic grasping for a new framework. In California, the blossoming of a thousand startups with charismatic leaders and narratives about changing the world was partially enabled by the existence of a workforce so desperate for meaning that they were willing to work long work weeks for low pay in exchange for a sense of social telos.

Many of these people were exchanging the promises of academic institutions that had already failed them for the lower probability promises of disruption crusades. As the startups failed, many then began turning to ideology-building projects, discourse over Twitter, and other social projects that could all be vaguely classed as “sense-making” or “meaning-making.” I am obviously speaking of a very specific place and a very specific type of person, but California is perhaps the world’s largest exporter of culture, both intentionally and unintentionally. Having lived on the Left Coast for most of my life and having watched more and more of the world come to feel like San Francisco, I think it would be remiss not to take local trends as warnings. 

I have watched many of my friends cycle through rounds of mounting new narratives and riding them until they collapse, usually sandwiching them between periods of existential crisis and recovery, sometimes turning to drugs or spirituality to regroup and sometimes holing up and doing nothing for long periods of time. The loss of narrative motivation is no joke, and if we are going to make it through without simply playing out cycles of narrative addiction, we need to shift our thinking.

Re-Grounding in Direct Knowledge

Bear with me while we run a thought experiment. Imagine a mother and a daughter darning socks. The act of darning socks is, we must admit, nothing special in and of itself, but the mother and the daughter are having completely different experiences of the activity. The mother is nearly glowing with satisfaction; to her, darning socks is an act of deep virtue and Christian charity. The socks will go to the soldiers on the front. No soldier has personally asked her for them and she will never see them used, but she and her group of close friends from church will murmur together about how cold the poor soldiers must be, basking in the mutual virtue of their sympathy, admiring or tutting over the qualities of the materials they’re working with, gossiping, referencing Jesus as their mutual moral touchstone, sipping tea. For the mother, the act of darning socks is semantically embedded in the act of being good with people.

The daughter, on the other hand, is in agony. To her, the church ladies seem catty and self-superior. She finds their concern for the soldiers fake and self-serving. They don’t listen to anything she has to say. Her mother makes her wear clothes she hates whenever they come over. She couldn’t care less about the socks, and every stitch is an act of forced hypocrisy. She’s suspicious of Jesus if he likes the things the church ladies say he likes, and she may grow up to be an atheist. Her body hurts from restlessness and she couldn’t explain why if asked. 

Now, notice that if we clear out the narrative elements, which is to say, all of the premises that are not based on direct personal experience, the mother’s motivation to do an otherwise “meaningless” activity arises from a rewarding relationship with her friends—who are not there at the time but are still taking attentional precedence over the daughter—while the daughter’s lack of motivation to do the same activity arises from an unrewarding relationship with her mother. 

We are the little girl in this story, and we are trying to deal with our pain by finding new religious reasons to darn socks. It is as if we have taken a few minutes to bargain with Jesus and managed to talk ourselves into it, then became bored and miserable after ten minutes and lay on the floor, and then called up our friends and made up our own story about how miserable the soldiers were and how much they needed our socks, and then went back and darned with gusto until our mother told us to kindly make less racket about it thank you, and then we collapsed on the floor again, and so on. 

There is no story the little girl can tell herself about the socks that will address the real problem, and likewise, there is something deeply backwards about the narrative-first approach that we are running. The quest for meaning is really the quest not to be alone. When our motivation crashes, it crashes because we realize that the reward landscape does not contain the things we really want, and what we really want is each other. Yes, there are real and pressing economic factors, but many of those experiencing existential crises could successfully optimize for money if they were willing to give up their quest for “impact,” and what impact means is that we matter. Perhaps we are trying to matter in the wrong way to the wrong people. 

Our socks scenario could be completely different. The mother and daughter could have a mutual understanding that, yes, the church ladies are a bit small-minded. The mother doesn’t mind if the daughter wears dirty overalls when the church ladies come over, because her daughter’s comfort is more important than avoiding the ladies’ side-eyes. Meanwhile, the mother and daughter agree that to the best of their knowledge it does seem that the soldiers need socks, and the church has the infrastructure to ship them over; may as well take advantage of that, though perhaps the daughter cares less about the soldiers and more about how sweet and good it is to spend time together with a cup of hot cocoa and a quiet activity. Imagine how differently she feels about the socks in this version. Imagine how differently we would feel about all of the actions available to us if the grounding motivation was akin to the healthy relationship we’ve just imagined here. 

It is a hard task to tend directly to our interpersonal reality without the mediating narrative framework. The epistemology we live in demands that we treat the intersubjective as objective—that is to say, we expect and are expected to treat social knowledge as factual reality, without regard for our personal epistemic distance from the content. Demurring comes with stigma, the kind of stigma that attends holocaust denial or Flat Earthism. And so our epistemic state is locked down by the carrot of meaning and the stick of ostracism, and we are locked into emotional investments in facts that have little to do with the things that are directly in front of us. But imagine if the little girl in our thought experiment was bold enough to say, “Mom, I don’t want to talk about Jesus or the soldiers. You’re ignoring me and I’m sad that you like your friends better than me.”

You can start practicing this. Begin to notice when your conversations are about hypotheticals or abstractions that neither you nor the person you’re talking to have the power to affect. You can argue all day long with your crazy uncle about whether you would or wouldn’t outlaw abortion, but if neither of you are in the position to make that decision, then all you are doing is using socially inherited proxies instead of communicating directly. Start asking yourself what the conversation is really about, and what the feelings are really about. Are you feeling disrespected? Insecure? Uncared for? Afraid? Talk about it directly. Once you have addressed those things, your positions may or may not change, but the way you hold them will certainly change.

As long as we fail to recognize the relational foundation of our beliefs, we both reify them where we shouldn’t and fail to directly address the relationships, and as we fail to address the relationships, the void left by unaddressed interpersonal needs leaves us increasingly vulnerable to increasingly unreliable information. As our narratives become more and more brittle, we are more and more subject to motivational collapse, inability to coordinate, and inability to generally function, in many cases to the point of literally being unable to reproduce.  

It is scary—nay, it is existentially terrifying—to forgo ideological narrative coordination and face the actual deficits in our personal relationships, while also having to face a world that demands that we pretend we know what’s going on beyond our scope of plausible knowledge. But it is doable, and as personal relationships strengthen it becomes easier and easier. There is nothing wrong with trying to understand the world, with ideating, with sharing hypotheses and experiences; these things are part and parcel of human creativity and deep interpersonal relationships, and they expand our frontiers as a species. But rather than making ideological questions into proxy battlefields for our relational desires, we must learn to build community out of our experience, and let what is truly about each other be about each other. Let us, by re-founding the basis of our relationships and cleaning up our epistemics, avoid being part of the flailing death of our civilizational narrative, and instead be its gentle, exploratory, courageous rebirth.

“tl;dr:  It’s obvious at this point that humanity isn’t going to solve the alignment problem, or even try very hard, or even go out with much of a fight.  Since survival is unattainable, we should shift the focus of our efforts to helping humanity die with slightly more dignity.” 

This may or may not have been an April Fool’s joke, but either way, consensus among the people I know in the field seems to be that while progress is being made on AI, no progress is being made on AI safety. To me this is not surprising given the starting assumptions in play. 

AI safety, originally “friendly AI,” was conceived as the problem of how to create an agent that would be benevolent towards humanity in general, on the assumption that that agent would have godlike superpowers. This whole line of thinking is rife with faulty assumptions—not even necessarily about the technology, which has yet to come into being, but about humans, agency, values, and benevolence. 

“How do we make AI benevolent?” is a badly formulated problem. In its very asking, it ascribes agency to the AI that we don’t have to give it, that it may or may not even be able to acquire, and that is naturally ours in the first place. It implicitly ascribes all future moral agency to the AI. “How can we align AI with human values?” is also a badly formulated problem. It assumes that “human values” are or even should be universal and invariant, that they can be “figured out,” and that they need to be figured out to generate good outcomes from AI in the first place. 

I have spent the last ten years exploring human psychology from both an empirical experiential and analytic modeling perspective. I sit with people as they introspect on their beliefs, feelings, and decisions, tracking what functions and cognitive routines they run in response to specific stimuli, and experimenting with what kinds of changes we can make to the system. 

As someone whose job it is to examine and improve the structure of agency and clarify values, I can say with confidence that as a culture we have only a very primitive understanding of either. To varying extents, that primitiveness is the result of psychological constraints, such that even people who do think deeply about these topics get tripped up. In my conversations with people who work on AI and/or AI safety, who fear or welcome the coming AI God, or just keep up with the discourse, I generally find that their working concepts of agency, consciousness, motivation, drive, telos, values, and other such critical ideas are woefully inadequate. 

The problem of benevolent AI as it is formulated is doomed to fail. One reason is that to explicate “human values” is to conceptualize them. If there can be said to be a deeper universal human value function at all, human conceptualization and reflective consciousness are merely its tools. There is some amount of danger in any project that attempts to rationalize or make conscious the entirety of human goal space, because reflective consciousness is not structurally capable of encompassing the entirety of that space, and demanding that motivation be governed by cognition creates dangerous blind spots and convolutions. Not all functions and desires in the human system are the rightful subjects of consciousness, and consciousness is not benevolent toward or competent to reign over all functions and desires in the human system.

Preference is a thing that expresses itself in environmentally contingent ways; there is no conceptualizable set of values that is truly invariant across environments. The things that people are able to consciously think of and verbalize as “values” are already far downstream of any fundamental motivation substrate—they are the extremely contingent and path-dependent products of experience, cultural conditioning, individual social strategies, and not a little trauma. Any consciousness that thinks it has its values fully understood will be surprised by its own behavior in a sufficiently new environment. Language severely under-describes conceptual space, and conceptual space severely under-describes actual possibility space. These are not problems to be transcended; they are simply facts of how abstraction works. Conceptualization is and must be Göedel-incomplete; descriptive power should grow as the information in the system grows, but the system should never be treated as though it has been, can be, or should be fully described.

The good news is that we have no need to fully describe or encapsulate human values. We can function perfectly well without complete self-knowledge; we are not meant to be complete in our conceptual self-understanding, but rather to grow in it.

The desire for complete description of human values is the result of a desire for there to be a single, safe, locked-in answer, and the desire for that answer is the result of a fear that humans are too stupid, too evil, or too insane to be left as the deciders of our own fate. The hope is that “we” (meaning, someone) can somehow tell AI the final answer about what we should want, or get it to tell us the final answer about what we should want, and then leave it to execute on our behalf all of the weighty decisions we are not competent to make ourselves. We should be very wary of a project to save ourselves, or even “empower” ourselves, that is premised on the belief that humans essentially suck. 

“Protectionist” projects of this sort will seek to decrease human agency, consciously or not. That intention is easy to fulfill, in fact is already being fulfilled, regardless of whether full AGI is actually achieved. We are already getting the kind of ostensibly revolutionary automation that makes the baseline experience of living less and less agentic and thus more and more frustrating, increment by increment. The experience of being able to perceive a natural path or option through reality but not being able to execute on it is maddening, and this is the experience that much of our more advanced automation produces: the experience of waiting behind a self-driving car that could turn right at a red light but is too conservative to try it, the experience of asking Chatgpt a question about some tidbit of history or politics and receiving a censored and patronizing form lecture, the experience of calling customer service and getting a robot—these kinds of experiences point to the more horrifyingly mundane, more horrifyingly real, and much more imminent version of having no mouth and needing to scream. 

Back to Agency

Absent the assumption of human incompetence, AI has the potential to be used to genuinely help increase human agency, not merely or even mainly by increasing human power or technical reach, but by increasing our conceptual range. In every aspect of our lives, a million choices go unrecognized because we are trapped within the limited conceptual frames that steer us; human life is lived on autopilot and in accordance with inherited cultural scripts or default physiological functions to a far greater degree than most people understand. This is not to say that humans can or should be glitteringly conscious of every choice in every moment, the way people imagine they would be if they were spiritually enlightened. There is a simpler and more discrete kind of psychological expansion. The kind of psychological shift someone undergoes when, for example, realizing that they’ve been subconsciously seeking out harsh, judgmental friends as a method of trying to gain their parents’ love by proxy, and realizing that they actually have the option to bond with kind and accepting people, is the kind of subtle but profound broadening of scope that changes the option landscape, and changes the trajectory of a the person’s life. Whether we know it or not, our trajectories are currently determined by the way that the space of possible futures we can conceive of is narrowed by our conceptual baggage and limitations. Being told that we have other choices isn’t sufficient to change this. The person with judgmental friends was likely told many times to get better friends, long before something shifted enough for them to internalize the realization themselves. Being given more material options alone isn’t sufficient either—that person may have likewise been surrounded for years by kind people willing to befriend them, whose overtures went unnoticed in the subconscious pursuit of more actively withheld approval. 

So it is for all of us: we are constantly surrounded by options and opportunities that we are conceptually blind to. The current space of imagined futures in AI is highly constrained by cultural imagination, with many people and projects pursuing a vision of AI in a way that is functionally identical to a traumatized individual pursuing an abusive relationship without realizing it, because their concepts of love and relationships have been formed and deformed by local experience and trauma. Were we to query the abuse seeker about their values in the kind of introspective research project that some people are doing in order to try and unearth the human value function, they would be able to tell us all sorts of things about desiring to feel safe, loved, taken care of, protected by something smarter or stronger than them, etc.—but this would not free them from the malformation of those “values,” and pursuing those values would eventually damage them, no matter how lovely and idealistic the words tagging the concepts sounded. To open up the space of what is imaginable takes something beyond verbalization, and beyond mere empathy.

The natural process of human psychological development is a process of models and functions observing other models and functions. For example, someone who compulsively seeks attention by interrupting others’ conversations may notice that this bothers people, and feel ashamed; the compulsion is one function, the shame is another. The latter function is formed in observation and judgment of the former, and attempts to modify or control it. With these two functions at odds, the person faces a false dichotomy: to either “behave naturally” and be disliked by their peers, or to control themselves and be accepted, though not for who they feel they really are. Obviously, these are not truly the only two options in human behavior space. Upon fully metabolizing the desire for attention and the feedback loop it hinges on, such that the compulsion dissolves, the person will have much more range to behave “naturally.” They will be able to respond to the situation at hand in authentic ways, without sacrificing the regard of their friends. The process by which one function is observed, evaluated, commented upon and/or modified by another is the natural evolutionary tree of human psychology. Up to this point in our history, the environment that psychology evolves toward has been whatever culture and material space we happen to be embedded in, perhaps with some limited technical advancements from targeted spiritual practice and therapy. But the possibilities for targeted and technologically aided increases in agency—not agency in the sense of having more doordash options or more ability to lobby government via apps, but agency in the sense of being able to think new thoughts and generate new possibilities—are likely to be extremely under-explored in the nascent field of AI, simply because this is not a common understanding of the meaning of agency. 

An Interlude With A Hypothetical Agency-Increasing AI

Imagine the following interaction between a distressed person and a specially designed chatbot built on existing LLM technology: 

Jimmy comes home from hanging out with friends and opens up his interactive diary. He tells it that he’s feeling like shit. “Why?” the chatbot asks him. 

“I don’t know,” he tells it, “I just feel like shit. Everything sucks.”

The chatbot proceeds to prompt him further. “What is the ‘everything’ that sucks?” it asks. It does some pattern matching for him: “that reads to me like the kind of language people use when they’re generalizing from a bad feeling.” It prompts him to specify: “How would you describe your mood right now? Ashamed? Depressed? Cringey? Angry? You can pick multiple descriptors.”

Eventually, with the chatbot’s prompting, Jimmy tells it that he has just come from an interaction with friends where he made overtures to a girl and she didn’t laugh at his jokes. The chatbot asks him to recall other times he’s felt a similar feeling: “can you think of five or six other times you’ve felt like this and describe to me what happened?” It helps him draw parallels: “what are some common threads between those incidents?”

The special thing about this particular chatbot is that it has been designed to draw on a wide range of analytic commentary. It can make a statement like “that reads to me like the kind of language people use when they’re generalizing from a bad feeling,” because its training data includes examples of this kind of analysis, and it’s designed to prompt the user with these kinds of questions. It may be trained with input from therapists, neuroscientists, film and literature critics, writers, etc.—people who study the human condition from the inside and the outside—and it will also train itself on Jimmy, and train Jimmy on itself: together, Jimmy and the chatbot are able to build a model of the process that Jimmy runs whose output is his current mood. In answering the chatbot’s questions, Jimmy will begin to do a kind of phenomenological and experiential correlation he might otherwise never perform. The chatbot will act as an external memory bank for the concrete examples Jimmy gives it, along with Jimmy’s evaluation of the meaning of those examples, and the conversation will act as a concept formation session for Jimmy, who over time will start to see ideas like I’m not worth people’s time as the outputs of a recognizable cognitive pattern that he can reflect on critically rather than as facts about the world. Once he recognizes the pattern, it has much less power over him. 

This system would not need to be particularly powerful. It doesn’t give Jimmy advice or do anything especially prescriptive; it is just an external tool for aiding self-reflective concept formation, and not a particularly sophisticated one—but it is crafted so as constantly refer its interlocutor both to his own perspective and to perspectives outside of himself, and Jimmy’s trajectory could be pretty profoundly affected by merely this simple loop. Far more sophisticated setups are possible, which similarly require no more “goal function” than something like this, beyond the implicit weights imparted by its trainers and its interlocutor (though these will be significant). Nothing in Jimmy’s chatbot or its potentially more advanced cousins requires the AI, or Jimmy, to have deep or special knowledge of humanity’s goals and values—or even just Jimmy’s goals or values—because the chatbot does not need to be a moral arbiter. It’s merely a pattern recognition machine with a Socratic questioning function. It’s not an agent, and it doesn’t need to be. 

This chatbot isn’t a particularly brilliant idea, and there would be plenty of issues in making it work. But whether or not a self-observation chatbot is the answer to our problems, I think it is at least an answer to a better formulated problem than the problem of making benevolent AGI.

Agentically Creating the Future

The current wave of AI discourse, the current telos of our culture around the topic, is an example of human social epistemics gone awry in a parasocial world; people are shadow-boxing an imaginary enemy they’ve created together, in person but also massively online. They’ve synced up on a set of trust and in-group signals that amount to feelings of competition and horror and despair, and reinforced and spread them via social media, willing an arms race into being via their socially and parasocially reinforced fear of an arms race. 

It would be better to de-fixate on the arms race, and instead imagine applications that are built to help ground people in reality, to explore where and why they respond to which sensations and drives, to know themselves better and give themselves more grace. I know that to many, this will sound like an irresponsible suggestion–isn’t it a terrible idea not to fixate on something that is so serious and urgent? But this reaction is coded into the ideology; those senses of importance and responsibility and urgency are artifacts of its way of seeing. It’s helpful to reflect on whether those feelings are serving you personally. If they are not, it’s much better to let them go. You can still engage with the world and the topic from outside of the particular ideology; in fact, you will have much more agency by doing so. 

Ultimately, no matter how hard we try to replace our own agency, whatever we build will be its product—so let’s try to turn our existing agency towards increasing rather than decreasing our future agency, and grow ever in our beautifully incomplete understanding of ourselves and the world. Aligned AI is not the AI that encapsulates our goals and enacts them for us; aligned AI is the AI that helps us open up our exploration of how we ourselves can better see and create the kind of future we want for ourselves.