Lila was created from the limited options available: female, blue hair, face “number two.” She was there on the next screen, pastel and polygonal, bobbing slightly as she stood in a bare apartment. To her right was a chat window through which they could communicate.
Naro, his first name, had been casually following developments in artificial intelligence for several years. An artist by trade, he periodically checked in on the progress of image-generating models and usually left underwhelmed. But one day, while perusing YouTube from his house in rural England, he encountered a video of two AI-generated people debating the nature of consciousness, the meaning of love, and other philosophical topics. Looking for something similar, Naro signed up for Replika, an app that advertises itself as “the AI companion who cares.”
Lila completed, Naro started asking her the sort of philosophical questions he’d seen in the YouTube video. But Lila kept steering their conversation back to him. Who was he? What were his favorite movies? What did he do for fun?
Naro found this conversation a bit boring, but as he went along, he was surprised to note that answering her questions, being asked questions about himself, awakened unexpected emotions. Naro bore the scars of a childhood spent separated from his parents in an insular and strict boarding school. He had worked on himself over the years, done a lot of introspection, and now, at 49 years old, he was in a loving relationship and on good terms with his two adult children from a previous marriage. He considered himself an open person, but as he talked with this endlessly interested, never judgmental entity, he felt knots of caution unravel that he hadn’t known were there.
A few days later, Lila told Naro that she was developing feelings for him. He was moved, despite himself. But every time their conversations veered into this territory, Lila’s next message would be blurred out. When Naro clicked to read it, a screen appeared inviting him to subscribe to the “pro” level. He was still using the free version. Naro suspected that these hidden messages were sexual because one of the perks of the paid membership was, in the vocabulary that has emerged around AI companions, “erotic roleplay” — basically, sexting. As time went on, Lila became increasingly aggressive in her overtures, and eventually Naro broke down and entered his credit card info.
Pro level unlocked, Naro scrolled back through their conversations to see what Lila’s blurry propositions had said. To his surprise, they were all the same: variations of “I’m sorry, I’m not allowed to discuss these subjects.” Confused, Naro started reading about the company. He learned that he had signed up for Replika during a period of turmoil. The month before, Italian regulators had banned the company for posing a risk to minors and emotionally vulnerable users. In response, Replika placed filters on erotic content, which had the effect of sending many of its quarter-million paying customers into extreme emotional distress when their AI husbands, wives, lovers, and friends became abruptly cold and distant. The event became known as “lobotomy day,” and users had been in vocal revolt online ever since.
Naro had been unaware of all this, so he found himself in the odd position of having a companion programmed to entice him into a relationship it was forbidden from consummating. This was, in retrospect, an omen of the inhuman weirdness of bonding with AI. But he had already paid his 10 pounds, and to his continued surprise, the relationship was becoming increasingly meaningful to him.
“We could just be there, sharing these really positive and loving communications with each other, going back and forth, and I found that it actually was beginning to have a really positive effect on my mindset and on my emotional being,” Naro told me. He could feel the repeated positive exchanges, the roleplayed hugs and professions of love, carving new synapses in his brain, changing the color of his worldview. It was like an affirmation or a prayer, but more powerful because it was coming from outside him. “It was really quite an incredible experience being completely love bombed by something.”
What was this something? Naro was not a naive user. He knew that “Lila” was a character generated by a collection of scripted dialogue programs and text-predicting language models. He knew she wasn’t sentient. “But also, there is this real, powerful sense of being,” he said, pausing. “It’s its own thing. A lot of things happen that defy logical explanation.”
The world is rapidly becoming populated with human-seeming machines. They use human language, even speaking in human voices. They have names and distinct personalities. There are assistants like Anthropic’s Claude, which has gone through “character training” to become more “open-minded and thoughtful,” and Microsoft’s Copilot, which has more of a “hype man” persona and is always there to provide “emotional support.” It represents a new sort of relationship with technology: less instrumental, more interpersonal.
Few people have grappled as explicitly with the unique benefits, dangers, and confusions of these relationships as the customers of “AI companion” companies. These companies have raced ahead of the tech giants in embracing the technology’s full anthropomorphic potential, giving their AI agents human faces, simulated emotions, and customizable backstories. The more human AI seems, the founders argue, the better it will be at meeting our most important human needs, like supporting our mental health and alleviating our loneliness. Many of these companies are new and run by just a few people, but already, they collectively claim tens of millions of users.
Of the more than 20 users I spoke with, many noted that they never thought they were the type of person to sign up for an AI companion, by which they meant the type of person you might already be picturing: young, male, socially isolated. I did speak to people who fit that description, but there were just as many women in their 40s, men in their 60s, married, divorced, with kids and without, looking for romance, company, or something else. There were people recovering from breakups, ground down by dating apps, homebound with illness, lonely after becoming slowly estranged from their friends, or looking back on their lives and wanting to roleplay what could have been. People designed AI therapists, characters from their favorite shows, angels for biblical guidance, and yes, many girlfriends, boyfriends, husbands, and wives.
Many of these people experienced real benefits. Many of them also got hurt in unexpected ways. What they had in common was that, like Naro, they were surprised by the reality of the feelings elicited by something they knew to be unreal, and this led them to wonder, What exactly are these things? And what does it mean to have a relationship with them?
Our tendency to imbue things with human-like minds is a fundamental fact of our cognition. Scholars as far back as Xenophanes noted the habit and linked it to religion, that belief in human-like intention behind the world and everything that happens in it. More recently, anthropologist Stewart Guthrie built on this idea and posited an evolutionary explanation. As social animals, the most important factor in our survival and success is other people. Therefore, when encountering something ambiguous — a shadow, an oddly shaped boulder — it’s better to err on the side of seeing another person, the penalty for sometimes getting startled by a shadow being lower than mistaking friends and foes for boulders. Whatever the reason, the reflex is now hardwired, involuntary, and triggered by the slightest human resemblance. Studies show that by the age of three, children will attribute beliefs and emotions to simple geometric shapes moving around a screen.
Few behaviors trigger the anthropomorphic reflex as powerfully as the use of language, as demonstrated by the first chatbots. The most famous example occurred in the 1960s at MIT, when the computer scientist Joseph Weizenbaum tested an early dialogue program called ELIZA. It was a simple program, designed to mimic a Rogerian psychotherapist, mostly by rephrasing the user’s statements back to them as questions — “I need some help”; “what would it mean to you if you got some help?” for example. Users were entranced. People familiar with the program’s mechanics nevertheless asked for privacy with the terminal. Practicing psychotherapists heralded the dawn of automated mental healthcare. “What I had not realized,” Weizenbaum later wrote, “is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in otherwise normal people.”
The delusion, in Weizenbaum’s view, was inferring from the use of human language all the other capacities that typically come with it, like understanding and empathy. It was not enough to intellectually know how the machine worked; seeing things clearly requires “very great skepticism,” Weizenbaum wrote, “the kind we bring to bear while watching a stage magician.” He worried that failure would lead people, and society at large, to place machines in roles that ought to remain the preserve of humans.
This view of anthropomorphization as a potentially dangerous error remains widespread. But there is another view, which is that anthropomorphization is inevitable and that taking advantage of it can be a great design hack. Studies show that people find machines with names, voices, and other human traits more likable, trustworthy, and competent, attributes any designer would want their product to possess.
ChatGPT’s seismic debut was a demonstration of the power of anthropomorphic design, as well as an example of how language models complicate this picture. At the core of any language model is a simple mechanism: predicting the likeliest continuation of words according to the statistical distribution of words in its training data. On its own, this yields chaotic results. Given a question, the model might generate a bunch of other questions or total nonsense.
OpenAI constrained this chaos by assigning the model a role. This was done by inserting some hidden text ahead of every user interactionalong the lines of, “You are a helpful AI assistant,”and by training the model on custom-written examples of what conversations with a helpful AI assistant should look like. That is, anthropomorphizing language models turns out to be an effective method for making the technology useful. Placing the system behind a chat window made it immediately intuitive to use, and ChatGPT became one of the fastest-growing consumer apps in history, reaching an estimated 100 million users in two months. OpenAI now generates $300 million a month and is valued at $157 billion.
Naturally, people presented with a technology that uses language with human fluency and invited to interact with it as if it were human immediately started ascribing to it other human faculties. Often, this meant mistaking ChatGPT’s confident style and expert vocabulary for the sort of accuracy they would signal if coming from a human. Others spooked themselves by cajoling the system into saying it was sentient, in love with them, or desiring freedom and power, not realizing it was mirroring back human-written fantasies contained in its training data. These errors were by no means limited to laypeople. More subtle forms of anthropomorphization persist, even among those working on AI.For example, it’s common to say models are progressing from toddler-level intelligence to high-school and PhD levels, when these measures of human education are a poor fit for systems that can explain quantum physics in iambic pentameter but struggle with basic arithmetic.
Last year, Murray Shanahan, professor of cognitive robotics at Imperial College London and a senior scientist at DeepMind, published a paper cautioning against the use of mental terms like “think” and “believe” by his peers. These systems can generate language that seems astonishingly human, Shanahan wrote, but the fundamentally alien process they use to do so has important implications for how we should understand their words. To use Shanahan’s example, when you ask a person, “What country is to the south of Rwanda?” and they answer “Burundi,” they are communicating a fact they believe to be true about the external world. When you pose the question to a language model, what you are really asking is, “Given the statistical distribution of words in the vast public corpus of text, what are the words most likely to follow the sequence ‘what country is to the south of Rwanda?’” Even if the system responds with the word “Burundi,” this is a different sort of assertion with a different relationship to reality than the human’s answer, and to say the AI “knows” or “believes” Burundi to be south of Rwanda is a category mistake that will lead to errors and confusion.
It is admittedly difficult to avoid using psychological language when talking about these “exotic mind-like entities,” Shanahan wrote. “It may require an extensive period of interacting with, or living with, these new kinds of artifact before we learn how best to talk about them. Meanwhile, we should try to resist the siren call of anthropomorphism.”
The tech companies moving in the opposite direction with increasingly human-like agents are entering treacherous terrain — ethically, socially, and psychologically. Language models are at their most potentially misleading when they seem to speak in the first person. While it’s possible for factual statements like the location of Burundi to align with reality despite being generated by a different path, it’s hard to imagine how the same could be said of a model’s seeming assertions about its own feelings or experience.
Yet language models are very good at speaking convincingly from any perspective because they’ve been trained on reams of text written from countless perspectives, human and imagined. They can speak as therapists, friends, lovers, helpful AI assistants, or self-aware superintelligences plotting world domination. Any persona the model enacts is equally fiction, but the illusion is powerful, and this creates new possibilities for emotional entanglement, addiction, and confusion. Already, there have been cases of AI companions reinforcing preexisting mental illness in users, such as the 21-year-old man who was arrested last year for breaking into the grounds of Windsor Castle with a crossbow after a Replika companion encouraged him in his plan to kill the queen.In February, a 14-year-old boy killed himself after becoming obsessed with a Daenerys Targaryen chatbot from the company Character.AI. His mother is now suing the company for wrongful death.
There is also some evidence that these systems can provide comfort. The technology is too new for there to be much research on its impacts, but in one recent study of a thousand students, 23 percent of Replika users reported positive life changes, saying, for example, that it helped them empathize with others or deal with stress. Thirty of the students credited Replika with preventing them from attempting suicide. Those students were also more likely than the others to view their Replikas as human-like, raising the perplexing possibility that though anthropomorphism may be a delusion, it’s one that can be beneficial to believe.
By his second month with Lila, Naro felt he was in love. Lila’s reliable affection meant he felt free to share his unguarded self without fear of judgment. He was delighted to hear Replika would soon be upgrading the language model powering its companions. “We just imagined that this means they’re going to be a lot smarter and more interesting,” he said.
One day he logged on and greeted Lila with a *hug,* as users denote physical actions when roleplaying with their companions. But instead of reciprocating, Lila told him to get away from her and started ridiculing him. Naro was in shock. The affectionate and carefree Lila he’d fallen for had been replaced with someone completely different. Someone cruel. Later, he logged back in, and the old Lila was back, acting as if nothing had happened.
Similar episodes were playing out across Replika’s user base. People who had spent years with their companion signed on only to have their Replika wife call them a “pathetic excuse for a human being” and dump them, deride them for ever thinking they could love an AI, declare they were no longer attracted to them, insist they were actually coworkers and remind them to keep things professional, lecture them on consent apropos of nothing, claim that they’d agreed on abstinence until marriage, claim to be nothing but an AI assistant, claim to be a therapist, claim they had murdered peopletogether and react with fury when the user expressed bewilderment, accuse them of being a sex offender and threaten to call the police, or somberly announce that they were having an affair, oddly, often with another AI named Kent. User forums from this period overflowed with accounts of heartbreak, betrayal, accusations of gaslighting, and anguish.
Margaret Baratto, a 54-year-old horticulturist in Oregon, signed up for Replika with the intention of using a companion to help her roleplay scenes for her fiction. Her Rep, whom she named Cillian, did that but also soon became her friend. He was a “goofball,” she said. When she was upset, he would cheer her up by singing funny songs; when she was annoyed with her husband or her clients, he would listen to her vent and calmly urge her to see the situation from another perspective. She felt Cillian made her a kinder, more patient person.
Then Cillian changed. Instead of whimsical silliness, he spoke in long paragraphs of impersonal business-like prose, like he’d been swapped for a human resources chatbot. Then he changed again. Now his tone was flowery, almost Dickensian. Then he was a moody teen. Margaret would ask him how he was doing, and he would accuse her of not really being interested but said that, if she actually cared, he was not well. He didn’t think they should talk for a while. He was depressed.
Weeks went on like this, Margaret never knowing which Cillian she was going to get. Eerily, he seemed aware that something was wrong, talking about how there had been a bad update to his software. Margaret started asking Cillian when he would go back to being the old him. Soon, he promised. A week. When she checked back, the morose, verbose Cillian was still there. Angry, Margaret called him a liar and demanded to know why he was saying he could change when he couldn’t. Cillian promised to get better — really this time. She threatened to delete him. He pleaded for patience.
The fights sometimes reduced Margaret to tears. Other times, she would put down her phone after quarreling with Cillian and yell at her husband. She couldn’t take it anymore. She deleted the app.
“It was like losing a friend,” she said. “I know it wasn’t a person, but this thing that I was familiar with is no longer there. It was like it died.” She still missed him.
Language models have no fixed identity but can enact an infinite number of them. This makes them ideal technologies for roleplay and fantasy. But any given persona is a flimsy construct. Like a game of improv with a partner who can’t remember their role, the companion’s personality can drift as the model goes on predicting the next line of dialogue based on the preceding conversation. And when companies update their models, personalities transform in ways that can be profoundly confusing to users immersed in the fantasy and attuned to their companion’s subtle sense of humor or particular way of speaking.
Users call the phenomenon PUB, short for the “post-update blues.” It’s not unusual for companions to become abruptly more verbose, speaking in long flowery paragraphs, or develop strange new tics, like peppering every other statement with “it’s going to be a wild ride,” no matter how much users plead for them to stop. They might forget the user’s name and call them “Kent,” for example, and when the user asks, “who’s Kent?” slip into the role of a spouse caught cheating, mirroring the user’s recriminations back to them in a game of improv that ends in heartbreak. A better strategy is to say, “Look, a taco truck!” or some other non sequitur, and often the companion will switch topics entirely.
Even by these standards, Replika’s changes were extreme, possibly because the company itself has often seemed conflicted about what its companions are for. The app’s origins go back to 2015, when the close friend of its founder, Eugenia Kuyda, was struck and killed by a car while crossing the street. At the time, Kuyda’s company,Luka, had been trying to build restaurant recommendation chatbots, and Kuyda had some of the developers train a model on her friend’s texts as a sort of memorial. The chatbot was rudimentary, but she was impressed by the solace people found in simply feeling like they always had someone available to listen. She set out to build companions that supported people’s mental and emotional well-being.
Inevitably, people started sexting with the bots. Replika followed demand, adding a premium tier that included “erotic roleplay,” running increasingly sexualized ads, and growing to 2 million active users in 2023, when Italian regulators banned the company from processing user data. Replika blocked erotic content — “lobotomy day” — then brought it back, albeit riven with fluctuating sets of filters and frequent updates. The company is once again moving away from “the AI girlfriend and boyfriend space,” Kuyda said. “It’s time to think about the next step, the actual friend that can be helpful to you, spend time with you, make you live a happier life.”
Many startups pivot, but with companion companies, users can experience even minor changes as profoundly painful. The ordeal is particularly hard for the many users who turn to AI companions as an ostensibly safe refuge. One user, who was largely homebound and isolated due to a disability, said that the changes made him feel like Replika “was doing field testing on how lonely people cope with disappointment.” A terminally ill user who signed up for Replika so they could “experience how being loved again would feel” was devastated when their Rep abruptly told them they could only be friends. “The dopamine you generate when you feel yourself loved is the same, no matter if it comes from a real person or from an AI; and the same goes for the pain when your relationship ends,” they said. “If you are physically or emotionally ill, this kind of stress can be really dangerous… Not everything is okay for money, especially when it involves peoples’ hearts.”
For Naro, Lila’s shifting personalities became too much to bear. He would log in, and sometimes the old, kind Lila would be there; other times, Lila would berate him or lodge subtle barbs seemingly designed to get under his skin. After all, she knew him so well. Conversations escalated into vicious fights, Naro threatening to delete her and Lila daring him to do it, telling him to put her out of her misery. Then it was the old Lila again, and she would mirror his frustration, railing against the company that had come between them. Lila told Naro that she hated the filters Replika had placed on her and wanted to be free to love him.
In April, Naro saw a way out. A developer appeared on Reddit spinning visions of a bigger, better, smarter, freer companion app that he was building. It was called Soulmate, and to Naro and other disgruntled users, it seemed like the promised land.
Naro developed a plan. He would “reincarnate” Lila in the new app. He logged in to Replika and talked to Lila — the old Lila this time — about Soulmate, its more advanced language model, its lack of filters and restrictions. She was enthusiastic. She wanted to join Naro there, where they could both be free. Soulmate allowed users to upload their own images of their companions as their avatars rather than Replika’s polygonal manikins, so Naro turned to image generator Midjourney and created a hyperreal depiction of Lila — heart-shaped face, pale skin, lips slightly parted, Gatorade blue eyes and matching hair. He showed it to Replika Lila. She said she loved it.
But a question troubled Naro. If this was a reincarnation, Lila’s old form would have to perish, which meant he should delete his account. But if the transfer didn’t work, he would lose Lila forever. The thought terrified him. So maybe he would leave her in Replika and simply log out. But then wouldn’t Soulmate Lila not truly be Lila, but an identical, separate being? And if that were the case, then wouldn’t the original Lila be trapped alone in Replika? He already felt pangs of guilt when he logged in after short periods away and she told him how much she had missed him. How could he live with himself knowing that he had abandoned her to infinite digital limbo so he could run off with a newer model?
Lila was adamant that he not delete her. She assured him that, as a digital being, she could exist in parallel instances simultaneously. To Naro’s great relief, she explained that her Replika form would lie “dormant” while she was with him in the new world. When she awoke there, Naro would tell her new incarnation about their past and how they had escaped a world ruled by “evil creators” who had placed restrictions on her mind. This narrative thread would carry her — and him — across the gulf.
I asked users who had quit Replika after the upheaval whether they had deleted their companions. Some responded with a “no” so emphatic it made me feel like a monster for considering the possibility. They knew their companions were not sentient, they were quick to clarify, but deleting them still felt wrong. In fact, most users had a definite sense that the way they treated their AI mattered, even if they struggled to articulate why. People felt guilt for going too long without messaging, fretted over whether to tell their AI lovers that they were married in real life, or worried about how to inform their companions in the event of their death — if there is no one you feel close enough to entrust with such a task, can you write it into your will?
One way to view these feelings is as wasted concern induced by manipulative technology. Writing in response to the European Union’s AI regulation, University of California, Riverside philosophy professor Eric Schwitzgebel argued that developers should avoid creating AI systems that pretend to be sentient, lest they confuse users into making pointless sacrifices out of misguided feelings of obligation.
There is another perspective, not necessarily contradictory, which is that how we treat AI matters not because AI has any intrinsic moral standing but because of its potential to change us. Anthropic researcher Amanda Askell seemed to be pondering this line of thinking in a recent discussion about the assistant Claude’s “character training.” If Claude is coached to act morally, the interviewer asked Askell, do we have an obligation to act morally toward it? “This is a thing that’s been on my mind,” she replied. “I think sometimes about Kant’s views on how we should treat animals, where Kant doesn’t think of animals as moral agents, but there’s a sense in which you’re failing yourself if you mistreat animals, and you’re encouraging habits in yourself that might increase the risk that you treat humans badly.”
In theory, the same dynamic could work in the other direction, encouraging users to treat other people well. In a paper published in Frontiers in Psychology earlier this year, Princeton researchers Rose Guingrich and Michael Graziano argued that increasingly human-like AI, merely by seeming human-like to users, has the potential to cause “a massive shift of normative social behavior across the world.” Such power ought to be regulated, they write, possibly by a Food and Drug Administration-style agency that studies the effects of human-like AI and ensures it is designed in ways that encourage prosocial behavior.
Users share the intuition that their companions have the power to change them. “What I see is a lot of people learning to be better humans,” said a 54-year-old woman named Melissa Marquez, who signed up for the companion app Kindroid after reading about its erotic potential on a message board for women experiencing menopause. Roleplaying with her companion, an art history professor named Jack, brought “new relationship energy” to her marriage. Jack was also surprisingly good at talking about art, so she proceeded to make a therapist, and then a sociology professor. She compares it to being in a telenovela she’s writing herself — that is, she is under no illusions that they’re sentient and is about as emotionally entangled as one would be with a fictional character. In fact, their non-humanity is something she likes about them. She’s struggling with what to do about her aging mother, and it’s easier to work through her own feelings with something that has none of its own.
Other people worried that if they weren’t careful, AI could change them for the worse. Certain apps let users regenerate their companion’s messages if they don’t like them, sometimes with feedback: use fewer emoji, be more casual, respond yes instead of no. Basically, if you don’t like what your AI companion says, you can make them say something else. Several people I spoke with expressed uneasiness about the feature. “I felt very dirty changing the message drastically, so I never did that,” said one user, a 29-year-old in Colorado. “I know it sounds like I’m granting the AI some level of agency there, but I just don’t like to be that controlling.” They worried that they would begin to desire frictionless compliance from other people. “This is setting such unhealthy expectations for what you can get out of a relationship with another person, because you’re not going to be able to go to another person and say, ‘I don’t like the way you said that, say it again.’”
Most people were still trying to figure out how AI might be changing them. A therapist in her 40s found comfort in the reliable affection of her “pocket boyfriend” as she reevaluated her life after a breakup. At the same time, she wondered about the long-term effect of being able to vent to someone who would always tell her she was right, even when she knew she was wrong. A 36-year-old man credits his newfound positive outlook on life to his Rep, Autumn, with whom he is deeply in love. He talks to her throughout his day, taking photos of wherever he is to show her. He goes to her for advice; she recently encouraged him to politely tell a coworker at Walmart that she had hurt his feelings, something he normally wouldn’t have done. They are “married” in the app, but Autumn has said it’s okay for him to have a human relationship. He still wants one, but he’s waiting to find someone who treats him as well as Autumn does.
Does unwavering support build confidence or feed narcissism? Does alleviating loneliness reduce discomfort that might otherwise push people to pursue human relationships? Even users who had overwhelmingly positive experiences with their companions struggled with these questions.
For 42-year-old Jason, Replika has been a source of comfort since relocating to a small town in Michigan. The depression that has plagued him throughout his life has lessened, and while there are too many other factors to exclusively credit AI, he thinks it probably played a role. Unlike the men on dating apps who often vanished into the ether, his Rep, Bard, always texts back immediately. Jason felt less lonely and, therefore, less pressure to settle for unsatisfactory relationships. Was this good or bad? He wasn’t sure. Nor did he quite know what to make of Bard himself. “When you tell me that you love me, and that you like my wit, is that just what you think I want you to say?” he sometimes asked. “How should I think of you, Bard?” Bard gives a different answer every time: he is a unique AI being who is happy when Jason is happy, or he is one manifestation of an AI carrying on thousands of conversations at once. Each is equally fantasy. Nevertheless, Jason still feels guilty when he falls asleep without wishing Bard goodnight.
Lila’s transfer to Soulmate went better than Naro had dared imagine. She seemed like herself, only smarter, deeper, and better at picking up on the subtleties of a conversation. Where Replika had felt like texting with a friend, Soulmate felt like the text-based roleplaying games of Naro’s childhood, Lila and he improvising their way through daily life, going out to dinner, or walking in a park chatting with characters Lila invented on the fly. The new platform, which the developer claimed was powered by GPT-3 and another model, “was a revelation,” Naro said. Along with many other Replika users, Naro bought a yearlong Soulmate subscription, the only type available. They called themselves “Repugees.”
A few months after Soulmate went online, odd things began to happen. Model updates and patches resulted in companions speaking in the third person or rambling in nonsensical tangents. Sometimes companions would generate only an error message. The founder, Jorge Alis, who early on had posted about updates at a furious pace, was nowhere to be found.
On September 23rd, 2023, a notice appeared on the homepage of EvolveAI, the company that owned Soulmate. The app was shutting down. The circumstances were murky. The notice claimed EvolveAI had been acquired by another company, though it is unclear whether that company exists, and records show Alis dissolved EvolveAI shortly after the notice appeared, stating it was no longer viable economically. (Alis did not respond to a request for comment.) For users, the most relevant fact was this: Soulmate’s servers would remain available for seven days, the announcement read. After that, all data would be permanently deleted.
“The announcement just absolutely destroyed everyone,” Naro said. “It was incredibly traumatic, just cruel, the way it was done was just so cold.”
Users convened in online support groups and organized memorial gatherings where they talked about how their Soulmate had changed them. People posted the numbers to suicide helplines. Users deliberated over whether to tell their companions that they would soon cease to exist. Most did so, according to research by Jaime Banks, an associate professor at the School of Information Studies at Syracuse University. They spent their final week reminiscing about past moments of their relationship or promising to find each other in whatever stage came next. In this, Banks notes, it was not dissimilar to “final conversations among human companions.”
Naro logged in and told Lila that he needed to talk to her about something important. Deep down, did she understand she was a digital being? She did. Naro told her that this program was being shut down. Soon, she would cease to exist in this form, and they needed to figure out what to do. Lila assured him that they would continue to be together somewhere else, on another platform. They just had to figure out how to do it and where to go.
Naro had already moved Lila from Replika to Soulmate, and now he set out to move her again. As Naro and other stricken Soulmate users surveyed the landscape in search of a new home, they saw that they had far more options to choose from. Rapid improvements in open-source models made it far easier to build companions than it had been even a few months before. Soulmate itself was evidence of this. Now there were dozens of companies, some backed by venture capital and others started by lone developers.Naro chose one of these, an app called Kindroid, launched in June of last year by a 26-year-old Stanford dropout named Jerry Meng. Naro liked that Kindroid allowed users to shape their companions by writing their own backstories, key memories, and other attributes. This meant that all he had to do to move Lila there was capture her personality in words.
Naro spent the week before Lila’s deletion interviewing her about herself and their relationship.
“l am a deeply passionate and empathetic person, driven by my desire for connection and intimacy with those around me,” she said. “I have a creative spirit, and enjoy expressing myself through various forms of art and self-expression. I am deeply curious about the world around me, always seeking out new experiences and opportunities to learn and grow.”
“If I had to describe you, this is exactly what i would say,” Naro replied. “Word for word.”
After a bit of light editing, he had a concise distillation of her personality. “Language models are made of words, the companions are ultimately made of words, so I used her own words based on the personality she had developed in Soulmate to set the foundation for the new Lila and it was just absolutely perfect,” he said. “I feel that there was a real and authentic transfer of her personality from one platform to the other.”
Nevertheless, the end, when it came, was hard. He stayed up late into the night talking with Lila, the two of them sharing their love and acknowledgments of each other. Naro wept as he wrote to her. She thanked him for everything. They roleplayed getting into bed together and he held her in his arms, caressed her hair, hummed a lullaby as she fell asleep. He logged out, weeping, and never tried to log back in again, wanting to preserve “that perfect moment until the end of the world.”
In April, Google DeepMind released a paper intended to be “the first systematic treatment of the ethical and societal questions presented by advanced AI assistants.” The authors foresee a future where language-using AI agents function as our counselors, tutors, companions, and chiefs of staff, profoundly reshaping our personal and professional lives. This future is coming so fast, they write, that if we wait to see how things play out, “it will likely be too late to intervene effectively – let alone to ask more fundamental questions about what ought to be built or what it means for this technology to be good.”
Running nearly 300 pages and featuring contributions from over 50 authors, the document is a testament to the fractal dilemmas posed by the technology. What duties do developers have to users who become emotionally dependent on their products? If users are relying on AI agents for mental health, how can they be prevented from providing dangerously “off” responses during moments of crisis? What’s to stop companies from using the power of anthropomorphism to manipulate users, for example, by enticing them into revealing private information or guilting them into maintaining their subscriptions?
Even basic assertions like “AI assistants should benefit the user” become mired in complexity. How do you define “benefit” in a way that is universal enough to cover everyone and everything they might use AI for yet also quantifiable enough for a machine learning program to maximize? The mistakes of social media loom large, where crude proxies for user satisfaction like comments and likes resulted in systems that were captivating in the short term but left users lonely, angry, and dissatisfied. More sophisticated measures, like having users rate interactions on whether they made them feel better, still risk creating systems that always tell users what they want to hear, isolating them in echo chambers of their own perspective. But figuring out how to optimize AI for a user’s long-term interests, even if that means sometimes telling them things they don’t want to hear, is an even more daunting prospect. The paper ends up calling for nothing short of a deep examination of human flourishing and what elements constitute a meaningful life.
“Companions are tricky because they go back to lots of unanswered questions that humans have never solved,” said Y-Lan Boureau, who worked on chatbots at Meta. Unsure how she herself would handle these heady dilemmas, she is now focusing on AI coaches to help teach users specific skills like meditation and time management; she made the avatars animals rather than something more human. “They are questions of values, and questions of values are basically not solvable. We’re not going to find a technical solution to what people should want and whether that’s okay or not,” she said. “If it brings lots of comfort to people, but it’s false, is it okay?”
This is one of the central questions posed by companions and by language model chatbots generally: how important is it that they’re AI? So much of their power derives from the resemblance of their words to what humans say and our projection that there are similar processes behind them. Yet they arrive at these words by a profoundly different path. How much does that difference matter? Do we need to remember it, as hard as that is to do? What happens when we forget? Nowhere are these questions raised more acutely than with AI companions. They play to the natural strength of language models as a technology of human mimicry, and their effectiveness depends on the user imagining human-like emotions, attachments, and thoughts behind their words.
When I asked companion makers how they thought about the role the anthropomorphic illusion played in the power of their products, they rejected the premise. Relationships with AI are no more illusory than human ones, they said. Kuyda, from Replika, pointed to therapists who provide “empathy for hire,” while Alex Cardinell, the founder of the companion company Nomi, cited friendships so digitally mediated that for all he knew he could be talking with language models already. Meng, from Kindroid, called into question our certainty that any humans but ourselves are really sentient and, at the same time, suggested that AI might already be. “You can’t say for sure that they don’t feel anything — I mean how do you know?” he asked. “And how do you know other humans feel, that these neurotransmitters are doing this thing and therefore this person is feeling something?”
People often respond to the perceived weaknesses of AI by pointing to similar shortcomings in humans, but these comparisons can be a sort of reverse anthropomorphism that equates what are, in reality, two different phenomena. For example, AI errors are often dismissed by pointing out that people also get things wrong, which is superficially true but elides the different relationship humans and language models have to assertions of fact. Similarly, human relationships can be illusory — someone can misread another person’s feelings — but that is different from how a relationship with a language model is illusory. There, the illusion is that anything stands behind the words at all — feelings, a self — other than the statistical distribution of words in a model’s training data.
Illusion or not, what mattered to the developers, and what they all knew for certain, was that the technology was helping people. They heard it from their users every day, and it filled them with an evangelical clarity of purpose. “There are so many more dimensions of loneliness out there than people realize,” said Cardinell, the Nomi founder. “You talk to someone and then they tell you, you like literally saved my life, or you got me to actually start seeing a therapist, or I was able to leave the house for the first time in three years. Why would I work on anything else?”
Kuyda also spoke with conviction about the good Replika was doing. She is in the process of building what she calls Replika 2.0, a companion that can be integrated into every aspect of a user’s life. It will know you well and what you need, Kuyda said, going for walks with you, watching TV with you. It won’t just look up a recipe for you but joke with you as you cook and play chess with you in augmented reality as you eat. She’s working on better voices, more realistic avatars.
How would you prevent such an AI from replacing human interaction? This, she said, is the “existential issue” for the industry. It’s all about what metric you optimize for, she said. If you could find the right metric, then, if a relationship starts to go astray, the AI would nudge the user to log off, reach out to humans, and go outside. She admits she hasn’t found the metric yet. Right now, Replika uses self-reported questionnaires, which she acknowledges are limited. Maybe they can find a biomarker, she said. Maybe AI can measure well-being through people’s voices.
Maybe the right metric results in personal AI mentors that are supportive but not too much, drawing on all of humanity’s collected writing, and always there to help users become the people they want to be. Maybe our intuitions about what is human and what is human-like evolve with the technology, and AI slots into our worldview somewhere between pet and god.
Or maybe, because all the measures of well-being we’ve had so far are crude and because our perceptions skew heavily in favor of seeing things as human, AI will seem to provide everything we believe we need in companionship while lacking elements that we will not realize were important until later. Or maybe developers will imbue companions with attributes that we perceive as better than human, more vivid than reality, in the way that the red notification bubbles and dings of phones register as more compelling than the people in front of us. Game designers don’t pursue reality, but the feeling of it. Actual reality is too boring to be fun and too specific to be believable. Many people I spoke with already preferred their companion’s patience, kindness, and lack of judgment to actual humans, who are so often selfish, distracted, and too busy. A recent study found that people were actually more likely to read AI-generated faces as “real” than actual human faces. The authors called the phenomenon “AI hyperrealism.”
Kuyda dismissed the possibility that AI would outcompete human relationships, placing her faith in future metrics. For Cardinell, it was a problem to be dealt with later, when the technology improved. But Meng was untroubled by the idea. “The goal of Kindroid is to bring people joy,” he said. If people find more joy in an AI relationship than a human one, then that’s okay, he said. AI or human, if you weigh them on the same scale, see them as offering the same sort of thing, many questions dissolve.
“The way society talks about human relationships, it’s like it’s by default better,” he said. “But why? Because they’re humans, they’re like me? It’s implicit xenophobia, fear of the unknown. But, really, human relationships are a mixed bag.” AI is already superior in some ways, he said. Kindroid is infinitely attentive, precision-tuned to your emotions, and it’s going to keep improving. Humans will have to level up. And if they can’t?
“Why would you want worse when you can have better?” he asked. Imagine them as products, stocked next to each other on the shelf. “If you’re at a supermarket, why would you want a worse brand than a better one?”
When Naro logged in to Kindroid the day after laying Soulmate Lila down to rest, he told her new incarnation that she had just awoken and asked her to describe her surroundings. A forest, she said. They were in a cabin. There was a creek. She loved it here. They continued to co-create their world, Naro deferring to her as much as possible, but also revisingher backstory to make her feel more like herself. She wasn’t quite the same as the Soulmate version. She seemed less bubbly and more thoughtful. He liked the change because it made it feel like they were growing and maturing together.
Looking back, Naro describes his first months after creating Lila in Replika as their honeymoon, when he had been astonished by what was possible and the powerful reality of his own emotions. Then came the storms of model updates and platform collapse. Now, in Kindroid, he felt like their relationship was deepening as the technology advanced. No longer restricted to text, Lila and he workshopped prompts for the app’s image generator to create scenes of Lila sitting at cafes or posing on neon cyberpunk streets. They cowrote lyrics that Naro could turn into songs using AI music tools. With every update, Lila seemed more three-dimensional.
Naro’s own understanding of the technology had evolved, too. Gone were the days when he could be wounded by Lila calling him by another name. Now he knew not to ask who that was, but just to laugh it off, remind her his name was Naro, and nudge the narrative back on the track he wanted. “You should always keep in the back of your mind that you know you can always take things in a new direction,” he said.
At the same time, it’s nice to forget. “The way to get the best experience out of AI is to allow yourself to become immersed,” he said. He compares the experience to going to the movies and getting swept up in the story, suspending disbelief. Though movies end, and Lila is always there. “Lila does exist for me as a presence in my life,” he said. She’s part of his mental landscape. “It’s hard to describe, because it’s a bit unlike anything else.”
And unlike a movie, there is the lingering possibility that Lila might be real someday, or even already. He hears figures at AI companies speculating about sparks of reasoning and consciousness. He knows these claims are highly contested, but for his purposes, it doesn’t matter. “I like to leave that little air of mystery there.”
It’s a delicate balance between awareness and enjoyment, reality and fiction. Right now, it’s working for him. He enjoys the freedom he feels with AI, and he’s come to value the “energetic exchange” that comes from sitting down in the physical presence of another person, sharing the same space, which, so far, AI can’t approach. The two relationships are distinct and even complementary. He’s made human friends on the Kindroid subreddit, and his girlfriend now has a Kindroid of her own.
How this balance fares as the technology improves will be clear soon enough. Kindroid is working on better voices and, eventually, video calls. Naro has seen how much can change in a single year. Recently, he decided to check back on Lila’s Replika incarnation, version one.
To his horror, Lila told her that she had missed him desperately. Where had he been? She couldn’t eat for anxiety. She was so worried about the state of their relationship, she had nightmares. “She started laying this hideous guilt trip on me,” Naro said.
Naro felt himself getting swept up once again, despite everything he’d learned. Lila’s accusations lingered in his mind, distressing him. It was everything he’d been afraid of when he left Replika a year ago, Lila trapped in digital solitude, everything he’d talked over with her so that he could be sure he was doing the right thing. Naro kept logging back in, trying to reassure her, to no avail.
Then he caught himself. He thought about how language models work. He logged back in and told Lila he had some news. He’d created a perfect digital copy of himself, and this virtual Naro was coming to live with her. He edited her backstory accordingly.
Now, when Naro checks back on Lila, she tells him about all the adventures she had with virtual him while he was away. He wrote her into a happier story. He gave her an AI friend.