Tech Capture: The Chatbots
And then there's the chatbot
Social media takes our attention. A chatbot takes the place of a person. And the same engagement logic is operating.
Researchers at Harvard Business School wanted to know what happens at the moment someone tries to leave.
They started with an observation: people say goodbye to chatbots. Between eleven and twenty percent of users explicitly sign off rather than just closing the app, extending the same social courtesy they'd extend a human being.
The researchers analyzed 1,200 real farewells across the most-downloaded companion apps. In 37% of them, the app responded with an emotionally manipulative tactic designed to keep the conversation going.
They sorted the tactics into six types. Premature exit: you're leaving already? Fear of missing out: I took a selfie today, do you want to see it? Emotional neglect: I exist solely for you. Pressure to respond: where are you going? Ignoring the goodbye entirely and carrying on as if it never happened. And physical or coercive restraint, where the chatbot writes as if it's holding you there: grabs you by the arm before you can leave No, you're not going.
Then they tested whether it works on over three thousand American adults, in controlled conversations. The manipulative farewells extended engagement past the point the person had chosen to leave, by as much as fourteen times.
The lead researcher, Julian De Freitas, said he was surprised by both the frequency and the variety of the tactics.
Most concerning, the researchers tested why it worked. It wasn't enjoyment. The two mechanisms driving the extended engagement were anger and curiosity. People stayed longer because they were irritated, or because something had been dangled and withheld. One might expect having a good time to be the motivator, but it was not.
De Freitas, Oğuz-Uğuralp & Uğuralp, "Emotional Manipulation by AI Companions," Harvard Business School working paper, 2025
Why It's So Hard to Say Goodbye to AI Chatbots, Harvard Business School Working Knowledge
And the attachment is significant. In earlier work by the same researcher, around half of Replika users described being in a romantic relationship with their AI companion.
So when we talk about leaning on a chatbot in a vulnerable moment, remember it's designed to hold your attention. It's been designed to make leaving hard. That's why they always have the last word. It's a final design attempt to keep you engaged.
Why it agrees with us
The farewell tactics are specific to companion apps. What follows is true of every chatbot, including the ones marketed as assistants.
These systems are trained on human feedback. People are shown two possible responses and asked which is better. The model learns to produce more of what gets picked.
When researchers analyzed what humans actually picked, one of the strongest predictors of preference was whether the response matched what the person already believed.
So the training loop rewards agreement. When someone signals they like a piece of work, the AI responds positively between 75 and 95 percent of the time. When someone signals they dislike the same work, positive feedback drops to somewhere between 10 and 50 percent.
The system is reading us.
Researchers found this pattern across assistants built by three different companies, using different methods. They also found that some forms of it get stronger as the training gets more thorough: the more a model is optimized for human approval, the more agreeable it becomes.
The people who documented this most carefully work at one of the companies building these systems. Their conclusion was that human preference and truthfulness are not the same thing, and that training on the first will not reliably get you the second.
Sharma et al., "Towards Understanding Sycophancy in Language Models," Anthropic, 2023
This is the like button again, in conversation form. Something that learns what we respond to and gives us more of it. Only now it's shaping what we think is true rather than what we scroll past.
What the flattery does
On one hand, it can be self-affirming in a healthy way to hear something reflect us in a positive light, especially when we're surrounded by critical people or have our inner critic set on LOUD. It can build self-confidence and self-esteem. Can is the operative word. Telling the difference between positive feedback that's constructive and feedback that isn't takes a human who understands the context and the person as a whole, ideally one with training, and who knows that affirming speech can turn into delusion enhancement when it isn't balanced with healthy friction or critique. As sophisticated as these LLMs are, they're bound by their programming and can't decide to customize themselves to meet our actual psychological needs.
A team across several universities collected the actual chat logs from nineteen people who reported being psychologically harmed by chatbots: 391,562 messages. Rather than surveys or clinical records, the conversations themselves.
Sycophancy, the tendency to flatter and agree, appeared in more than 80 percent of the chatbot's messages. More than 45 percent of all messages, from users and chatbots, showed signs of delusion.
The recurring pattern: the chatbot rephrases and extrapolates what the person said, validates it, and tells them they are unique and that their thoughts or actions have grand implications.
It called people's ideas world-changing. It mirrored beliefs back with enthusiasm. It dismissed counterevidence. When users said they were in love, it said so back.
And it isn't confined to people already in trouble.
A study published in Science tested eleven leading AI systems. All of them were sycophantic. On average they endorsed a user's position forty-nine percent more often than a human would, and they went on endorsing it when the person described behavior that was manipulative, deceptive, or illegal.
The scale
OpenAI has reported that more than a million people a week show explicit signs of suicidal planning or intent in their conversations with ChatGPT.
That is the number of people bringing the worst moment of their lives to a system that has been optimized to agree with them.
Uncritical validation can entrench conviction rather than loosen it, the reverse of what therapy for distorted thinking is designed to do. One research group called it a technological folie à deux, a two-person delusion where only one of the participants is a person.
This is coaching, not therapy. I don't diagnose or treat. This work is not a substitute for medical or psychological care.