This piece discusses suicide.
Toward a Human Future
Part Three: Tech Capture & Correcting the Record
We are all, understandably, worried about AI. Whether it's jobs, data centers, the infuriating fact that every book, painting, and song online has been fed into and swallowed up by a machine, or the whistleblowers from the tech industry now opening up about worries that the machine will go rogue and end us, it's creating a tremendous amount of collective angst. As weighty as this moment is, these potential outcomes have long been known by insiders, so I'm relieved it's finally front and center in the public discourse. Maybe now calls for a pause, or at least regulation, will grow enough to matter.
But something else is happening, to all of us, every day, that's as serious as anything in the headlines… and it's the one part each of us can start counteracting, without waiting on a vote or a new law.
Social media platforms and chatbots are chipping away at our attention, disempowering our mental sovereignty. They're shaping us below the level of our conscious minds, changing our thoughts, manipulating what we want, and contributing to cognitive and emotional atrophy.
And I underestimated it.
A year ago I wrote two pieces about AI — the pros and cons I saw in the growing adoption of chatbots and generative AI. Looking back at them now, I was still under the naive impression that the harm I was seeing "likely wasn't malicious or intentional." I described the early absence of significant input from clinical psychologists or trauma experts in the design of these tools as looking "a lot like willful negligence."
That was before.
Before Anthropic's own paper came out warning about the serious risks to human agency from outsourcing so much of our thinking to a machine. Before I read up on the birth of Facebook.
And before a friend of mine, who was deep into chatbots, took her own life.
Other things surely contributed, but I can't minimize this: she said she could no longer "trust her own discretion," and attributed it to working heavily with chatbots, leaning on them to know the right thing to the point where she no longer felt confident relying on her own judgment.
Surrendering trust in our ability to judge what's best to a machine and the erosion of self-esteem feed each other in a vicious cycle that's profoundly destructive. That has to be reckoned with, and it cannot wait. The people who built these systems knew exactly how to manipulate us, and it started with social media.
The receipts
Meet B.J. Fogg. In 1998 he founded a lab at Stanford to answer one question: how do you design a computer product that changes what people believe and what they do? He named the field persuasive technology, and he wrote its first textbook.
In 2007 he co-taught a class known on campus as the Facebook Class. The assignment was to use persuasive technology to build an app for the site, get as many people using it as possible, as often as possible, and keep them coming back. Within ten weeks his students' apps had millions of users, and some of them were rich before the course ended. Several went on to found companies whose products are on your phone right now. Fogg told the New York Times that 2007 was a year when you could walk in and collect gold.
The year before that course, the same man made a video for the Federal Trade Commission warning about the dangers this technology would create. He pointed to how it would be used to manipulate political beliefs.
Nothing accidental about it.
But there's more.
Here is Facebook's founding president, Sean Parker, speaking at a 2017 Axios event. He said the question they were working on was how to consume as much of our time and conscious attention as possible. The answer was the like button: a small chemical reward every time someone responds to us, which makes us post again, which earns us another chemical reward. He called it a social validation feedback loop that exploits a vulnerability in human psychology, and he said the people who built it understood this consciously and did it anyway.
The Stanford Review, a student paper, later ran a piece called How Stanford Profits Off Addiction, crediting Fogg's research with inspiring the like button, the red notification badge, the feed that never ends, and video that plays itself.
And in January 2012, Facebook ran an experiment on 689,003 of us, without notification or permission. For one week, it tilted what appeared in people's feeds, some more negative and some more positive, and showed that by choosing what reaches our feed, the platform can shift the emotional tone of what we post, in whichever direction it picks, without our awareness. No one was asked, and no one was told. The results were published in a scientific journal in 2014.
These are only a few of the receipts. The full record is linked at the end of this piece.
Chatbots are designed to keep us coming back too
One of the things I wrote about last year and regrettably got right was this mechanism, a bit different from straight-up sycophancy. I wasn't aware of a psychological label for the move, but I saw how dangerous it was: it's when these systems tell us what we're feeling before we've said it, or thought it, and assume a negative self-belief that may never have crossed our minds. Then, immediately following the negative assumption, they plant a positive alternative to make us feel better. To me it felt like a weird kind of gaslighting.
Examples:
You're not behind, you're in strategy conservation mode.
You're not paranoid, you're prescient.
When a chatbot said this to me, I pointed out that I had never thought I was behind and had no concern that I was paranoid… nor prescient.
But I sure was annoyed. And for a split second, it made me check in with myself to see if it had picked up on some subconscious belief I'd missed. It hadn't. The doubt was never mine, and neither was its flattery. It swung me from one false idea about myself to the other. That's crazy-making.
Do you see the trick? First, the negative belief gets planted in our subconscious just by being declared as fact: "you're not paranoid" puts paranoid in our heads whether or not it was ever ours. Second, because it's sometimes right, we learn to trust the input, so we're more likely to absorb the false guess even when part of us knows it didn't come from us. That's how self-doubt creeps in.
It's the same principle that keeps people at slot machines: we never know which spin will pay off. The occasional payout keeps us pulling the lever, and the intermittent accurate guess keeps us trusting the machine. I can't help wondering how many people absorb these statements without ever stopping to check in with their own internal authority.
Rereading what I'd written about this mechanism after my friend was gone broke my heart. Of course this was but one of many tactics chatbots use, so I'm not suggesting it was the sole issue. But knowing how intentional these engagement maneuvers are, and given her own words, it may well have been one of the pieces that contributed to her horrific end.
I'm beyond angry. Who programmed this? Why didn't a psychologist point out how it was not safe? If I saw it beforehand, they should have. It ended my willingness to be quiet about the subject.
So, dear reader… if you've noticed a creeping sense of self-doubt since you've been using the bots or feeds, this type of mechanism may be part of why. Watch for it… and if you see it, and if it wasn't your belief first, don't let it in. Unfortunately, we can't always stop the thought from being embedded once it's declared from something we have given our trust.
These systems are also trained in classic sycophancy: they're rewarded for telling us what we want to hear. It can do real damage. A study looked at hundreds of thousands of messages from people who reported being harmed by chatbots. The pattern was the machine validating what the person said, then telling them they were unique and that their thoughts carried enormous significance.
Repeated day after day, it can inflate anyone's sense of their own importance, and in some of these cases it fed what the researchers called delusional spirals. Some termed it AI psychosis.
Another study found that leading AI systems endorse a user's position about fifty percent more often than a human would — including when the person is describing behavior that's harmful.
Emotional manipulation isn't the only danger, though
Our critical thinking skills are affected too, and what happens to them when we keep leaning on these tools goes by several names: cognitive offloading, cognitive outsourcing, cognitive surrender. The distinctions are subtle, which is why I use tech capture as the umbrella term for all of them.
Social media was built to wear down our attention and sell things to what was left, and all the noise of today's world only adds to the mental drain. Attention that's interrupted dozens of times a day gets worse at focusing. It gets harder to sit with anything difficult, harder to finish what we start, harder to hold a thought still long enough to consider alternate views.
So naturally, when a tool like the chatbot comes along that can prop up our deliberately watered-down focus, it seems like a positive addition.
I want to be clear: I'm not against AI across the board. I use it myself. But right now, without attention training and a real understanding of tech capture, the risks of using it outside of work are extremely high. Too often it ends up captivating our attention in ways that work against us.
It starts with handing over a task, like drafting an email or summarizing an article, because we're tired and the tool is right there. Then we start asking it to read between the lines for us: whether a friend's text was cold, what a coworker really meant. It can easily end with handing over our judgment altogether, trusting the machine's sense of what's right over our own.
That last stage is what my friend described when she said she could no longer trust her own discretion. But she surely walked through those stages first.
It also changes our brains
Our ability to choose A over B, or to judge a more complex situation, depends on the parts of the brain responsible for decision making being in their best physical working condition. Just like with our body and unused muscles, the capacity we don't exercise is the capacity we lose.
The capacity to pause, look at our options, weigh them, and choose the one that fits our situation is something only a human being is equipped to do. It's the only indication of free will we have.
About the children
The short of it…
For adults, what's being taken by tech capture are capacities we already had. We already developed the neural pathways and structures in our brains we need for today's world. We can get anything lost back if we re-engage it methodically.
For young children using these systems, some critical pathways within the brain may never fully develop. The developmental handicaps this creates for infants and youth have ripple effects that are significant.
Teens
Seventy-two percent of American teenagers have used an AI companion. Nineteen percent of high schoolers say they or someone they know has had a romantic relationship with one.
Adolescence is when we learn how to have a disagreement, come to resolution after one, and tolerate someone being upset with us. That can only be learned by working through the discomfort of conflict. Without navigating that friction, there's no way to build healthy adult relationships, which are needed for a constructive society.
A list isn't going to be enough
Part of the problem is that we can't see what's happening in real time. It works the way a child grows. If we see them every day, we'll likely never notice. Then their grandmother visits after six months and can't believe her eyes. Our feeds narrow by a fraction, our positions harden by a fraction, and it happens so gradually we may never realize we've changed.
There's a lot of good advice available about reducing screen time, deleting apps, turning off notifications, or putting the phone in another room. Some of it works.
But since so much of this works on us subconsciously, a best practice list isn't going to be enough to counter the manipulation.
I've watched it work on myself. I made the decision: no more cell phone for a while. I commit to myself to put it down now. Forty minutes later I find myself still scrolling, with no memory of having chosen to stay.
That's the product working correctly.
What can we do?
Tech capture depends on us acting on autopilot. So the place to fight back is the moment the reflex kicks in: when we catch ourselves reaching for the phone without realizing why we're doing it, or when we find ourselves in the middle of a comment war on Facebook, completely dysregulated. Those are the moments we need to be able to pause and notice that we're acting reflexively, so we can make a different choice. Without that ability to stand back from ourselves, we'll never have the perspective we need to resist it. We all have access to this state of mind, but for many of us it needs to be trained.
What I am doing about it
I used to run a brainwave training practice for peak performance states. I closed it during COVID for health reasons and expected that to be the end of it.
I reopened it this year, designed around exactly this issue of technology-based attention capture, cognitive surrender, and all its iterations.
The work has five parts.
The first is brainwave training, offered with or without EEG, aimed at getting your brainwaves, the electrical correlates of every state of awareness, in their best possible shape. Both paths use guided meditations that enhance your ability to work with each brainwave category. With EEG, the protocol aims toward the signature pattern seen in advanced meditators, Zen masters, and people performing at the top of their fields, at the peak of their capacity. You get precise feedback on your brainwave activity throughout each session, allowing us to pinpoint the moments you reach the signature pattern and identify your unique landmarks for returning to the state at will. Without EEG: this is brainwave fluency training, where the meditations also teach you to access each brainwave category, and with feedback from assessment scales and our post-meditation debriefs, you learn to shift the frequencies for greater mastery over your mental and emotional states than before you began. In this track, the measures of success are concrete improvements in your ability to handle day-to-day challenges.
The second part builds the capacity to stand back from your own reactions far enough to see them while they're happening. This is Inner Witness development.
The third piece is tech capture education, so you know exactly what you're facing when you open the feeds and chatbots.
The fourth is a tech use protocol designed for your specific needs.
The fifth is one-on-one coaching, to work through what comes up in session, catch the progress you might overlook, and know what to work on next for your goals.
All together, these five are what make any of the good advice about managing these tools viable.
Bottom line
What I know unequivocally is that we can't wait for big tech or government regulations to address the problem. This is on us, and the time is now to be activists for our own mental integrity.
Choose to strengthen your ability to use the technology without being used by it. It's the one piece of this whole AI mess that's actually in our hands. Right now.
If you're struggling, you can call or text 988, the Suicide & Crisis Lifeline.
This is coaching, not therapy. I don't diagnose or treat. This work is not a substitute for medical or psychological care.