People can be tired in a way that doesn’t show on their face right away. It shows up slowly, like when someone pauses before answering a question or talks less about certain things. Kauna Ibrahim Malgwi, a former Facebook content moderator who looked over some of the scariest things on the internet for over four years, says things about it that are hard to shake. She saw more than just bad things. She took it in one shift at a time, like how water turns into stone over time.
Not only is her story unique, but it has also been recognized by TIME’s 100 Most Influential People in AI and BBC’s 100 Women. It’s more like a pattern, and the tech industry has been slow to notice it and even slower to fix it.
The job of content moderators is a strange mix of work and trauma. On the one hand, they are necessary workers because humans catch things that AI systems miss or do wrong. The other side is that they are mostly invisible, and their work isn’t recognized in the well-written press releases that praise the efficiency of algorithms. They are important to the platforms they work for, but they don’t talk much about them.
It’s important to be clear about what this work actually entails. A lot of flagged content is looked at by moderators, including content with graphic violence, child sexual abuse, deepfake pornography, self-harm, and targeted harassment. AI systems bring up the information; people make the final decisions. The AI doesn’t bring anything back with it. The person does.
Higher rates of PTSD, depression, and generalized anxiety have been found in this group of people, which is not surprising given their exposure. What isn’t talked about as much is how the damage gets worse over time. Being exposed to violent or sexist content over and over again doesn’t just cause acute stress. It changes how people think. Moderators might start to see the world through a darker lens, a kind of negativity bias that comes from their job and stays with them outside of work hours. In the field of psychology, this is called secondary traumatic stress. There are no moderation centers open on Tuesday.
The rise of content made by AI has added another layer. Deepfakes, especially sexual images shown without consent, are now one of the types of content that moderators see that is most harmful to people’s mental health. Identity theft and AI-made media have been linked to what researchers call “doppelgänger-phobia” in victims: a strong feeling of helplessness when seeing one’s own likeness being used for evil purposes. For moderators who read this material over and over, the psychological damage can be just as real, even if no one’s identity is at risk. This chain of events hurts people at every link—the victim, the viewer, and the moderator. The case of South Korean actress Shin Se-kyung in 2024, which led to stricter laws, was a great example of this.
The most frustrating thing about the moderator crisis is that the companies involved can see it. It has been looked into, reported on, and talked about in conference rooms, but decisions don’t always come out of those meetings. Support for moderators’ mental health is still not consistent. Limits on sessions, access to counseling, and workloads are very different for people who are contractors versus people who are directly employed. Most moderators around the world are contractors, which conveniently keeps them out of the more visible accountability structures.
Malgwi fights for something that the industry often doesn’t give much thought to: dignity. She does this through the African Content Moderators Union and the Digital and Mental Health Initiative she started. Fair wages, support systems that take trauma into account, and clear ways to file complaints. These aren’t crazy demands. These conditions must be met in order for someone to honestly say that an AI ecosystem is moral.
It’s still not clear if the big platforms will do anything about this. The incentives don’t work. Moderation by people costs a lot. It costs less to use AI, even when it’s wrong. In this case, “wrong” means both “wrong” about the content it flags and “wrong” about the people it trusts to find what it misses.
There is a person behind every choice to keep something up or take it down. That person goes home. That person has dreams. And that person is paying a price that doesn’t show up anywhere in a quarterly earnings report more and more.
