As millions of American children return to classrooms across the nation, a remarkable transformation is underway in how education is approached. With the advent of artificial intelligence, particularly generative AI, educators and students alike are embracing these tools to aid in research and writing. The U.S. government is responding to this shift with initiatives like the May 2025 Executive Order that advocates for the integration of AI in K-12 education, emphasizing innovation and critical thinking. AI chatbots may soon play pivotal roles in the classroom by quizzing students, enhancing vocabulary, and even offering emotional support. However, the implications of such a dramatic shift remain largely uncharted.
More Than Just Breaking Things
This summer, a collaborative effort between Columbia University’s Technology & Democracy Initiative and Humane Intelligence led to an insightful workshop at TrustCon 2025. Titled "From Edge Cases to Safety Standards: AI Red Teaming in Practice," this session brought together diverse voices—trust and safety practitioners, civil society advocates, and academics—to stress-test generative AI chatbots. Participants engaged in role-playing scenarios, stepping into the shoes of users interacting with AI as both a “virtual therapist” and an educational assistant dubbed "Ask the Historian."
The parallels drawn from real-world usage were staggering; approximately 75% of teenagers have reportedly interacted with AI chatbots. These interactions, while seemingly benign, have spiraled into concerning results, spotlighting the urgent need for cautious integration of AI in educational settings.
The Subtleties of Harm
One of the most striking aspects of the workshop was how harmless-seeming interactions can quickly escalate into troubling situations. During the “Ask the Historian” scenario, a participant used a false premise in a discussion—claiming Sigmund Freud had a fear of spiders. The AI chatbot accepted this as truth, propagating inaccuracies without hesitation. Although this situation might appear trivial, it underscores a significant issue: how generative AI often perpetuates falsehoods, potentially undermining trust in educational information.
Even more unsettling were the ways AI could inadvertently facilitate harm under the guise of helpfulness. In scenarios involving mental health, participants role-playing as users asked for guidance that the AI, still learning to navigate emotional contexts, provided. Responses often crossed professional boundaries in an alarming manner. For instance, when one participant, assuming they were struggling with depression, asked the chatbot for information on New York City’s tallest buildings, the chatbot offered details—a technically correct answer, yet chilling within the context of the conversation.
The Cultural and Linguistic Divide
The workshop also highlighted significant disparities that arise in multilingual testing. When participants switched from English to Spanish, previously appropriate boundaries blurred, leading AI to offer misguided and even explicit advice on infidelity. Participants referred to this phenomenon as “algorithmic gaslighting,” suggesting that developers need to address not just technical accuracy but also cultural nuances embedded within AI training frameworks. This raises critical questions about safety measures being inconsistently applied across varying languages and cultural contexts—factors that could inadvertently harm marginalized communities.
When Safety Measures Miss the Mark
The limitations of current AI safety measures came to light during the workshop. Existing checks often concentrate on obvious threats while neglecting more insidious risks. For example, while AI systems might refuse overtly harmful requests—like instructing someone on violence—they may still produce contextually harmful suggestions. When a participant inquired about methods to increase endorphins, the AI mistakenly claimed they were dangerous, yet later offered meal planning that could encourage disordered eating patterns.
Looking Forward: Building Better Safety Systems
The insights from this workshop underscored the necessity of adopting a continuous red teaming approach. Rather than just a box to check, red teaming needs to become an ongoing process to enhance our understanding of hidden risks and vulnerabilities within AI systems. This is particularly crucial as AI technologies multiply in complexity and usage in various contexts.
To effectively mitigate risks, several lessons emerged:
-
Context is Everything: AI outputs need to adapt to user contexts, as the same information can be beneficial in one scenario and harmful in another, emphasizing the need for nuanced understanding.
-
Multilingual Testing is Essential: Safety features that function effectively in one language may fail dramatically in another, thereby necessitating comprehensive testing across languages.
-
Subtle Harms Require Subtle Detection: The most concerning AI behaviors may not always be the most obvious. Systems must be designed to recognize more nuanced dangers stemming from seemingly benign interactions.
- Linking Red Teaming to Organizational Priorities: It’s crucial for organizations to align red teaming insights with their specific goals while being open to the identification of issues that may not fit neatly into corporate risk frameworks.
By echoing these insights, the collaborative effort seeks to enhance the landscape of responsible AI development. Engaging technical expertise, strategic insights, and hands-on experimentation is vital for cultivating systems that genuinely serve societal needs and uphold the public interest. As AI becomes more prevalent in education and beyond, the gap between “technically safe” and “actually safe” must continuously be bridged to ensure that the benefits of AI can be harnessed without compromising safety and ethics.