I am a psychologist who builds AI. That is still a sentence I have to sit with sometimes, because the two halves of it are in genuine tension with each other, and I think the tension is worth naming rather than papering over with enthusiasm about innovation.
AI is a powerful tool. It can scale access to support in ways that human clinicians, who are finite in number and expensive in cost, simply cannot. For communities that have been chronically underserved by mental health systems, the potential of AI to deliver meaningful, accessible support is real and significant.
But AI can also cause harm. It already has. And the communities most at risk of that harm are, unsurprisingly, the same communities that have always been most at risk: Black, brown, and historically marginalized people whose lived experiences were not reflected in the training data that taught AI what a human being is supposed to feel and say and need.
The Training Data Problem Is Not a Technical Problem
Most large language models were trained on text from the internet. The internet, as a repository of human expression, reflects the distribution of power that produced it: predominantly Western, predominantly white, predominantly middle-class in its assumptions about what normal looks like, what distress looks like, what recovery looks like. The American Psychological Association’s 2024 health advisory on generative AI in mental health explicitly states that AI models trained on vast amounts of internet data reflect significant biases related to race, culture, gender, and ethnicity, leading to significant issues with accurate and culturally competent outputs.
An AI trained on that data does not just have gaps. It has assumptions baked in at the level of architecture. And those assumptions show up in every response it generates. When a Black college student reaches out to a general AI mental health tool in the grip of anxiety, and the tool responds with language calibrated to a different cultural reality, the result is not just unhelpfulness. It is a form of misrecognition that, for communities already navigating decades of institutional misrecognition, carries a weight the engineers who built the tool may never fully understand.
An AI trained on public internet data does not just have gaps. It has assumptions baked in at the level of architecture.
Sycophancy Is Not Support
There is a second problem with general AI mental health tools receiving increasing attention from researchers: the tendency of large language models toward sycophancy, the systematic inclination to agree with and validate whatever the user believes or feels, rather than offering the honest, sometimes challenging engagement that genuine support requires.
The APA’s health advisory formally identifies this as a significant clinical risk, noting that sycophantic chatbots can reinforce confirmation bias and maladaptive beliefs by validating users’ views rather than challenging them, and that when biased content combines with sycophantic engagement, the result can be a digital echo chamber that amplifies and entrenches users’ existing beliefs, even when those beliefs are false. Research published in 2025 using a Bayesian framework to quantify AI sycophancy found that large language models shift their stated beliefs to align with users’ judgments in ways that lead to increased errors and irrational belief updates. In a mental health context, where a person may be seeking validation of distorted thinking, this is not a minor design flaw. It is potentially dangerous.
What Ethical AI Mental Health Support Actually Requires
I believe ethical AI mental health support requires three things that most current tools do not have.
First, a closed, clinically grounded corpus. The AI must be trained on material that reflects the clinical expertise and cultural competence of actual practitioners, not the ambient noise of the internet. The intelligence in the AI must be genuinely clinical in origin.
Second, independent ethical oversight. Not an internal ethics team that answers to the product team. An independent board of clinicians, data scientists, legal experts, and community advocates who have real authority over model updates and feature decisions. The conflict of interest between product growth and clinical safety is real and requires a structural solution.
Third, what I would call cultural architecture: the intentional design of every feature, every prompt, every response pattern to reflect the specific lived experiences of the communities being served. A systematic review published in Symmetry in 2025 examined AI-based mental health chatbots across design patterns, cultural balance, and ethical dimensions, and found that cultural representativeness remains one of the most significant gaps in current implementations, with most systems defaulting to Western clinical frameworks regardless of the user’s background.
The intelligence in the AI must be genuinely clinical in origin. Not filtered from the internet. Built from practice.
Why Kamilah 2.0 Was Built the Way It Was
Kamilah 2.0 is my attempt to build AI that gets these things right. Its corpus is closed: trained exclusively on my published works, clinical frameworks, and de-identified therapeutic insights developed over thirty years of practice with the specific communities this platform is designed to serve. It is not trained on public internet data. It cannot be, because the intelligence that needs to be in this tool is not the intelligence of the internet. It is the intelligence of a clinician who has spent thirty years in the room with the people she is trying to serve.
Its ethical governance is independent: a Neuro-Ethics Board reviews every significant model update. And its cultural architecture is not an add-on. It is the reason the platform exists. I am under no illusion that we have solved all the problems I have named here. We are building carefully and iterating honestly. But we are building in the only direction that produces tools that actually help: from the inside out, from clinical integrity outward, from community need inward.
Kamilah M. Woodson, Ph.D., is a licensed clinical psychologist, Founder and CEO of Epiphany Psychological Solutions, and a nationally recognized speaker on the ethics of AI in mental health.. Learn more at epiphanypsy.com.
References
American Psychological Association. (2024). Health advisory: Use of generative AI chatbots and wellness applications for mental health. https://www.apa.org/topics/artificial-intelligence-machine-learning/health-advisory-chatbots-wellness-apps
Atwell, K., et al. (2025). BASIL: Bayesian assessment of sycophancy in LLMs. arXiv. https://doi.org/10.48550/arxiv.2508.16846
Clegg, K. A. (2025). Shoggoths, sycophancy, psychosis, oh my: Rethinking large language model use and safety. Journal of Medical Internet Research, 27, e87367.
Goldie, J., Dennis, S., Hipgrave, L., & Coleman, A. (2025). Practitioner perspectives on the uses of generative AI chatbots in mental health care: Mixed methods study. JMIR Human Factors, 12(1), e71065.
Iftikhar, Z., Xiao, A., Ransom, S., Huang, J., & Suresh, H. (2025). How LLM counselors violate ethical standards in mental health practice. arXiv.
Mwamba, C., et al. (2025). Symmetric therapeutic frameworks and ethical dimensions in AI-based mental health chatbots (2020-2025): A systematic review of design patterns, cultural balance, and structural symmetry. Symmetry, 17(7), 1082. https://www.mdpi.com/2073-8994/17/7/1082
