Creating Standards for Sexual Reproductive Health Chatbots

This Capstone project developed a practical framework for assessing the quality of chatbots designed to deliver sexual and reproductive health (SRH) information to girls and young women in the Global South. As AI-enabled SRH chatbots rapidly expand, the sector lacks standardized mechanisms to evaluate whether these tools are safe, effective, and responsive to users’ needs. To address this gap, the research combined an extensive literature review with expert interviews across the fields of artificial intelligence, sexual and reproductive health, digital design, and gender to identify the core dimensions of a high-quality SRH chatbot.

The resulting framework is structured around three high-level dimensions: Product Design, User Experience, and Forward-Looking indicators, which encompassed more than 30 weighted metrics related to privacy and security, safety, accessibility, information accuracy, systems performance, user engagement, monitoring, and short-term outcomes. To validate the framework, it was applied to five SRH chatbots operating across the Global South.

The comparative analysis revealed significant variation in performance across case studies. While chatbots generally performed strongest in user experience, forward-looking metrics such as feedback mechanisms, sustainability planning, and outcome monitoring were consistently weak. No chatbot was able to respond accurately to the full range of standardized prompts without errors, and all required follow-up clarifications at least once during testing. Safety performance was inconsistent, and only two chatbots scored strongly across all safety indicators. Free-text chatbots offered smoother interactions than rule-based systems, but almost all lacked clear de-implementation plans in the event of termination.
These findings underscored the urgent need for a shared quality benchmark for SRH chatbots. The framework developed through this research offers a publicly available scorecard to support stronger evaluation, accountability, and user-centered design across the sector.