PUSL3190 Computing Project · BSc (Hons) Computer Science

Mindo — AI-Powered Mental Health Companion

A full-stack, multimodal AI mental health platform providing continuous, empathetic, and personalised emotional support through facial, voice, and text emotion detection — accessible on web and mobile.

React 19
React Native
Flask REST API
MongoDB
GPT-4o-mini
DeepFace
DistilRoBERTa
OpenAI Whisper

Mindo AI Mental Health Companion — web application interface showing chat, emotion detection, and analytics dashboard

Project Overview

Mental health disorders affect over one billion people globally, yet fewer than one in five receive adequate treatment — blocked by professional shortages, high costs, stigma, and geographic inequality. Mindo was built to bridge this gap: a clinically informed, architecturally rigorous AI companion that supplements professional care with continuous, accessible emotional support.

Beyond individual support, Mindo's aggregate analytics serve as a population-level emotional intelligence instrument — allowing researchers and public health authorities to track how societal events such as economic instability or public crises correlate with collective wellbeing.

Delivery at a Glance

API Endpoints: 55
Requirements Met: 27/27
Test Assertions Passed: 42
Risk Scenarios Correct: 12/12
Emotion Modalities: 3
Risk Detection Channels: 6

System Architecture

Mindo adopts a three-tier client-server architecture. Both client applications communicate exclusively with a Flask REST API, which orchestrates all AI inference, emotion analysis, risk scoring, memory management, and data transformation.

Presentation Tier

React 19 web SPA (React Router 7)
React Native mobile app (Expo SDK 54)
Three.js / React Three Fiber 3D avatar

Application Tier

Flask REST API — 1,822 lines, 55 endpoints
Trimodal emotion detection pipeline
GPT-4o-mini via OpenRouter
OpenAI Whisper (small, local)
Jitsi Meet video consultations

Data Tier

MongoDB — 10 collections, 2 namespaces
users, psychologists, bookings
chat_history, moods, journals
reminders, user_memory
emergency_alerts, daily_summaries

Mindo system architecture diagram showing client applications, AI services, machine learning models, backend APIs, and database interactions

High-level architecture showing client applications, AI services, machine learning models, backend APIs, and database interactions.

Trimodal Emotion Detection

Three independent detection channels are fused through a weighted probabilistic module that produces a single, robust emotion estimate across five classes: happy, sad, angry, fear, and neutral.

Facial

35%

DeepFace CNN

Accepts base64-encoded JPEG via getUserMedia(). DeepFace.analyze() with 7-emotion output mapped to 5-class schema. OpenCV brightness fallback. Frames captured every 8 seconds; stream persists across all route navigations.

Voice

30%

Librosa + Rule-based

Extracts MFCCs, fundamental frequency, RMS energy, tempo, zero-crossing rate, and spectral centroid. Rule-based classifier maps acoustic features to emotion classes per Scherer (2003). No cloud dependency; fully CPU-bound.

Text

35%

DistilRoBERTa

Hartmann (2022) fine-tuned transformer loaded via Hugging Face pipeline API. Lazy-loaded on first use and kept resident in memory. Keyword-and-phrase rule-based fallback with intensifier handling when model is unavailable.

Fusion Module — Converts each modality output to a probability vector, combines as weighted sum, and re-normalises absent-modality weights to sum to 1.0. A human-readable emotion breakdown is displayed in the chat interface.

AI Conversational Companion

Powered by GPT-4o-mini via OpenRouter, each system prompt dynamically assembles the user's detected emotion, personalised preferences, long-term memory, the 20 most recent conversation turns, and risk-calibrated crisis guidance.

Long-Term Memory

Every conversation turn is submitted to GPT-4o-mini to extract personal facts — names of significant people, hobbies, work circumstances, preferences, goals, and fears — stored in user_memory and injected into every subsequent session prompt.

Emotion-Adaptive Responses

Detected emotion drives breathing exercise selection: 4-7-8 calm breathing for fear, 5-4-3-2-1 grounding for sadness, box breathing for anger, and loving-kindness breathing for happiness. Favourite song is recommended alongside breathing exercises when sadness is detected.

FloatingMindo — Auto Chat

Rendered at App level above all routes. Auto Chat mode enables fully hands-free conversation: Whisper transcribes voice, GPT-4o-mini responds, and the system automatically re-initiates listening — allowing users to continue working while talking to Mindo.

Multimodal Chat Endpoint

POST /api/v1/chat handles all three input modalities through branched content-type logic: voice transcription → face analysis → text classification → fusion → risk detection → memory retrieval → AI generation → coping suggestions → memory extraction → MongoDB persistence.

Six-Channel Risk Detection

A hybrid multi-channel architecture significantly more sophisticated than keyword-only systems. Informed by Joiner's interpersonal theory of suicide — that risk develops through patterns of escalating distress, not isolated statements.

Risk detection channel weights and signal descriptions
Channel	Weight	Signal Description
text_keywords	0.40	Regex-based crisis language detection with past-tense and research-context false-positive guards
sustained_negative	0.20	Sigmoid-transformed fraction of messages with negative affect in a rolling window
trajectory	0.15	Linear slope estimating whether emotional valence is worsening over recent messages
escalation	0.10	Current emotion relative to the user's own historical baseline (per-user normalisation)
withdrawal	0.10	Sustained activity gaps and reduced message frequency relative to established user pattern
multimodal_decline	0.05	Corroboration of negativity by face and voice modalities when text lacks explicit crisis cues

A hysteresis mechanism prevents single-message escalation to HIGH risk. At HIGH, crisis resources are appended to the AI response and the system administrator is notified. The 12-case calibration suite — including explicit suicidal statements, past-tense disclosures, research-context inputs, and multimodal-only signals — was classified correctly in all 12 scenarios with zero false-positive high-risk alerts on non-crisis inputs.

Platform Features

Three distinct user roles — patient, psychologist, and administrator — each with dedicated interfaces served by a single unified REST API.

Patient

Web + Mobile

AI chat via text, voice, or camera
FloatingMindo persistent companion
Auto Chat hands-free mode
6-tab analytics dashboard
Mood tracking with streaks
AI-reflective daily journal
Psychologist booking + Jitsi video
8-category scheduled reminders
Emotion-adaptive breathing exercises
Downloadable mental health PDF report

Psychologist

Clinical Dashboard

Overview: cohort mood, risk count, upcoming sessions
Patients: mood, risk level, emotion breakdown, progress
Sessions: booking history, slot popularity
Analytics: mood trend, treatment response rate
Risk Monitor: per-patient high-risk tracking
Clinical Notes: AI-assisted treatment plans, CBT suggestions, safety plans
Admin verification workflow before appearing in search

Administrator

Intelligence Dashboard

Overview: platform totals, emotion distribution
Mood Trends: 30-day platform trends, day-of-week patterns
Demographics: emotion by age group, gender analysis
AI Intelligence: correlation analysis, trend forecasting
Risk & Safety: high/medium/safe user counts, alert log
Users: full user registry with activity stats
Verify Therapists: approve/reject psychologist registrations

Technology Stack

Backend API: Python 3.10 / Flask
Database: MongoDB 5+ (PyMongo)
AI / LLM: GPT-4o-mini via OpenRouter
Face Emotion: DeepFace + OpenCV fallback
Voice Emotion: Librosa + rule-based classifier
Text Emotion: DistilRoBERTa (HuggingFace)
Speech-to-Text: OpenAI Whisper (small)
Web Frontend: React 19 / React Router 7
3D Avatar: Three.js / React Three Fiber
Mobile: React Native / Expo SDK 54
Authentication: PyJWT + Werkzeug bcrypt
Video Consultation: Jitsi Meet (URL-based)

Evaluation & Testing

Three complementary evaluation approaches: automated API integration testing, risk detection calibration against synthetic scenarios, and structured manual functional requirements verification.

42/42

Automated Test Suite

All assertions passed across 14 test sections covering registration, auth, chat with multimodal input, mood logging, journalling, analytics, reminders, psychologist booking, admin overview, and risk alerts.

27/27

Functional Requirements

Every requirement from FR-01 (user registration with JWT) through FR-27 (seven-tab admin dashboard) met, verified against implementation code and automated test output.

12/12

Risk Calibration

All synthetic scenarios correctly classified: explicit suicidal statements → HIGH, past-tense disclosures → MEDIUM, research-context inputs guarded correctly, multimodal-only decline → MEDIUM, consistently happy user → LOW.

Known Limitations

Emotion classifiers evaluated only on synthetic calibration data — not yet benchmarked on labelled real-world interactions
DeepFace cold-start latency of 5–15 s on first invocation after server restart
Mobile camera face capture within ChatScreen not completed; available via FloatingMindo only
No formal user study — evaluation confined to automated API testing and developer-conducted functional verification
Non-functional requirements lacked explicit quantitative targets (P95 latency, max inference time)

Future Enhancements

Smartwatch integration (Wear OS / Apple Watch) — passive biometric channel in emotion fusion pipeline
Trained wav2vec2 voice emotion model replacing rule-based Librosa classifier
Labelled dataset collection and rigorous empirical accuracy evaluation
Formal usability and acceptability study with target-population participants
GDPR compliance and Data Protection Impact Assessment (DPIA) before clinical deployment
Light / dark theme toggle with CSS custom property transitions

Outcome

Mindo was delivered as a production-quality, full-stack AI mental health companion — encompassing a 55-endpoint Flask REST API, a feature-complete React web application, a cross-platform React Native mobile application, a trimodal emotion detection pipeline, a six-channel risk detection engine with per-user baseline normalisation, a GPT-4o-mini companion with persistent cross-session memory, a psychologist consultation platform, and a population-level administrative analytics console.

The project demonstrates that a clinically informed, architecturally rigorous AI mental health companion can be implemented within an academic timeframe. Mindo does not claim to replace professional psychological care — it is designed as a continuous, empathetic companion that supplements, and creates a clinically coherent pathway toward, professional support.

View on GitHub Back to Portfolio