Improved remote mental health illness assessment and detection using facial emotion detection and speech emotion detection

Vasundhara Rathod; Abhishek Chohan; Sakshi Nema; Adhney Nawghare; Prateeksha Devikar; Rahul Agrawal

doi:10.53730/ijhs.v6nS2.7508

Authors

Vasundhara Rathod Department of Computer Science and Engineering Shri Ramdeobaba College of Engineering and Management, Nagpur, India
Abhishek Chohan Department of Computer Science and Engineering Shri Ramdeobaba College of Engineering and Management, Nagpur, India
Sakshi Nema Department of Computer Science and Engineering Shri Ramdeobaba College of Engineering and Management, Nagpur, India
Adhney Nawghare Department of Computer Science and Engineering Shri Ramdeobaba College of Engineering and Management, Nagpur, India
Prateeksha Devikar Department of Computer Science and Engineering Shri Ramdeobaba College of Engineering and Management, Nagpur, India
Rahul Agrawal Department of Computer Science and Engineering Shri Ramdeobaba College of Engineering and Management, Nagpur, India

Keywords:

emotion recognition speech, emotions through facial expressions, psychological disorders, remote diagnosis, Twitter sentiment analysis

Abstract

The use of Artificial Intelligence in the healthcare sector has been recently observed. Projects are being made that integrate AI and therapeutic sessions[1]. A new area of study evolved where doctors, along with technicians, collaborate to create projects which will help give the old school therapy an advanced technical form. This study uses the original therapy techniques for mental health assessment and integrates it with machine learning models for facial emotion recognition and speech pattern recognition to get a better understanding of a patient’s mental health condition and help them deal with it. This project assists the patient in the diagnosis of 11 different mental health conditions where the patient’s emotional state is taken into consideration while diagnosis. Patients’ social interactions are also being checked and analyzed regularly. This project calculates a verdict which declares whether the patient suffers from the diagnosed illness or whether they should retake the tests. In addition, the project maintains records of the patient's emotional and mental health journey.

Downloads

Download data is not yet available.

References

T. C. P. O. K. K. J. M. C. O. M. J. V. L. P. Rachel M. Satter, "Avatar-based simulation in the evaluation of diagnosis and management of mental health disorders in primary care," Journal of Biomedical Informatics, vol. 45, no. 6, pp. 1137-1150, 2012.

"Prevalence of stress, anxiety, depression among the general population during the COVID-19 pandemic: a systematic review and meta-analysis.," Global Health, vol. 57, no. 16, 2020.

G. L. M. F. C. I. F. a. J. M. M.S. Bartlett, "Automatic recognition of facial actions in spontaneous expressions.," Journal of Multimedia, 2006.

T. Rashid, ". Convolutional neural networks based method for improving facial expression recognition," International symposium on intelligent systems technologies and applications, pp. 73-84, 2016.

S. P. M. a. B. P. Roopa, "Speech emotion recognition using deep learning," Int. J. Recent Technol. Eng.

W. J. D. a. L. T. Lim, "Speech emotion recognition using convolutional and recurrent neural networks," Asia-Pacific signal and information processing association annual summit and conference, 2016.

S. S. I. I. Y. a. W. N. Hamsa, "Emotion recognition from speech using wavelet packet transform cochlear filter bank and random forest classifier," IEEE, pp. 96994-97006, 2020.

J. B.-G. S. S. R. a. P. B. Hansen, "Getting started with SUSAS: a speech under simulated and actual stress database," In Eurospeech, vol. 97, no. 4, pp. 1743-46, 1997.

Z. L. X. Z. Z. U. M. D. J. a. A. M. Peng, "Speech emotion recognition using 3d convolutions and attention-based sliding recurrent networks with auditory front-ends," IEEE Access, 8, pp. 16560-16572, 2020.

R. J. A. G. B. L. L. a. R. M. Fulmer, "Using psychological artificial intelligence (Tess) to relieve symptoms of depression and anxiety: randomized controlled trial," JMIR mental health, vol. 5, no. 4, p. 64, 2018.

B. S. S. a. S. V. Inkster, "An empathy-driven, conversational artificial intelligence agent (Wysa) for digital mental well-being: real-world data evaluation mixed-methods study," JMIR mHealth and uHealth, 2018.

C. S. S. a. Y. T. Ittichaichareon, "Speech recognition using MFCC," International conference on computer graphics, simulation and modeling, pp. 135-138, 2012.

R. C. T. O. P. K. K. M. J. O. C. J. M. a. P. V. Satter, "Avatar-based simulation in the evaluation of diagnosis and management of mental health disorders in primary care.," Journal of Biomedical Informatics, vol. 45, no. 6, pp. 1137-1150, 2012.

S. B. C. C. W. a. H. L. Šabanović, "PARO robot affects diverse interaction modalities in group sensory therapy for older adults with dementia," IEEE 13th international conference on rehabilitation robotics (ICORR) , pp. 1-6, 2013.

Kargarbideh, "Design and Development of the eBear: A Socially Assistive Robot for Elderly People with Depression," Doctoral dissertation, University of Denver, 2016.

K. S. R. a. W. J. Choo, "The PHQ-9," Journal of General Internal Medicine, vol. 16, no. 9, pp. 606-613, 2001.

P. a. C. A. Carrier, "Challenges in representation learning: Facial expression recognition challenge," Kaggle, 2021.

K. P. T. a. P. C. Potdar, "A comparative study of categorical variable encoding techniques for neural network classifiers.," International journal of computer applications, vol. 175, no. 4, pp. 7-9, 2017.

Y. K. K. a. F. C. LeCun, "Convolutional networks and applications in vision," Proceedings of 2010 IEEE international symposium on circuits and systems, pp. 253-256, 2010.

L. H. M. a. K. K. Ankile, "Deep Convolutional Neural Networks: A survey of the foundations, selected improvements, and some current applications," 2020.

D. a. D. L. Yu, " Automatic Speech Recognition," Springer London limited, 2016.

S. a. R. F. Livingstone, "RAVDESS Emotional speech audio," Kaggle, 2019.

E. J. Lok, "Toronto emotional speech set (tess)," Kaggle, 2019.

C. a. T. G. Panagiotakis, "A speech/music discriminator using RMS and zero-crossings.," 2002 11th European Signal Processing Conference. IEEE., pp. 1-4, 2002.

M. N. A. S. A. a. S. D. Kattel, "Chroma feature extraction," Conference: Chroma Feature Extraction using Fourier Transform, 2019.

C. a. T. G. Panagiotakis, "A speech/music discriminator using RMS and zero-crossings," 2002 11th European Signal Processing Conference, IEEE, 2002.

B. Thornton, "Audio recognition using mel spectrograms and convolution neural networks," 2019.

M. K. Z. a. V. D. Kos, "Acoustic classification and segmentation using modified spectral roll-off and variance-based features," Digital Signal Processing, vol. 23, no. 2, pp. 659-674, 2013.

M. F. M. B. E. P. D. a. T. A. Massar, "Fast computation of spectral centroids," Advances in Computational Mathematics, vol. 35, no. 1, pp. 83-97, 2011.

T. J. B. S. J. K. S. T. L. a. C. H. Emser, "Assessing ADHD symptoms in children and adults: evaluating the role of objective measures.," Behavioral and Brain Functions, vol. 14, no. 1, pp. 1-14, 2018.

P. Gillig, "Dissociative identity disorder," A controversial diagnosis. Psychiatry (Edgmont), vol. 6, no. 3, p. 24, 2009.

R. Hirschfeld, "The Mood Disorder Questionnaire: a simple, patient-rated screening instrument for bipolar disorder.," Primary care companion to the Journal of Clinical Psychiatry, vol. 4, no. 1, p. 9, 2002.

L. B. J. T. S. B. E. K. J. R. D. W. E. G. K. C. E. J. P. a. P. M. Horowitz, "Ask Suicide-Screening Questions (ASQ): a brief instrument for the pediatric emergency department. Archives of pediatrics & adolescent medicine,," vol. 166, no. 12, pp. 1170-1176.

C. T. J. G. D. a. B. S. Lancaster, "Posttraumatic stress disorder: overview of evidence-based assessment and treatment. Journal of clinical medicine," vol. 5, no. 11, p. 105, 2016.

E. Freeman, "Premenstrual syndrome and premenstrual dysphoric disorder: definitions and diagnosis.," Psychoneuroendocrinology, vol. 28, pp. 25-37, 2003.

B. R. P. J. a. M. J. Rapp, "Evidence-based assessment of obsessive–compulsive disorder. Journal of central nervous system disease," 2016.

H. M. U. J. A. Y. F. K. F. K. a. K. H. Takeuchi, "A questionnaire-based study of the views of schizophrenia patients and psychiatric healthcare professionals in Japan about the side effects of clozapine," Clinical Psychopharmacology and Neuroscience, vol. 14, no. 3, p. 286, 2016.

W. S. S. a. Y. M. Mahmud, "Postpartum depression: a survey of the incidence and associated risk factors among Malay women in Beris Kubor Besar, Bachok, Kelantan.," The Malaysian journal of medical sciences, vol. 9, no. 1, p. 41, 2002.

H. Murugan, " Speech emotion recognition using cnn.," International Journal of Psychosocial Rehabilitation, vol. 24, no. 6, 2020.

U. Barańczuk, "The five factor model of personality and emotion regulation: A meta-analysis. Personality and Individual Differences," vol. 139, pp. 217-227, 2019.

S. Morgan, "Categorical and dimensional ratings of emotional speech: Behavioral findings from the Morgan emotional speech set.," Journal of Speech, Language, and Hearing Research, vol. 62, no. 11, pp. 4015-4029, 2019.

G. B. R. A. S. A. F. a. T. E. Mancini, "Recognition of facial emotional expressions among Italian pre-adolescents, and their affective reactions," Frontiers in psychology, vol. 9, p. 1303, 2018.

R. R. a. A. P. Manguri, "Twitter sentiment analysis on worldwide COVID-19 outbreaks," Kurdistan Journal of Applied Research, pp. 54-65, 2020.

B. H. K. a. M. P. Burns, "Effective treatment for mental disorders in children and adolescents.," Clinical child and family psychology review, vol. 2, no. 4, pp. 199-254, 1999.

R. a. P. A. Poojary, "Comparative Study of Model Optimization Techniques in Fine-Tuned CNN Models.," International Conference on Electrical and Computing Technologies and Applications (ICECTA), pp. 1-4, 2019.

E. J. Lok, "Crowd Sourced Emotional Multimodal Actors Dataset (CREMA-D)," Kaggle, 2019.

T. Sunkaraneni, "SAVEE database," Kaggle, 2019.