Interpreting Arabic sign alphabet by utilizing a glove with sensors

Ahmed Saeed Ibrahim Al-Obaidi; Raid Rafi Omar Al-Nima; Tingting Han

doi:10.53730/ijhs.v6nS6.12018

Authors

Ahmed Saeed Ibrahim Al-Obaidi
techengahmed0@gmail.com
Northern Technical University, Mosul, Iraq
Raid Rafi Omar Al-Nima Northern Technical University, Mosul, Iraq
Tingting Han Birkbeck, University of London, United Kingdom

Keywords:

deep learning, long short-term memory, recurrent neural network, sign language

Abstract

People who are deaf or dumb in Arab communities face several challenges. The most important challenge is to communicate with people. In this study, a new approach for identifying the alphabet in the Iraqi Sign Language (IrSL) is proposed, which makes use of a suggested deep neural network called the Deep Recurrent Alphabet Sign Language (DRASL). It utilizes the Long Short-Term Memory (LSTM) technique for classifying the outputs and recognizing the alphabet in the SL. The dataset is constructed with the use of a glove that is coupled to flex sensors on each finger; each sensor gives a variable value based on the curvature ratio of the fingers. The sensors were connected to an Arduino which was then linked to a computer to transfer the data we collected. The data were divided into three groups, which had 29 different movements. All of these groups had a remarkably high accuracy equal to 100%.deep learning

Downloads

Download data is not yet available.

References

A. A. Gomaa and R. Gla Elrayes, “Egyptian Sign Language Recognition Using CNN and LSTM,” 2021.

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural,” Handbook of Approximation Algorithms and Metaheuristics. pp. 1–1432, 2007, doi: 10.1201/9781420010749.

A. Suri, S. K. Singh, R. Sharma, P. Sharma, N. Garg, and R. Upadhyaya, “Development of Sign Language using Flex Sensors”, Proceedings - International Conference on Smart Electronics and Communication, ICOSEC 2020, no.Icosec,pp.102–106,2020,doi: 10.1109/ICOSEC49089.2020.9215392.

A. Wienholz, D. Nuhbalaoglu, M. Steinbach, A. Herrmann, and N. Mani, “Phonological priming in German Sign Language: An eye tracking study using the Visual World Paradigm,” Sign Language & Linguistics, vol. 24, no. 1, pp. 4–35, 2019, [Online]. Available: https://www.jbe-platform.com/content/journals/10.1075/sll.19011.wie.

Arwa Hamid Salih Hamdany, Lubab H. Albak, and Raid Rafi Omar Al-Nima, “Wireless Waiter Robot”, TEST Engineering & Management, The Mattingley Publishing Co., Inc., Vol. 81, pp. 2486-2494, November-December 2019.

B. G. Lee, T. W. Chong, and W. Y. Chung, “Sensor fusion of motion-based sign language interpretation with deep learning,” Sensors (Switzerland), vol. 20, no. 21, pp. 1–17, 2020, doi: 10.3390/s20216256.

C. Selvarathi, P. Indu, B. Kavyadharshini, S. Logesh Kumar, and R. Mohamed Yasher, “Human computer interaction using hand gesture recognition,” International Journal of Advanced Trends in Computer Science and Engineering, vol. 9, no. 2, pp. 1600–1603, 2020, doi: 10.30534/ijatcse/2020/106922020.

D. Economou, M. G. Russi, I. Doumanis, M. Mentzelopoulos, V. Bouki, and J. Ferguson, “Using serious games for learning british sign language combining video, enhanced interactivity, and vr technology,” Journal of Universal Computer Science, vol. 26, no. 8, pp. 996–1016, 2020, doi: 10.3897/jucs.2020.053.

D. S. Battina, “Innovative study of an AI voice based smart Device to assist deaf people in understanding and responding to their body language,” no. March, 2022.

E. Abraham, A. Nayak, and A. Iqbal, “Real-Time Translation of Indian Sign Language using LSTM,” 2019 Global Conference for Advancement in Technology, GCAT 2019, pp. 1–5, 2019, doi: 10.1109/GCAT47503.2019.8978343.

G. Halvardsson, J. Peterson, C. Soto-Valero, and B. Baudry, “Interpretation of Swedish Sign Language Using Convolutional Neural Networks and Transfer Learning”, SN Computer Science, vol. 2, no. 3, pp. 1–15, 2021, doi: 10.1007/s42979-021-00612-w.

G. Latif, J. Alghazo, N. Mohammad, and R. Alghazo, “Communicating with the Deaf and Hard of Hearing through Automatic Arabic Sign Language Translator,” Journal of Physics: Conference Series, vol. 1962, no. 1, 2021, doi: 10.1088/1742-6596/1962/1/012055.

In-Depth: Interfacing Flex Sensor with Arduino”. [Online]. Available: https://lastminuteengineers.com/flex-sensor-arduino-tutorial/ (accessed May 27, 2022).

J. Kotowicz, B. Woll, and R. Herman, “Adaptation of the British Sign Language Receptive Skills Test into Polish Sign Language,” Language Testing, vol. 38, no. 1, pp. 132–153, 2021, doi: 10.1177/0265532220924598.

K. Greff, R. K. Srivastava, J. Koutnik, B. R. Steunebrink, and J. Schmidhuber, “LSTM: A Search Space Odyssey,” IEEE Transactions on Neural Networks and Learning Systems, vol. 28, no. 10, pp. 2222–2232, 2017, doi: 10.1109/TNNLS.2016.2582924.

K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, pp. 1–14, 2015.

K. Smagulova and A. P. James, “A survey on LSTM memristive neural network architectures and applications,” European Physical Journal: Special Topics, vol. 228, no. 10, pp. 2313–2324, 2019, doi: 10.1140/epjst/e2019-900046-x.

Lubab H. Albak, Raid Rafi Omar Al-Nima, and Arwa Hamid Salih, “Palm print Verification Based Deep Learning”, TELKOMNIKA Telecommunication, Computing, Electronics and Control, Vol. 19, No. 3, 2021.

M. G. Grif and Y. K. Kondratenko, “Development of a software module for recognizing the fingerspelling of the Russian Sign Language based on LSTM,” Journal of Physics: Conference Series, vol. 2032, no. 1, 2021, doi: 10.1088/1742-6596/2032/1/012024.

M. Iqbal, E. Supriyati, and T. Listyorini, “SIBI Blue: Developing Indonesian Sign Language Recognition System Based On The Mobile Communication Platform,” International Journal of Information

M. M. Kamruzzaman, “Arabic Sign Language Recognition and Generating Arabic Speech Using Convolutional Neural Network,” Wireless Communications and Mobile Computing, vol. 2020, 2020, doi: 10.1155/2020/3685614.

M. R. Khalil, M. S. Majeed, and R. R. Omar, “Personal Identification With Iris Patterns”, AL-Rafidain Journal of Computer Sciences and Mathematics, College of Computer Sciences and Math / University of Mosul / Iraq, Vol. 6, No. 1, 2009.

M. T. Al-Kaltakchi, R. R. Omar, H. N. Abdullah, T. Han, and J. A. Chambers, “Finger Texture Verification Systems Based on Multiple Spectrum Lighting Sensors With Four Fusion Levels”, Iraqi Journal of Information & Communications Technology, Vol. 1, Issue 3, 2018.

Mabroukah M. A. Abuqadumah, Musab A. M. Ali, and Raid R. O. Al-Nima, “Personal Authentication Application Using Deep Learning Neural Network”, in 16th IEEE International Colloquium on Signal Processing & its Applications (CSPA), Langkawi, Malaysia, pp. 186-190, 2020.

Marwa MohamedSheet AL-Hatab, Raid Rafi Omar Al-Nima, Ilaria Marcantoni, Camillo Porcaro, and Laura Burattini, “Comparison Study Between Three Axis Views of Vision, Motor and Pre-Frontal Brain Activities”, Journal of Critical Reviews, Vol. 7, Issue 5, pp. 2598-2607, 2020. doi:10.31838/jcr.07.05.430

Musab T. S. Al-Kaltakchi, Raid R. O. Al-Nima, and Mohammed A. M. Abdullah, “Comparisons of extreme learning machine and backpropagation-based i-vector approach for speaker identification”, Turkish Journal of Electrical Engineering & Computer Sciences, The Scientific and Technological Research Council of Turkey, Vol. 38, No. 3, pp. 1236-1245, 2020.

Mustafa, A. R., Ramadany, S., Sanusi, Y., Made, S., Stang, S., & Syarif, S. (2020). Learning media applications for toddler midwifery care about android-based fine motor development in improving midwifery students skills. International Journal of Health & Medical Sciences, 3(1), 130-135. https://doi.org/10.31295/ijhms.v3n1.290

N. C. Camgoz, S. Hadfield, O. Koller, H. Ney, and R. Bowden, “Neural Sign Language Translation,” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 7784–7793, 2018, doi: 10.1109/CVPR.2018.00812.

N. Tajbakhsh et al., “Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning”, IEEE Transactions on Medical Imaging, vol. 35, no. 5, pp. 1299–1312, 2016, doi: 10.1109/TMI.2016.2535302.

N. Tavakoli, “Modeling genome data using bidirectional LSTM,” Proceedings - International Computer Software and Applications Conference, vol. 2, no. October, pp. 183–188, 2019, doi: 10.1109/COMPSAC.2019.10204.

N. Tavakoli, “Modeling genome data using bidirectional LSTM,”Proceedings - International Computer Software and Applications Conference, vol. 2, no. October, pp. 183–188, 2019, doi:10.1109/COMPSAC.2019.10204.

R. R. Al-Nima and S. Kasim, “Picture Recognition by Using Linear Associative Memory Neural Network”, Tikrit Journal of Pure Science, Vol. 13, No. 3, 2008.

R. R. Al-Nima, “Design a Biometric Identification System Based on the Fusion of Hand Geometry and Backhand Patterns”, Iraqi Journal of Statistical Science, Vol. 17, 2010.

Raid R. O. Al-Nima, Tingting Han, Taolue Chen, Satnam Dlay, and Jonathon Chambers, “Finger Texture Biometric Characteristic: a Survey”, arXiv preprint arXiv:2006.04193, 2020.

Raid Rafi Omar Al-Nima, Tingting Han, Saadoon Awad Mohammed Al-Sumaidaee, Taolue Chen, and Wai Lok Woo, “Robustness and performance of Deep Reinforcement Learning”, Elsevier, Applied Soft Computing, Vol. 105, 2021.https://doi.org/10.1016/j.asoc.2021.107295

S. Johnny and S. J. Nirmala, “Sign Language Translator Using Machine Learning”, SN Computer Science, vol. 3, no. 1, 2022, doi: 10.1007/s42979-021-00896-y.

S. K. Ko, C. J. Kim, H. Jung, and C. Cho, “Neural sign language translation based on human keypoint estimation,” Applied Sciences (Switzerland), vol. 9, no. 13. 2019, doi: 10.3390/app9132683.

S. Sharma and K. Kumar, “ASL-3DCNN: American sign language recognition technique using 3-D convolutional neural networks,” Multimedia Tools and Applications, vol. 80, no. 17, pp. 26319–26331, 2021, doi: 10.1007/s11042-021-10768-5.

S. Sharma and S. Singh, “Vision-based hand gesture recognition using deep learning for the interpretation of sign language,” Expert Systems with Applications, vol. 182, no. June, p. 115657, 2021, doi: 10.1016/j.eswa.2021.115657.

S. Theodorakis, V. Pitsikalis, and P. Maragos, “Dynamic-static unsupervised sequentiality, statistical subunits and lexicon for sign language recognition”, Image and Vision Computing, vol. 32, no. 8, pp. 533–549, 2014, doi: 10.1016/j.imavis.2014.04.012.

Suryasa, I. W., Rodríguez-Gámez, M., & Koldoris, T. (2021). Get vaccinated when it is your turn and follow the local guidelines. International Journal of Health Sciences, 5(3), x-xv. https://doi.org/10.53730/ijhs.v5n3.2938

Technology,vol.1,no.1,2017,[Online].Available:https://www.researchgate.net/publication/315782947.

W. H. Organization, “Deafness and hearing loss”. [Online]. Available:https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss (accessed Feb. 22, 2022).

Workshops, vol. 2018-June, pp. 2145–2154, 2018, doi:10.1109/CVPRW.2018.00280.

X. Jiang, B. Hu, S. Chandra Satapathy, S. H. Wang, and Y. D. Zhang, “Fingerspelling Identification for Chinese Sign Language via AlexNet-Based Transfer Learning and Adam Optimizer,” Scientific Programming, vol. 2020, 2020, doi: 10.1155/2020/3291426.

Y. Ye, Y. Tian, M. Huenerfauth, and J. Liu, “Recognizing american sign language gestures from within continuous videos,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition