A fully automatic method for recognizing hand configurations of Brazilian sign language
Costa Filho, Cícero Ferreira Fernandes; Souza, Robson Silva de; Santos, Jonilson Roque dos; Santos, Bárbara Lobato dos; Costa, Marly Guimarães Fernandes
Abstract
Introduction: Sign language is a collection of gestures, postures, movements, and facial expressions used by deaf people. The Brazilian sign language is Libras. The use of Libras has been increased among the deaf communities, but is still not disseminated outside this community. Sign language recognition is a field of research, which intends to help the deaf community communication with non-hearing-impaired people. In this context, this paper describes a new method for recognizing hand configurations of Libras - using depth maps obtained with a Kinect® sensor. Methods: The proposed method comprises three phases: hand segmentation, feature extraction, and classification. The segmentation phase is independent from the background and depends only on pixel value. The feature extraction process is independent from rotation and translation. The features are extracted employing two techniques: (2D)2LDA and (2D)2PCA. The classification employs two classifiers: a novelty classifier and a KNN classifier. A robust database is constructed for classifier evaluation, with 12,200 images of Libras and 200 gestures of each hand configuration. Results: The best accuracy obtained was 96.31%. Conclusion: The best gesture recognition accuracy obtained is much higher than the studies previously published. It must be emphasized that this recognition rate is obtained for different conditions of hand rotation and proximity of the depth camera, and with a depth camera resolution of only 640×480 pixels. This performance must be also credited to the feature extraction technique, and to the size standardization and normalization processes used previously to feature extraction step.
Keywords
References
Al-Jarrah O, Halawani A. Recognition of gestures in Arabic sign language using neuro-fuzzy systems. Artificial Intelligence. 2001; 133(1-2):117-38. http://dx.doi.org/10.1016/S0004-3702(01)00141-2.
Anjo MS. Avaliação das técnicas de segmentação, modelagem e classificação para o reconhecimento automático de gestos e proposta de uma solução para classificar gestos da libras em tempo real [dissertation]. São Carlos: UFSCar; 2013.
Bragatto TA, Ruas GI, Lamar MV. Real-time video based finger spelling recognition system using low computational complexity Artificial Neural Networks. In: Proceedings of the IEEE International Telecommunications Symposium; 2006 Sept 3-6; Fortaleza, BR. New York: IEEE; 2006. p. 393-7.
Brito LF. Por uma gramática de Língua de Sinais. 1st ed. Rio de Janeiro: Tempo Brasileiro; 2010.
Carneiro TS, Cortez PC, Costa RCS. Recognizing Libras gestures with neural classifiers, using Hu invariant moments. In: Proceedings of Interaction 09 - South America; 2009 Nov 26-28; São Paulo, BR. São Paulo: IXDA; 2009. p. 190-5.
Costa CFF Fo, Falcão AT, Costa MGF, Gomes JR. Proposing the novelty classifier for face recognition. Revista Brasileira de Engenharia Biomédica. 2014; 30(4):301-11. http://dx.doi.org/10.1590/1517-3151.0543.
Costa CFF Fo, Pinheiro CFM, Costa MGF, Pereira WCA. Applying a novelty filter as a matching criterion to iris recognition for binary and real-valued feature vectors. Signal Image Video Process. 2013; 7(2):287-96. http://dx.doi.org/10.1007/s11760-011-0237-5.
Dong C, Leu MC, Yin Z. American sign language alphabet recognition using microsoft kinect. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops; 2015 June 7-12; Boston, MA. New York: IEEE; 2015. p. 4452.
Gonzalez RC, Woods RE. Digital image processing, 3rd ed. New Jersey: Pearson Prentice Hall; 2008.
Kohonen T. Self-organization and associative memory. 3rd ed. New York: Springer-Verlag; 1989. http://dx.doi.org/10.1007/978-3-642-88163-3.
Lee GC, Yeh FH, Hsiao YH. Kinect based Taiwanese sign language recognition system. Multimedia Tools and Applications. 2016; 75(1):261-79. http://dx.doi.org/10.1007/s11042-014-2290-x.
Maraqa M, Al-Zboun F, Dhyabat M, Zitar RA. Recognition of Arabic sign language (ArSL) using recurrent neural networks. J Intell Learn Sys Appl. 2012; 4(1):41-52. http://dx.doi.org/10.4236/jilsa.2012.41004.
Mehdi SA, Khan YN. Sign language recognition using sensor gloves. In Wang L, Rajapakse JC, Fukushima K, Lee SY, Yao X, editors. ICONIP'02: Proceedings of the 9th International Conference on Neural Information Processing: Computational Intelligence for the e-age. 2002 Nov 18-22; Singapore, SG. New York: IEEE; 2002. p. 2204-6.
Neris MN, Silva AJ, Peres SM, Flores FC. Self organizing maps and bit signature: a study applied on signal language recognition. 2008 IEEE International Joint Conference on Neural Networks. 2008 June 1-8; Hong Kong. New York: IEEE; 2008. p. 2934-41.
Noushath S, Kumar GH, Shivakumara P. (2D)(2)LDA: an efficient approach for face recognition. Pattern Recognition. 2006; 39(5):1396-400. http://dx.doi.org/10.1016/j.patcog.2006.01.018.
Penrose R, Todd JA. A generalized inverse for matrices. Proceedings of Mathematical Proceedings of the Cambridge Philosophical Society. 1955; 51(3):406-13. http://dx.doi.org/10.1017/S0305004100030401.
Peres SM, Flores FC, Veronez D, Olguin CJ. Libras signals recognition: a study with learning vector quantization and bit signature. In: Canuta AMP, Souto MCP, Silva ACR, editors. SBRN'06 Ninth Brazilian Symposium on Neural Networks. 2006 Oct 23-27; Ribeirão Preto, SP. New York: IEEE; 2006. p. 119-24.
Pimenta N, Quadros RM. Curso de Libras 1. 4th ed. Rio de Janeiro: Vozes; 2010.
Pizzolato EB, Anjo MS, Pedroso GC. Automatic recognition of finger spelling for libras based on a two layer architecture. In: Proceedings of the 25th Symposium on Applied Computing - SAC'10; 2010 Mar 22-26; Sierre, CH. New York: ACM; 2010. p. 970-4.
Porfirio AJ, Wiggers KL, Oliveira LES, Weingaertner D. Libras sign language hand configuration recognition based on 3D meshes. In: Proceedings of SMC2013 IEEE International Conference on Systems, Man, and Cybernetics. 2013 Oct 13-16; Manchester, UK. Los Alamitos: CPS; 2013. p. 1588-93.
Rakun EM, Andriani I, Wiprayoga W, Danniswara K, Tjandra A. Combining depth image and skeleton data from Kinect for recognizing words in the sign system for Indonesian language (SIBI [Sistem Isyarat Bahasa Indonesia]). In: Proceedings of International Conference on Advanced Computer Science and Information Systems (ICACSIS); 2013 Sept 28-29; Bali, ID. Indonesia: Faculty of Computer Science Universitas Indonesia; 2013. p. 387-92.
Santos JRD. Reconhecimento das configurações de mãos Libras baseado na análise discriminante de Fisher bidimensional utilizando imagens de profundidade [dissertation]. Manaus: Universidade Federal do Amazonas; 2015.
Silva JP, Lamar MV, Bordim JL. A study of the ICP algorithm for recognition of the hand alphabet. In: Aguilar J, Cerqueira E, editors. In: Proceedings of XXXIX Latin American Computing Conference (CLEI 2013); 2013 Oct 7-11; Naiguata, VE. New York: IEEE; 2013 p. 1-9.
Silva RS. Reconhecimento das configurações de mão da língua brasileira de sinais, Libras, em imagens de profundidade através da análise de componentes principais e do classificador k-vizinhos mais próximos [dissertation]. Manaus: Universidade Federal do Amazonas; 2015.
Wang H, Leu MC, Oz C. American Sign Language recognition using multi-dimensional hidden Markov models. Journal of Information Science and Engineering. 2006; 22(5):1109-23.
Wang H. Nearest neighbors by neighborhood counting. IEEE T Pattern Anal. 2006; 28(6):942-53. PMid:16724588. http://dx.doi.org/10.1109/TPAMI.2006.126.
Yang J, Zhang D, Yong X, Yang JY. Two-dimensional discriminant transform for face recognition. Pattern Recognition. 2005; 38(7):1125-9. http://dx.doi.org/10.1016/j.patcog.2004.11.019.
Zhang D, Zhou Z 2nd. PCA:Two-directional two-dimensional PCA for efficient face representation and recognition. Neurocomputing. 2005; 69(1-3):224-31. http://dx.doi.org/10.1016/j.neucom.2005.06.004.
Facebook
Google+
Twitter
LinkedIn
Mendeley
StumbleUpon
CiteULike
Reddit
Email