Research on Biomedical Engineering
http://www.rbejournal.periodikos.com.br/article/doi/10.1590/2446-4740.01515
Research on Biomedical Engineering
Original Article

Mobile Augmented Reality enhances indoor navigation for wheelchair users

Oliveira, Luciene Chagas de; Soares, Alcimar Barbosa; Cardoso, Alexandre; Andrade, Adriano de Oliveira; Lamounier Júnior, Edgard Afonso

Downloads: 1
Views: 1022

Abstract

Abstract Introduction: Individuals with mobility impairments associated with lower limb disabilities often face enormous challenges to participate in routine activities and to move around various environments. For many, the use of wheelchairs is paramount to provide mobility and social inclusion. Nevertheless, they still face a number of challenges to properly function in our society. Among the many difficulties, one in particular stands out: navigating in complex internal environments (indoors). The main objective of this work is to propose an architecture based on Mobile Augmented Reality to support the development of indoor navigation systems dedicated to wheelchair users, that is also capable of recording CAD drawings of the buildings and dealing with accessibility issues for that population. Methods: Overall, five main functional requirements are proposed: the ability to allow for indoor navigation by means of Mobile Augmented Reality techniques; the capacity to register and configure building CAD drawings and the position of fiducial markers, points of interest and obstacles to be avoided by the wheelchair user; the capacity to find the best route for wheelchair indoor navigation, taking stairs and other obstacles into account; allow for the visualization of virtual directional arrows in the smartphone displays; and incorporate touch or voice commands to interact with the application. The architecture is proposed as a combination of four layers: User interface; Control; Service; and Infrastructure. A proof-of-concept application was developed and tests were performed with disable volunteers operating manual and electric wheelchairs. Results: The application was implemented in Java for the Android operational system. A local database was used to store the test building CAD drawings and the position of fiducial markers and points of interest. The Android Augmented Reality library was used to implement Augmented Reality and the Blender open source library handled the basis for implementing directional navigation arrows. OpenGL ES provided support for various graphics and mathematical transformations for embedded systems, such as smartphones. Experiments were performed in an academic building with various labs, classrooms and male and female bathrooms. Two disable volunteers using wheelchairs showed no difficulties to interact with the application, either by entering touch or voice commands, and to navigate within the testing environment with the help of the navigational arrows implemented by the augmented reality modules. Conclusion: The novel features implemented in the proposed architecture, with special emphasis on the use of Mobile Augmented Reality and the ability to identify the best routes free of potential hazards for wheelchair users, were capable of providing significant benefits for wheelchair indoor navigation when compared to current techniques described in the literature.

Keywords

Mobile augmented reality, Assistive technology, Pervasive computing, Indoor wheelchair navigation.

References

Aditya SK, Vikash KK. Android sQLite essentials. Birmingham: Packt Publishing Ltd; 2014.

Ahluwalia P, Varshney U, Koong KS, Wei J. Ubiquitous, mobile, pervasive and wireless information systems: current research and future directions. International Journal of Mobile Communications. 2014; 12(2):103-41. http://dx.doi.org/10.1504/IJMC.2014.059738.

Alm N, Arnott JL, Murray IR, Buchanan I. Virtual reality for putting people with disabilities in control. IEEE International Conference on Systems, Man, and Cybernetics; 1998; San Diego, USA. San Diego: IEEE; 1998. p. 1174-9.

Barberis C, Bottino A, Malnati G, Montuschi P. Experiencing indoor navigation on mobile Devices. IT Professional. 2014; 16(1):50-7. http://dx.doi.org/10.1109/MITP.2013.54.

Blum JR, Bouchard M, Cooperstock JR. What's around me? Spatialized audio augmented reality for blind users with a smartphone. New York: Springer; 2011. http://dx.doi.org/10.1007/978-3-642-30973-1_5.

Brito A. Blender 3D: architecture, buildings, and scenery: create photorealistic 3D architectural visualizations of buildings, interiors, and environmental scenery. Birmingham: Packt Publishing Ltd; 2008.

Burnette E. Hello, android: introducing Google’s mobile development platform. 2nd ed. Raleigh: Pragmatic Bookshelf; 2009.

Cankaya IA, Koyun A, Yigit T, Yuksel AS. Mobile indoor navigation system in iOS platform using augmented reality. AICT: Proceedings of the 9th International Conference on Application of Information and Communication Technologies; 2015 Oct 14-16; Rostov-on-Don, Russia. Rostov-on-Don: IEEE; 2015. p. 281-4.

Chakraborty B, Hashimoto T. A framework for user aware route selection in pedestrian navigation system. ISAC: Proceedings of the 2nd International Symposium on Aware Computing; 2010 Nov 1-4; Tainan, Taiwan. Tainan: IEEE; 2010. p. 150-3. http://dx.doi.org/10.1109/isac.2010.5670466.

Chawathe SS. Low-latency indoor localization using bluetooth beacons. ITSC '09: Proceedings of the 12th International IEEE Conference on Intelligent Transportation Systems; 2009 Oct 4-7; St. Louis, USA. St. Louis: IEEE; 2009. p. 1-7.

Cheein FAA, De La Cruz C, Filho TFB, Carelli R. Maneuverability strategy for assistive for assistive vehicles navigating within confined space. International Journal of Advanced Robotic Systems. 2011; 8:62-75.

Chen P, Peng Z, Li D, Yang L. An improved augmented reality system based on AndAR. Journal of Visual Communication and Image Representation. 2016; 37(1):63-9. http://dx.doi.org/10.1016/j.jvcir.2015.06.016.

Chiara DD, Paolino L, Romano M, Sebillo M, Tortora G, Vitiello G. Augmented map navigation through customizable mobile interfaces. Proceedings of the 16th International Conference on Distributed Multimedia Systems; 2010; Oak Brook, USA. Oak Brook: Knowledge Systems Institute Graduate School; 2010. p. 265-70.

De La Cruz C, Bastos TF, Cheein FAA, Carelli R. SLAM-based robotic wheelchair navigation system designed for confined spaces. ISIE: Proceedings of the 2010 IEEE International Symposium on Industrial Electronics; 2010 Jul 4-7; Bari, Italy. Bari: IEEE; 2010. p. 2331-6. http://dx.doi.org/10.1109/isie.2010.5637760.

De La Cruz C, Celeste WC, Bastos TF. A robust navigation system for robotic wheelchairs. Control Engineering Practice. 2011; 19(6):575-90. http://dx.doi.org/10.1016/j.conengprac.2010.11.007.

Deepesh PC, Rath R, Tiwary A, Rao VN, Kanakalata N. Experiences with using iBeacons for indoor positioning. Proceedings of the 9th India Software Engineering Conference; 2016 Feb 18-20; Goa, India. Rajasthan: Birla Institute of Technology and Science; 2016. p. 184-9.

Delail BA, Weruaga L, Zemerly MJ. CAViAR: context aware visual indoor augmented reality for a university Campus. WI-IAT: Proceedings of the 2012 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology; 2012 Dec 4-7; Macau, China. Macau: IEEE; 2012. p. 286-90. http://dx.doi.org/10.1109/wi-iat.2012.99.

Denning PJ, Metcalfe RM. Beyond calculation: the next fifty years of computing. Göttingen: Copernicus; 1998.

Deruwe G, Wall R. Pedestrian navigation and integration with distributed smart signal traffic controls. IECON 2008: Proceedings of the 34th Annual Conference of IEEE - Industrial Electronics; 2008 Nov 10-13; Orlando, USA. Orlando: IEEE; 2008. p. 2923-8. http://dx.doi.org/10.1109/iecon.2008.4758424.

Dong J, Xiao Y, Noreikis M, Ou Z, Yl A. iMoon: using smartphones for image-based indoor navigation. Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems. 2015 Nov 1-4; Seoul, South Korea. Seoul: ACM; 2015. p. 85-97.

Ferreira ALS, Santos SRD, Miranda LCD. TrueSight: sistema de navegação para pedestres baseado na detecção e Extração automática de landmark em smartphone Android. SVR 2012: Anais do XIV Simpósio de Realidade Virtual e Aumentada; 2012 May 21-31; Niterói, Brasil. Niterói: Universidade Federal Fluminense; 2012. p. 91-9. http://dx.doi.org/10.1109/SVR.2012.14.

Golfarelli M, Maio D, Rizzi S. Correction of dead-reckoning errors in map building for mobile robots. IEEE Transactions on Robotics and Automation. 2001: 17(1):37-47.

Hansen R, Thomsen B. Using weighted graphs for computationally efficient WLAN location determination. MobiQuitous 2007: Proceedings of the Fourth Annual International Conference on Mobile and Ubiquitous Systems: Networking & Services; 2007 Aug 6-10; Philadephia, PA, USA. Philadelphia: IEEE; 2007. p. 1-5.

Hervas R, Bravo J, Fontecha J. An assistive navigation system based on augmented reality and context awareness for people with mild cognitive impairments. IEEE Journal Biomedical and Health Informatics. 2014; 18(1):368-74. http://dx.doi.org/10.1109/JBHI.2013.2266480. PMid:24403436.

Höllerer T, Feiner S, Terauchi T, Rashid G, Hallaway D. Exploring MARS: developing indoor and outdoor user interfaces to a mobile augmented reality system. Computers & Graphics. 1999; 23(6):779-85. http://dx.doi.org/10.1016/S0097-8493(99)00103-X.

Hub A. Precise indoor and outdoor navigation for the blind and visually impaired using augmented maps and the TANIA system. Vision 2008: Proceedings of the 9th International Conference on Low Vision; 2008 Jul 7-11; Montreal, Canada. Canada: Online; 2008. p. 1-4.

Jia P, Hu HH, Lu T, Yuan K. Head gesture recognition for hands-free control of an intelligent wheelchair. Industrial Robot: International Journal. 2007; 34(1):60-8.

Ju JS, Shin Y, Kim EY. Vision based interface system for hands free control of an Intelligent Wheelchair. Journal of Neuroengineering and Rehabilitation. 2009; 6(1):1-17. http://dx.doi.org/10.1186/1743-0003-6-33. PMid:19660132.

Kalkusch M, Lidy T, Knapp M, Reitmayr G, Kaufmann H, Schmalstieg D. Structured visual markers for indoor pathfinding. Proceedings of the First IEEE International Workshop on Augmented Reality Toolkit; 2002; Darmstadt, Germany. Germany: IEEE; 2002. p. 8. http://dx.doi.org/10.1109/art.2002.1107018.

Koch C, Neges M, König M, Abramovici M. Natural markers for augmented reality-based indoor navigation and facility maintenance. Automation in Construction. 2014; 48:18-30. http://dx.doi.org/10.1016/j.autcon.2014.08.009.

Kotsakos D, Sakkos P, Kalogeraki V, Gunopulos D. Using smart mobile devices for monitoring in assistive environments. Proceedings of the 6th International Conference on Pervasive Technologies Related to Assistive Environments; 2013 May 29-31; Rhodes, Greece. USA: ACM; 2013. p. 1-4.

Kouroupetroglou G. Disability informatics and web accessibility for motor limitations. In: Kouroupetroglou G, editor. Advances in medical technologies and clinical. Hershey: IGI Global; 2013.

Lokuge Y, Madumal P, Kumara T, Ranasinghe N. Indoor navigation framework for mapping and localization of multiple robotic wheelchairs. ISMS 2014: Proceedings of the 5th International Conference on Intelligent Systems, Modelling and Simulation; 2014 Jan 27-29; Lagkawi, Malaysia. Lagkawi: IEEE; 2014. p. 197-200. http://dx.doi.org/10.1109/isms.2014.39.

Maghdid HS, Lami IA, Ghafoor KZ, Lloret J. Seamless outdoors-indoors localization solutions on smartphones: implementation and challenges. ACM Computing Surveys. 2016; 48(4):1-34. http://dx.doi.org/10.1145/2871166.

Marston JR, Loomis JM, Klatzky RL, Golledge RG, Smith EL. Evaluation of spatial displays for navigation without sight. ACM Transactions on Applied Perception. 2006; 3(2):110-24. http://dx.doi.org/10.1145/1141897.1141900.

Matuszka T, Gombos G, Kiss A. A new approach for indoor navigation using semantic webtechnologies and augmented reality. In: Shumaker R, editor. Virtual augmented and mixed reality. Lecture notes in computer science: designing and developing augmented and virtual environments. Heidelberg: Springer; 2013. v. 8021, p. 202-10. http://dx.doi.org/10.1007/978-3-642-39405-8_24.

Mirza R, Tehseen A, Kumar AVJ. An indoor navigation approach to aid the physically disabled people. ICCEET 2012: Proceedings of the International Conference on Computing, Electronics and Electrical Technologies; 2012 March 21-22; Nagercoil, India. Kumaracoil: IEEE; 2012. p. 979-83. http://dx.doi.org/10.1109/icceet.2012.6203860.

Mithwick R, Michael MV, Muhtasib L. Pro opengl es for android. New York: Apress; 2012.

Mulloni A, Grubert J, Seichter H, Langlotz T, Grasset R, Reitmayr G, Schmalstieg D. Experiences with the impact of tracking technology in mobile augmented reality. MobileHCI and MobiVis 2012: Proceedings of Mobile Vision (MobiVis 2012): vision-based applications and HCI; Workshop at MobileHCI 2012. 2012 Sept 21; San Fransisco, USA. San Fransisco: ACM; 2012. p. 1-4

Neges M, Koch C, König M, Abramovici M. Combining visual natural markers and IMU for improved AR based indoor navigation. Advanced Engineering Informatics. 2015. In press. http://dx.doi.org/10.1016/j.aei.2015.10.005.

Newman J, Wagner M, Bauer M, Macwilliams A, Pintaric T, Beyer D, Pustka D, Strasser F, Schmalstieg D, Klinker G. Ubiquitous tracking for augmented reality. ISMAR 2004: Proceedings of the 3rd IEEE/ACM International Symposium on Mixed and Augmented Reality; 2004 Nov 2-5; Arlington, USA. Arlington: IEEE; 2004. p. 192-201.

Onorati T, Malizia A, Diaz P, Aedo I. Modeling an ontology on accessible evacuation routes for emergencies. Expert Systems with Applications. 2014; 41(16):7124-34. http://dx.doi.org/10.1016/j.eswa.2014.05.039.

Park K-H, Bien Z, Lee J-J, Kim BK, Lim J-T, Kim J-O, Lee H, Stefanov DH, Kim D-J, Jung J-W, Do J-H, Seo K-H, Kim CH, Song W-G, Lee W-J. Robotic smart house to assist people with movement disabilities. Autonomous Robots. 2006; 22(2):183-98. http://dx.doi.org/10.1007/s10514-006-9012-9.

Parker C, Tomitsch M. Data visualisation trends in mobile augmented reality applications. Proceedings of the 7th International Symposium on Visual Information Communication and Interaction; 2014 Aug 5-8; Sydney, Australia. USA: ACM; 2014. p. 228-31.

Poslad S. Ubiquitous computing: smart devices, environments and interactions. Hoboken: Wiley; 2009. http://dx.doi.org/10.1002/9780470779446.

Postolache O, Silva PG, Pinheiro E, Pereira MD, Madeira R, Mendes J, Cunha M, Postolache G, Moura CM. Pervasive sensing and computing for wheelchairs users health assessment. Proceedings of the 1st Portuguese Meeting in Bioengineering; 2011 Mar 1-4; Lisbon, Portugal. Lisbon: IEEE; 2011. p. 1-4. http://dx.doi.org/10.1109/enbeng.2011.6026070.

Ran L, Helal S, Moore S. Drishti: an integrated indoor/outdoor blind navigation system and service. PerCom 2004: Proceedings of the Second IEEE International Conference on Pervasive Computing and Communications; 2004 Mar 14-17; Orlando, USA. Orlando: IEEE; 2004. p. 23.

Rehman U, Cao S. Augmented reality-based indoor navigation using Google Glass as a wearable head-mounted display. SMC 2015: Proceedings of the 2015 IEEE International Conference on Systems, Man, and Cybernetics; 2015 Oct 9-12; Hong Kong, China. Hong Kong: IEEE; 2015. p. 1452-7. http://dx.doi.org/10.1109/smc.2015.257.

Rovadosky DS, Pavan W, Dalbosco J, Cervi CR. Uma ferramenta de realidade aumentada usando dispositivo móvel com sistema operacional Android. Revista Brasileira de Computação Aplicada. 2012; 4(1):25-37.

Rui Z, Yuanqing L, Yongyong Y, Hao Z, Shaoyu W, Tianyou Y, Zhenghui G. Control of a Wheelchair in an Indoor Environment Based on a Brain. 2013; Computer Interface and Automated Navigation. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2016; 24(1):128-39. http://dx.doi.org/10.1109/TNSRE.2015.2439298. PMid:26054072.

Ruta M, Scioscia F, Ieva S, Filippis DD, Sciascio ED. Indoor/outdoor mobile navigation via knowledge-based POI discovery in augmented reality. WI-IAT 2015: Proceedings of the 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology; 2015 Dec 6-9; Singapore. Singapore: IEEE; 2015. p. 26-30. http://dx.doi.org/10.1109/wi-iat.2015.243.

Sanchez JH, Aguayo FA, Hassler TM. Independent outdoor mobility for the blind. Proceedings of the Virtual Rehabilitation 2007; 2007 Sep 27-29; Venice, Italy. Venice: IEEE; 2007. p. 114-20. http://dx.doi.org/10.1109/icvr.2007.4362150.

Santarosa LMC, Conforto D, Basso LDO. Eduquito: ferramentas de autoria e de colaboração acessíveis na perspectiva da web 2.0. Revista Brasileira de Educação Especial. 2012; 18(3): 449-68.

Sharhan SMH, Zickau S. Indoor mapping for location-based policy tooling using Bluetooth Low Energy beacons. WiMob 2015: Proceedings of the IEEE 11th International Conference on Wireless and Mobile Computing, Networking and Communications; 2015 Oct 19-21; Abuu-Dhabi, United Arab Emirates. Abuu-Dhabi: IEEE; 2015. p. 28-36. http://dx.doi.org/10.1109/WiMOB.2015.7347937.

Sinyukov DA, Ran L, Otero NW, Runzi G, Padir T. Augmenting a voice and facial expression control of a robotic wheelchair with assistive navigation. SMC 2014: Proceedings of the 2014 IEEE International Conference on Systems, Man and Cybernetics; 2014 Oct. 5-8; San Diego, USA. San Diego: IEEE; 2014. p. 1088-94. http://dx.doi.org/10.1109/smc.2014.6974059.

Stefanov DH, Bien Z, Bang W-C. The smart house for older persons and persons with physical disabilities: structure, technology arrangements, and perspectives. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2004; 12(2):228-50. http://dx.doi.org/10.1109/TNSRE.2004.828423. PMid:15218937.

Takahashi C, Kondo K. Indoor positioning method for augmented audio reality navigation systems using iBeacons. GCCE 2015: Proceedings of the IEEE 4th Global Conference on Consumer Electronics; 2015 Oct 27-30. Osaka, Japan. Osaka: IEEE; 2015. p. 451-2. http://dx.doi.org/10.1109/gcce.2015.7398636.

Tsetsos V, Anagnostopoulos C, Kikiras P, Hadjiefthymiades S. Semantically enriched navigation for indoor environments. International Journal of Web and Grid Services. 2006; 2(4):453-78. http://dx.doi.org/10.1504/IJWGS.2006.011714.

Tsetsos V, Anagnostopoulos C, Kikiras P, Hasiotis T, Hadjiefthymiades S. A human-centered semantic navigation system for indoor environments. ICPS '05: Proceedings of the 2005 International Conference on Pervasive Services; 2005 Jul 11-14; Santorini, Greece. Greece: IEEE; 2005. p. 146-55.

Xing L, Alpcan T, Bauckhage C. Adaptive wireless services for augmented environments. MobiQuitous '09: Proceedings of the 6th Annual InternationalMobile and Ubiquitous Systems: Networking & Services; 2009 Jul 13-1; Toronto, Canada. Toronto: IEEE; 2009. p. 1-8. http://dx.doi.org/10.4108/icst.mobiquitous2009.6821.

Yayan U, Akar B, Inan F, Yazici A. Development of indoor navigation software for intelligent wheelchair. INISTA 2014: Proceedings of the 2014 IEEE International Symposium on Innovations in Intelligent Systems and Applications; 2014 Jun 23-25; Alberobelo, Italy. Alberobelo: IEEE; 2014. p. 325-9. http://dx.doi.org/10.1109/inista.2014.6873639.

Yohan C, Talipov E, Hojung C. Autonomous management of everyday places for a personalized location provider. IEEE Transactions on Systems, Man and Cybernetics. Part C, Applications and Reviews. 2012; 42(4):518-31. http://dx.doi.org/10.1109/TSMCC.2011.2131129.
5889fbfc5d01231a018b48a4 rbejournal Articles
Links & Downloads

Res. Biomed. Eng.

Share this page
Page Sections