Advances in Indian Sign Language Processing: A State-of-the-Art Review and a Real-Time Bidirectional Translation Architecture
DOI:
https://doi.org/10.63503/j.ijcma.2026.255Keywords:
Indian Sign Language (ISL), Deep Learning, Dataset Benchmarking, MediaPipe, Convolutional Neural Network (CNN), Transformer, ISL-CSLTR, INCLUDE Dataset, SignFlow, HWGATAbstract
This paper presents a comprehensive review of existing datasets, methodologies, and experimental frameworks for Indian Sign Language (ISL) recognition and translation, highlighting recent advancements driven by deep learning architectures such as Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) networks, and Transformers. Despite this progress, the review finds that current systems remain limited by dataset diversity, lack of standardization, and insufficient support for real-time, user-centric applications. Building on these findings, this study proposes an integrated ISL framework, “Beyond,” that combines dataset curation with real-time, bidirectional, and multilingual (Hindi-English) translation, supported by a finger-spelling fallback for out-of-vocabulary words. The framework further incorporates word-level video alignment, background-normalized gesture recognition, predictive text support, and accessibility-oriented controls, along with a proposed multimodal dataset designed for linguistic alignment and temporal coherence. Together, these contributions aim to bridge ISL research and real-world deployment, supporting scalable, real-time, and inclusive communication for the Deaf and Hard-of-Hearing community.
References
[1] V. Adithya, P. R. Vinod, and U. Gopalakrishnan, "Artificial neural network based method for Indian sign language recognition," in Proc. IEEE Conf. Information and Communication Technologies (ICT), Apr. 2013, pp. 1080–1085.
[2] V. Adithya and R. Rajesh, "A deep convolutional neural network approach for static hand gesture recognition," Procedia Computer Science, vol. 171, pp. 2353–2361, 2020.
[3] S. C. Agrawal, A. S. Jalal, and C. Bhatnagar, "Recognition of Indian Sign Language using feature fusion," in Proc. 4th Int. Conf. Intelligent Human Computer Interaction (IHCI), Dec. 2012, pp. 1–5.
[4] W. Ahmed, K. Chanda, and S. Mitra, "Vision based hand gesture recognition using dynamic time warping for Indian sign language," in Proc. Int. Conf. Information Science (ICIS), 2016, pp. 120–125.
[5] K. Amrutha and P. Prabu, "A comparative study on Indian sign language representation," in Proc. Int. Conf. Advancements in Electrical, Electronics, Communication, Computing and Automation (ICAECA), Oct. 2021, pp. 1–6.
[6] G. Anantha Rao, P. V. V. Kishore, A. S. C. S. Sastry, D. Anil Kumar, and E. Kiran Kumar, "Selfie continuous sign language recognition with neural network classifier," in Proc. 2nd Int. Conf. Micro-Electronics, Electromagnetics and Telecommunications (ICMEET 2016), Singapore: Springer, 2017, pp. 31–40.
[7] G. Arun Prasath and K. Annapurani, "A review on deaf and dumb communication system based on various recognitions aspect," in Proc. Int. Conf. Deep Learning, Computing and Intelligence (ICDCI 2021), Singapore: Springer Nature, 2022, pp. 191–203.
[8] G. Arun Prasath and K. Annapurani, "Prediction of sign language recognition based on multi layered CNN," Multimedia Tools and Applications, vol. 82, no. 19, pp. 29649–29669, 2023.
[9] M. K. Bhuyan, "FSM-based recognition of dynamic hand gestures via gesture summarization using key video object planes," International Journal of Computer and Communication Engineering, vol. 6, pp. 248–259, 2012.
[10] N. C. Camgoz, O. Koller, S. Hadfield, and R. Bowden, "Sign language transformers: Joint end-to-end sign language recognition and translation," in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2020, pp. 10023–10033.
[11] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, "Realtime multi-person 2D pose estimation using part affinity fields," in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2017, pp. 7291–7299.
[12] A. Dumbre, S. Jangada, S. Gosavi, and J. Gupta, "Classification of Indian sign language characters utilizing convolutional neural networks and transfer learning models with different image processing techniques," in Proc. IEEE World Conf. Applied Intelligence and Computing (AIC), Jun. 2022, pp. 423–430.
[13] E.-S. M. El-Alfy and H. Luqman, "A comprehensive survey and taxonomy of sign language research," Engineering Applications of Artificial Intelligence, vol. 114, p. 105198, 2022.
[14] R. Elakkiya and K. Selvamani, "Enhanced dynamic programming approach for subunit modelling to handle segmentation and recognition ambiguities in sign language," Journal of Parallel and Distributed Computing, vol. 117, pp. 246–255, 2018.
[15] R. Elakkiya and K. Selvamani, "Subunit sign modeling framework for continuous sign language recognition," Computers and Electrical Engineering, vol. 74, pp. 379–390, 2019.
[16] J. Forster, C. Schmidt, O. Koller, M. Bellgardt, and H. Ney, "Extensions of the sign language recognition and translation corpus RWTH-PHOENIX-Weather," in Proc. Language Resources and Evaluation Conf. (LREC), May 2014, pp. 1911–1916.
[17] A. S. Ghotkar and G. K. Kharate, "Dynamic hand gesture recognition and novel sentence interpretation algorithm for Indian sign language using Microsoft Kinect sensor," Journal of Pattern Recognition Research, vol. 1, pp. 24–38, 2015.
[18] R. Gogoi, P. Kumar, and R. Damdoo, "End to end simple Indian sign language sentence translation using sign transformer network," in Proc. 15th Int. Conf. Computing Communication and Networking Technologies (ICCCNT), Jun. 2024, pp. 1–13.
[19] V. Gomathi, "Indian sign language recognition through hybrid ConvNet-LSTM networks," EMITTER International Journal of Engineering Technology, vol. 9, no. 1, pp. 182–203, 2021.
[20] R. Gupta and S. Rajan, "Comparative analysis of convolution neural network models for continuous Indian sign language classification," Procedia Computer Science, vol. 171, pp. 1542–1550, 2020.
[21] M. Hasan, T. H. Sajib, and M. Dey, "A machine learning based approach for the detection and recognition of Bangla sign language," in Proc. Int. Conf. Medical Engineering, Health Informatics and Technology (MediTec), Dec. 2016, pp. 1–5.
[22] P. Jadav, "Indian sign language recognition system," International Journal of Engineering and Technology.
[23] A. Joshi et al., "iSign: A benchmark for Indian sign language processing," in Findings of the Assoc. Computational Linguistics (ACL 2024), Aug. 2024, pp. 10827–10844.
[24] G. Joshi, A. Gaur, and Sheenu, "Interpretation of Indian sign language using optimal HOG feature vector," in Proc. Int. Conf. Advances in Computing and Data Sciences, Singapore: Springer, Apr. 2018, pp. 65–73.
[25] P. V. V. Kishore, M. V. D. Prasad, D. A. Kumar, and A. S. C. S. Sastry, "Optical flow hand tracking and active contour hand shape features for continuous sign language recognition with artificial neural networks," in Proc. IEEE 6th Int. Conf. Advanced Computing (IACC), Feb. 2016, pp. 346–351.
[26] O. Koller, J. Forster, and H. Ney, "Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers," Computer Vision and Image Understanding, vol. 141, pp. 108–125, 2015.
[27] D. R. Kothadiya, C. M. Bhatt, T. Saba, A. Rehman, and S. A. Bahaj, "SIGNFORMER: deepvision transformer for sign language recognition," IEEE Access, vol. 11, pp. 4730–4739, 2023.
[28] L. Krishnan, "A systematic literature review on the robustness of sign language recognition methods in low-light environments," Authorea Preprints, 2024.
[29] E. K. Kumar, P. V. V. Kishore, A. S. C. S. Sastry, M. T. K. Kumar, and D. A. Kumar, "Training CNNs for 3-D sign language recognition with color texture coded joint angular displacement maps," IEEE Signal Processing Letters, vol. 25, no. 5, pp. 645–649, 2018.
[30] E. K. Kumar, P. V. V. Kishore, M. T. K. Kumar, and D. A. Kumar, "3D sign language recognition with joint distance and angular coded color topographical descriptor on a 2-stream CNN," Neurocomputing, vol. 372, pp. 40–54, 2020.
[31] E. K. Kumar, P. V. V. Kishore, D. A. Kumar, and M. T. K. Kumar, "Early estimation model for 3D-discrete Indian sign language recognition using graph matching," Journal of King Saud University – Computer and Information Sciences, vol. 33, no. 7, pp. 852–864, 2021.
[32] P. Kumar, R. Saini, S. K. Behera, D. P. Dogra, and P. P. Roy, "Real-time recognition of sign language gestures and air-writing using leap motion," in Proc. 15th IAPR Int. Conf. Machine Vision Applications (MVA), Nagoya, Japan, May 2017, pp. 157–160.
[33] A. Mittal, P. Kumar, P. P. Roy, R. Balasubramanian, and B. B. Chaudhuri, "A modified LSTM model for continuous sign language recognition using leap motion," IEEE Sensors Journal, vol. 19, no. 16, pp. 7056–7063, 2019.
[34] M. Nakrani, "A review paper on sign language recognition system for deaf and dumb people using image processing," International Journal of Engineering Research and Applications, 2016.
[35] P. A. Nanivadekar and V. Kulkarni, "Indian sign language recognition: database creation, hand tracking and segmentation," in Proc. Int. Conf. Circuits, Systems, Communication and Information Technology Applications (CSCITA), Apr. 2014, pp. 358–363.
[36] S. C. Ong and S. Ranganath, "Automatic sign language analysis: A survey and the future beyond lexical meaning," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 6, pp. 873–891, Jun. 2005.
[37] M. Oudah, A. Al-Naji, and J. Chahl, "Hand gesture recognition based on computer vision: a review of techniques," Journal of Imaging, vol. 6, no. 8, p. 73, 2020.
[38] C. Oz and M. C. Leu, "Linguistic properties based on American Sign Language isolated word recognition with artificial neural networks using a sensory glove and motion tracker," Neurocomputing, vol. 70, no. 16–18, pp. 2891–2901, 2007.
[39] D. U. Patel and J. M. Joshi, "Review of Indian dynamic sign language recognition system," in Proc. Int. Conf. Communication and Computational Technologies (ICCCT 2022), Singapore: Springer Nature, 2022, pp. 17–31.
[40] S. Patra, A. Maitra, M. Tiwari, K. Kumaran, S. Prabhu, S. Punyeshwarananda, and S. Samanta, "Hierarchical windowed graph attention network and a large scale dataset for isolated Indian sign language recognition," arXiv preprint arXiv:2407.14224, 2024.
[41] R. Rastgoo, K. Kiani, and S. Escalera, "Sign language recognition: A deep survey," Expert Systems with Applications, vol. 164, p. 113794, 2021.
[42] M. A. Rahaman, M. Jasim, M. H. Ali, and M. Hasanuzzaman, "Real-time computer vision-based Bengali sign language recognition," in Proc. 17th Int. Conf. Computer and Information Technology (ICCIT), Dec. 2014, pp. 192–197.
[43] J. Rekha, J. Bhattacharya, and S. Majumder, "Shape, texture and local movement hand gesture features for Indian sign language recognition," in Proc. 3rd Int. Conf. Trendz in Information Sciences and Computing (TISC), Dec. 2011, pp. 30–35.
[44] S. Reshna and M. Jayaraju, "Spotting and recognition of hand gesture for Indian sign language recognition system with skin segmentation and SVM," in Proc. Int. Conf. Wireless Communications, Signal Processing and Networking (WiSPNET), Mar. 2017, pp. 386–390.
[45] E. J. Robert and H. J. Duraisamy, "A review on computational methods based automated sign language recognition system for hearing and speech impaired community," Concurrency and Computation: Practice and Experience, vol. 35, no. 9, p. e7653, 2023.
[46] B. Samal and M. Panda, "Integrative review on vision-based dynamic Indian sign language recognition systems," in Proc. 1st Odisha Int. Conf. Electrical Power Engineering, Communication and Computing Technology (ODICON), 2021, pp. 1–6.
[47] A. Sharma, N. Sharma, Y. Saxena, A. Singh, and D. Sadhya, "Benchmarking deep neural network approaches for Indian Sign Language recognition," Neural Computing and Applications, vol. 33, no. 12, pp. 6685–6696, 2021.
[48] K. Shenoy, T. Dastane, V. Rao, and D. Vyavaharkar, "Real-time Indian sign language (ISL) recognition," in Proc. 9th Int. Conf. Computing, Communication and Networking Technologies (ICCCNT), Jul. 2018, pp. 1–9.
[49] A. Shinde and R. Kagalkar, "Sign language to text and vice versa recognition using computer vision in Marathi," International Journal of Computer Applications, vol. 975, p. 8887, 2015.
[50] A. Shinde and R. Kagalkar, "Advanced Marathi sign language recognition using computer vision," International Journal of Computer Applications, vol. 118, no. 13, 2015.
[51] O. M. Sincan and H. Y. Keles, "AUTSL: A large scale multi-modal Turkish sign language dataset and baseline methods," IEEE Access, vol. 8, pp. 181340–181355, 2020.
[52] A. Sivan and C. Eswaran, "Indian sign language alphabet recognition and speech synthesis using a hybrid deep learning approach," Archives of Acoustics, 2026.
[53] N. V. Tavari, A. V. Deorankar, and P. N. Chatur, "Hand gesture recognition of Indian sign language to aid physically impaired people," International Journal of Engineering Research and Applications, vol. 5, p. 60, 2014.
[54] K. Tripathi, N. Baranwal, and G. C. Nandi, "Continuous dynamic Indian Sign Language gesture recognition with invariant backgrounds," in Proc. Int. Conf. Advances in Computing, Communications and Informatics (ICACCI), Aug. 2015, pp. 2211–2216.
[55] M. Varsha and C. S. Nair, "Indian sign language gesture recognition using deep convolutional neural network," in Proc. 8th Int. Conf. Smart Computing and Communications (ICSCC), Jul. 2021, pp. 193–197.
[56] M. Vázquez-Enríquez, J. L. Alba-Castro, L. Docío-Fernández, and E. Rodríguez-Banga, "Isolated sign language recognition with multi-scale spatial-temporal graph convolutional networks," in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2021, pp. 3462–3471.
[57] A. Wadhawan and P. Kumar, "Deep learning-based sign language recognition system for static signs," Neural Computing and Applications, vol. 32, no. 12, pp. 7957–7968, 2020.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 International Journal on Computational Modelling Applications
This work is licensed under a Creative Commons Attribution 4.0 International License.
