Comparative Analysis: Traditional Models VS Transformers in Hate Speech Detection
DOI:
https://doi.org/10.63503/j.ijaimd.2025.219Keywords:
hate speech, Natural Language Processing, Long Short Term Memory Neural Networks, RoBERTa-Base, RoBERTa-Large, TransformersAbstract
Detecting hate speech is a significant task in understanding the contents published online, particularly in natural language processing. Strong automated systems that can identify and minimize harmful online communication are therefore becoming necessary. The aim of this study is to compare classical deep learning models against transformer-based architectures to find a technique for identifying hate speech written in English. Using a hate speech classification labeled dataset, the study determine the performance of each model across evaluation metrics like accuracy, precision, recall and F1-score. For addressing class imbalance and improve generalization, methods like focal loss and data augmentation were applied. Our findings indicate that the models based on transformers, particularly RoBERTa-large, significantly outperform traditional architectures in finding out subtle and context-dependent instances of hate speech. This research highlights emerging importance of large pre-trained language models and hybrid ways for enhancing automated hate speech detection systems.
References
S. G. Roy, U. Narayan, T. Raha, Z. Abid, and V. Varma, “Leveraging multilingual transformers for hate speech detection,” arXiv preprint, arXiv:2101.03207, 2021.
[2] J. S. Malik, H. Qiao, G. Pang, and A. van den Hengel, “Deep learning for hate speech detection: A comparative study,” International Journal of Data Science and Analytics, pp. 1–16, 2024.
[3] D. Liu, M. Wang, and A. G. Catlin, “Detecting anti-semitic hate speech using transformer-based large language models,” arXiv preprint, arXiv:2405.03794, 2024.
[4] V. Dwivedy and P. K. Roy, “Deep feature fusion for hate speech detection: A transfer learning approach,” Multimedia Tools and Applications, vol. 82, no. 23, pp. 36279–36301, 2023.
[5] G. Ramos, F. Batista, R. Ribeiro, P. Fialho, S. Moro, A. Fonseca, and C. Silva, “A comprehensive review on automatic hate speech detection in the age of the transformer,” Social Network Analysis and Mining, vol. 14, no. 1, Art. no. 204, 2024.
[6] Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, vol. 30, 2017.
[7] D. Putra and H.-C. Wang, “Advanced BERT-CNN for hate speech detection,” Procedia Computer Science, vol. 234, pp. 239–246, 2024.
[8] P. K. Roy, A. K. Tripathy, T. K. Das, and X. Z. Gao, “A framework for hate speech detection using deep convolutional neural network,” IEEE Access, vol. 8, pp. 204951–204962, 2020.
[9] M. Madhavi, S. Agal, N. D. Odedra, H. Chowdhary, T. S. Ruprah, V. A. Vuyyuru, and Y. A. B. El-Ebiary, “Elevating offensive language detection: CNN-GRU and BERT for enhanced hate speech identification,” International Journal of Advanced Computer Science and Applications, vol. 15, no. 5, 2024.
[10] Mandal, G. Roy, A. Barman, I. Dutta, and S. K. Naskar, “Attentive fusion: A transformer-based approach to multimodal hate speech detection,” arXiv preprint, arXiv:2401.10653, 2024.
[11] M. Subramanian, et al., “A survey on hate speech detection and sentiment analysis using machine learning and deep learning models,” Alexandria Engineering Journal, vol. 80, pp. 110–121, 2023.
[12] H. Saleh, A. Alhothali, and K. Moria, “Detection of hate speech using BERT and hate speech word embedding with deep model,” Applied Artificial Intelligence, vol. 37, no. 1, Art. no. 2166719, 2023.
[13] L. Yuan, et al., “Transfer learning for hate speech detection in social media,” Journal of
Computational Social Science, vol. 6, no. 2, pp. 1081–1101, 2023.
[14] O. E. Ojo, et al., “Automatic hate speech detection using deep neural networks and word embedding,” Computación y Sistemas, vol. 26, no. 2, pp. 1007–1013, 2022.
[15] S. Abro, S. Shaikh, Z. H. Khand, A. Zafar, S. Khan, and G. Mujtaba, “Automatic hate speech detection using machine learning: A comparative study,” International Journal of Advanced Computer Science and Applications, vol. 11, no. 8, 2020.
[16] R. Cao, R. K. W. Lee, and T. A. Hoang, “DeepHate: Hate speech detection via multi-faceted text representations,” in Proceedings of the 12th ACM Conference on Web Science, Jul. 2020.
[17] S. Loria, “TextBlob documentation,” Release 0.15, p. 269, 2018.
[18] F. M. Miranda, N. Köhnecke, and B. Y. Renard, “HiClass: A Python library for local hierarchical classification compatible with scikit-learn,” Journal of Machine Learning Research, vol. 24, no. 29, pp. 1–17, 2023.[19] S. Anjali Devi and S. Sivakumar, “An efficient contextual GloVe feature extraction model on large textual databases,” International Journal of Speech Technology, vol. 25, no. 4, pp. 793–802, 2022.
[20] Y. Kim, “Convolutional neural networks for sentence classification,” arXiv preprint, arXiv:1408.5882, 2014.
[21] Y. Yu, et al., “A review of recurrent neural networks: LSTM cells and network architectures,” Neural Computation, vol. 31, no. 7, pp. 1235–1270, 2019.
[22] J. Chung, et al., “Empirical evaluation of gated recurrent neural networks on sequence modeling,” arXiv preprint, arXiv:1412.3555, 2014.
[23] Y. Liu, et al., “RoBERTa: A robustly optimized BERT pretraining approach,” arXiv preprint, arXiv:1907.11692, 2019.
[24] H. U. Khan, et al., “Analyzing student mental health with RoBERTa-Large: A sentiment analysis and data analytics approach,” Frontiers in Big Data, vol. 8, Art. no. 1615788, 2025.
[25] P. Timsina, Building Transformer Models with PyTorch 2.0: NLP, Computer Vision, and Speech Processing with PyTorch and Hugging Face. New Delhi, India: BPB Publications, 2024.
[26] Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint, arXiv:1711.05101, 2017
