Sentiment Analysis of Generative AI Adoption: A Comparative Study of Naive Bayes and Support Vector Machine Algorithms

Yosep Moleng; Felix Wijaya; Manda Reksi Saputri; Besus Maula Sulthon

doi:10.59890/ijasr.v4i6.250

Authors

Yosep Moleng Universitas Bina Sarana Informatika
Felix Wijaya Bina Sarana Informatika University
Manda Reksi Saputri Bina Sarana Informatika University
Besus Maula Sulthon Bina Sarana Informatika University

DOI:

https://doi.org/10.59890/ijasr.v4i6.250

Keywords:

Analisis Sentimen, AI Generatif, Naive Bayes, Support Vector Machine, TF-IDF, SMOTE.

Abstract

The rapid adoption of Generative Artificial Intelligence (AI) has led to increasing public discussions and diverse opinions across social media platforms. This study aims to analyze public sentiment toward the adoption of Generative AI and compare the performance of the Naive Bayes and Support Vector Machine (SVM) algorithms in multi-class sentiment classification. The dataset consists of 1,125 Twitter/X posts categorized into positive, neutral, and negative sentiments. The research methodology includes text preprocessing through case folding, tokenization, stopword removal, and stemming, followed by TF-IDF feature weighting and data balancing using the Synthetic Minority Over-sampling Technique (SMOTE). Model performance was evaluated using 10-fold cross-validation to ensure reliable and robust results. The findings show that SVM achieved an accuracy of 62.13%, outperforming Naive Bayes, which achieved 52.44%, representing an improvement of 9.69%. These results demonstrate that SVM is more effective in handling high-dimensional TF-IDF feature spaces for multi-class sentiment classification and can support the development of public sentiment monitoring systems for Generative AI technologies.

References

A. Brasoveanu, M. Moodie, and R. Agrawal, “Textual evidence for the perfunctoriness of independent medical reviews,” in CEUR Workshop Proceedings, CEUR-WS, 2020, pp. 1–9. doi: 10.1145

A. Dabis and C. Csáki, “AI and ethics: Investigating the first policy responses of higher education institutions to the challenge of generative AI,” Humanit. Soc. Sci. Commun., vol. 11, no. 1, Dec. 2024, doi: 10.1057/s41599-024-03526-z.

A. Haleem, M. Javaid, and R. P. Singh, “An era of ChatGPT as a significant futuristic support tool: A study on features, abilities, and challenges,” BenchCouncil Transactions on Benchmarks, Standards and Evaluations, vol. 2, no. 4, Oct. 2022, doi: 10.1016/j.tbench.2023.100089.

A. Palanivinayagam, C. Z. El-Bayeh, and R. Damaševičius, “Twenty Years of Machine-Learning-Based Text Classification: A Systematic Review,” Algorithms, vol. 16, no. 5, May 2023, doi: 10.3390/a16050236.

C. Leiter et al., “ChatGPT: A Meta-Analysis after 2.5 Months,” Feb. 2023, [Online]. Available: http://arxiv.org/abs/2302.13795

D. Heaton, J. Clos, E. Nichele, and J. E. Fischer, “‘The ChatGPT bot is causing panic now – but it’ll soon be as mundane a tool as Excel’: analysing topics, sentiment and emotions relating to ChatGPT on Twitter,” Pers. Ubiquitous Comput., vol. 28, no. 6, pp. 875–894, Dec. 2024, doi: 10.1007/s00779-024-01811-x.

D. R. E. Cotton, P. A. Cotton, and J. R. Shipway, “Chatting and cheating: Ensuring academic integrity in the era of ChatGPT,” Innovations in Education and Teaching International, vol. 61, no. 2, pp. 228–239, 2024, doi: 10.1080/14703297.2023.2190148.

D. Wilimitis and C. G. Walsh, “Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial,” JMIR AI, vol. 2, no. 1, 2023, doi: 10.2196/49023.

H. Setyawan, L. Ma’rifatul Azizah, and A. Y. Pradani, “Sentiment Analysis of Public Responses on Indonesia Government Using Naïve Bayes and Support Vector Machine,” 2023. [Online]. Available: https://t.co/FKZjLtyWaN

M. Siino, I. Tinnirello, and M. La Cascia, “Is text preprocessing still worth the time? A comparative survey on the influence of popular preprocessing methods on Transformers and traditional classifiers,” Inf. Syst., vol. 121, Mar. 2024,

N. Arifah Wahyuni, D. Rizky Amelia, and tishom Al Khoiry, “Exploring Public Sentiment on ChatGPT-An Analysis of Twitter Data Article Info,” JEE, vol. 13, no. 1, pp. 51–60, 2024, [Online]. Available:

OpenAI et al., “GPT-4 Technical Report,” Mar. 2024, [Online]. Available: http://arxiv.org/abs/2303.08774

P. P. Ray, “ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope,” Jan. 01, 2023, KeAi Communications Co. doi: 10.1016/j.iotcps.2023.04.003.

R. Bommasani et al., “On the Opportunities and Risks of Foundation Models,” Jul. 2022, [Online]. Available: http://arxiv.org/abs/2108.07258

R. Peres, M. Schreier, D. Schweidel, and A. Sorescu, “Editorial: On ChatGPT and beyond: How generative artificial intelligence may affect research, teaching, and practice,” International Journal of Research in Marketing, vol. 40, no. 2, pp. 269–275, Jun. 2023, doi: 10.1016/j.ijresmar.2023.03.001.

S. Bubeck et al., “Sparks of Artificial General Intelligence: Early experiments with GPT-4,” Apr. 2023, [Online]. Available: http://arxiv.org/abs/2303.12712

S. Farhadpour, T. A. Warner, and A. E. Maxwell, “Selecting and Interpreting Multiclass Loss and Accuracy Assessment Metrics for Classifications with Class Imbalance: Guidance and Best Practices,” Remote Sens. (Basel)., vol. 16, no. 3, Feb. 2024,

S. Szeghalmy and A. Fazekas, “A Comparative Study of the Use of Stratified Cross-Validation and Distribution-Balanced Stratified Cross-Validation in Imbalanced Learning,” Sensors, vol. 23, no. 4, Feb. 2023, doi: 10.3390/s23042333.

V. M. Ngo, “Does ChatGPT change artificial intelligence-enabled marketing capability? Social media investigation of public sentiment and usage,” Global Media and China, vol. 9, no. 1, pp. 101–125, Mar. 2024, doi: 10.1177/20594364241228880.

W. M. Lim, A. Gunasekara, J. L. Pallant, J. I. Pallant, and E. Pechenkina, “Generative AI and the future of education: Ragnarök or reformation? A paradoxical perspective from management educators,” International Journal of Management Education, vol. 21, no. 2, Jul. 2023, doi: 10.1016/j.ijme.2023.100790.

Y. K. Dwivedi et al., “‘So what if ChatGPT wrote it?’ Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy,” Int. J. Inf. Manage., vol. 71, Aug. 2023, doi: 10.1016/j.ijinfomgt.2023.102642.