Jurnal Kecerdasan Buatan dan Teknologi Informasi Vol.
No.
May 2025, pp.
e-ISSN: 2964-2922, p-ISSN: 2963-6191 DOI : https://doi.
org/10.
69916/jkbti.
BERT SENTIMENT ANALYSIS FOR DETECTING FRAUDULENT MESSAGES
Yuyun Yusnida Lase1.
Arif Aryaguna Nauli*2.
Doni Ganda Marbungaran Mahulae3
1,2,3
Software Engineering Technology.
Information Technology.
Politeknik Negeri Medan.
Medan.
Indonesia Email: 1yuyunlase@polmed.
id, 2arifaryagunanauli@polmed.
donigandamarbungaran@students.
(Received: December 22, 2024.
Revised: May 20, 2025.
Accepted: May 24, 2.
Abstract With the increasing prevalence of digital communication, fraudulent SMS messages have become a growing This study employs a BERT-based sentiment approach to classify SMS messages into four categories:
fraud, gambling.
Unsecured Credit (KTA Ae Kredit Tanpa Aguna.
, and others.
These categories were determined based on content analysis and common patterns found in high-risk messages, such as suspicious transaction invitations .
, betting promotions .
, offers for unsecured loans (KTA), and other messages that do not fall into the three main categories.
The dataset used consists of approximately 20,000 message records, which underwent data cleaning, tokenization, and manual labeling based on the aforementioned criteria.
The model was trained using the AdamW optimizer with CrossEntropyLoss as the loss function for multi-class classification.
Training was conducted over 3 epochs, a number chosen based on observations of evaluation metrics on the validation data, which showed that model accuracy began to plateau after the third epoch, while overfitting started to occur in subsequent epochs.
After training, the model achieved an average accuracy of 92%.
This result indicates that the BERT model is effective in understanding patterns in text messages and capable of classifying message categories with a high level of accuracy.
These findings support the application of BERT technology in the efficient detection and identification of fraudulent messages.
Keywords: BERT, fraud detection, machine learning, sentiment analysis.
SMS classification INTRODUCTION The development of technology today greatly affects the information circulating, information technology is increasing so rapidly that it has a significant impact in various aspects of daily human life.
One of the impacts is the increasing volume of digital communication, including the use of short messages and social media as a means of Behind its enormous benefits there is a threat that often occurs.
Cell phones are now considered to be a kind of loyal friend to their users.
The widespread use of mobile phones, particularly for SMS communication, has become an integral part of modern life.
Short Message Service (SMS) is a valuable service offered by the telecommunications industry and contributes significantly to the Gross National Income (GNI) of developing countries .
This facility is used by millions of users every day due to its simplicity, accessibility, instant delivery, and low rates compared to phone calls.
However, the widespread use of SMS has also led to an increase in unwanted spam messages, including advertisements and scams.
Nigeria, in particular, faces a significant SMS spam problem that has compromised the privacy of mobile phone users with phishing and fraud .
Smishing, a hybrid of SMS and phishing, is a rapidly increasing mobile security threat .
where attackers use text messages to deceive users by including email IDs, website links, or phone numbers to extract sensitive information or lure victims with fraudulent offers .
In contrast to traditional email phishing, smishing exploits the immediacy and exceptionally high open-rates of SMS messagesAioften above 90% within minutes of receiptAi making it a highly effective and dangerous attack vector .
The urgency to address this threat is underscored by recent scams related to COVID-19, insurance, food delivery, and government programs, resulting in significant financial losses .
Spam detection has traditionally relied on keyword filters to distinguish between spam and legitimate messages for the past two decades .
Recently, advanced methods such as Statistical Learning Theory.
Artificial Neural Networks (ANN), and Support Vector Machines (SVM) have emerged.
However, according to .
many SMS spam detection methodsAiincluding Statistical Learning Theory.
Artificial Neural Networks (ANN), and Support Vector Machines (SVM)Aiexhibit unpredictable and inconsistent performance when trained on outdated or imbalanced datasets, with no clear explanation for the variations.
There are many spam filtering however, as each of these techniques has its own strengths and weaknesses, no single spam filtering strategy can be guaranteed to be 100% effective in eradicating the spam problem.
In practice, rule-based filtering is commonly applied, where predefined keywords and sender blacklists are used to block suspicious SMS messages.
Statistical techniques such as Bayesian filtering analyze word probabilities to classify messages, while machine Yuyun yusnida lase, et al.
, bert sentiment analysis for detecting fraudulent messages learning methods like Support Vector Machines (SVM) and Random Forests have been employed to detect spam based on textual features like term frequency-inverse document frequency (TF-IDF) and message structure.
More recently, deep learning models, including Convolutional Neural Networks (CNN.
and Recurrent Neural Networks (RNN.
, have been implemented to capture semantic relationships in text, providing higher accuracy in spam The application of text mining techniques to SMS continues to enhance the effectiveness of detecting and classifying spam messages.
Moreover, according to .
fine-tuned a RoBERTa variant on a benchmark SMS spam dataset, achieving 99.
84 % accuracy in spam classification.
They also applied Explainable AI techniques to compute positive and negative coefficient scores, providing insights into the key features driving model predictions and enhancing transparency in text-based SMS spam detection.
Here are some literature reviews related to this topic:
BERT for Smishing and SMS Scam Detection BERT is a transformer-based NLP model capable of capturing deep contextual relationships in text.
BERT was fine-tuned for smishing detection by employing optimized tokenization strategies and contextual embedding techniques.
This approach significantly enhanced the modelAos classification accuracy on smishing datasets, demonstrating BERTAos effectiveness for identifying phishing SMS .
The proposed SMS scam detection system first applies optical character recognition to extract text from image-based messages, then leverages unsupervised feature learning alongside a deep semi-supervised classifier to achieve high accuracy in identifying fraudulent SMS .
BERT in SMS Spam and Fraud Message Detection BERT (Bidirectional Encoder Representations from Transformer.
is used to generate deep contextual embeddings for SMS text, which are then combined with traditional classifiersAisuch as Nayve Bayes.
SVM.
Logistic Regression.
Gradient Boosting, and Random ForestAito distinguish spam from legitimate messages.
The Nayve Bayes BERT model achieved the highest accuracy of 97.
31% with a runtime of just 0.
3 seconds on the test set .
A BERT-based spam detector was fine-tuned on four benchmark corporaAiSMS Spam Collection.
SpamAssassin.
Ling-Spam, and EnronAiachieving classification accuracies of 98.
62%, 97.
13%, and 99.
28%, respectively .
Other BERT Applications and Context Three target-dependent variants of the BERT_BASE model were implementedAiwith special input representations that mark the target termAito perform aspect-level sentiment classification.
By incorporating target information into BERTAos contextual embeddings, the proposed TD-BERT models achieve new state-ofthe-art performance on the SemEval-2014 Laptop.
Restaurant, and Twitter datasets, outperforming both traditional feature-based methods and earlier embedding-based approaches .
A systematic review of 34 empirical studies identified three primary factors influencing susceptibility to online fraud: message characteristics .
, urgency framing and emotional appeal.
, dispositional traits .
personality factors and cognitive biase.
, and prior experience .
, knowledge of scams and past Understanding these dimensions can inform the design of more effective anti-fraud interventions and policies .
Understanding and Optimizing BERT: Attention Patterns.
Aspect-Based Sentiment Analysis, and Compression Techniques.
By probing BERTAos attention layers and hidden representations on annotated ABSA datasets.
Xu et al.
show that only a small number of selfattention heads encode aspect and opinion terms, whereas most representation capacity captures fineAagrained domain semantics.
They argue that these insights can guide future improvements in self-supervised pre-training and fine-tuning strategies for aspect-based sentiment analysis .
Rogers et al.
survey over 150 studies on the BERT model to synthesize our understanding of how BERT learns and represents different types of informationAifrom syntactic structures to semantic meanings.
They review common modifications to BERTAos pre-training objectives and architecture for improved efficiency, discuss the challenges of overparameterization, and outline compression methods such as distillation and pruning to reduce model size without sacrificing performance.
Finally, they propose directions for future research to further demystify and optimize BERT-based systems .
Using a subset of GLUE tasks and a set of handcrafted features-of-interest, we carry out a qualitative and quantitative analysis of information encoded by individual BERTAos self-attention heads.
Our findings indicate that a small number of attention patterns recur across different headsAisuggesting model overparameterizationAiand that disabling certain heads can actually improve performance over the standard fine-tuned BERT models .
This research aims to develop and apply a BERT-based sentiment analysis approach to detect fraudulent messages, particularly in short text formats such as SMS.
By leveraging BERTAos ability to capture deep contextual meanings, the study focuses on classifying messages into high-risk categories such as fraud, gambling, unsecured loans (KTA), and others.
The proposed method is expected to improve detection accuracy and provide a more effective solution to challenges such as model overparameterization and inconsistent performance commonly faced Yuyun yusnida lase, et al.
, bert sentiment analysis for detecting fraudulent messages by traditional approaches.
The findings of this research are intended to support the development of smarter and more responsive digital security systems that protect users from smishing threats and high-risk spam messages.
RESEARCH METHODS
1 Datasets The dataset used is based on messages in the message application.
the total data obtained is 20,829 data which is divided into 3 columns, 1 column for numeric, namely the type_pred column as a category of message type and 2 more columns, namely message as message content and sender, namely the number of the message sender.
The data will be analyzed using sentiment which will produce a model that can detect the possibility of an input message is a fraudulent message or not a fraudulent message.
2 Data Cleaning.
The data cleaning process is an important step in this research to ensure that the dataset used is of high quality and ready for further analysis.
In this process, we will remove missing values or empty data, remove duplicate data and normalize the data .
hange to lowercase, remove punctuation marks, remove excess spaces and other.
so that the data used will be usable and get good results.
Figure 1.
The original data before any cleaning was applied Become:
Figure 2.
Final data after data cleaning was 3 Data Pre-processing In Data Pre-processing, the data in the type_pred column is converted into numeric values through a label encoding process.
The fraud category is encoded as 0, gambling as 1.
KTA as 2, and others as 3.
This process is important to facilitate the machine learning model in understanding and processing the data.
Next, the message column will undergo tokenization using the BERT tokenizer to convert the text into a format that can be understood by the model.
4 Model Training The model training phase is a crucial step in the development of a message classification system based on BERT.
The BERT model is used to classify short messages (SMS) into four categories: fraud, gambling.
KTA .
nsecured loan.
, and others.
The following are the steps carried out during the model training process:
Text Preprocessing Before the data model can be trained, the data must first be pre-processed so that it can be used by the model, data pre-processing includes 2 stages, namely:
Tokenization At this stage, each message in the dataset will be converted into tokens that the BERT model can understand.
In this case, the tokenizer will convert the messages in the message into a sequence of tokens.
Yuyun yusnida lase, et al.
, bert sentiment analysis for detecting fraudulent messages Label Encoding At this stage, the category label of the message, namely type_pred which consists of 4 classes, namely fraud, gambling.
KTA, and others, will be converted into numerical values.
where the results will be 0 = Fraud, 1 = Gambling, 2 = KTA, and 3 = Other others.
Figure 3.
Text preprocessing using Pyhton Train-Test Split At this stage, the dataset that has been processed earlier will be processed again and then divided into two subsets, namely the training set and also the testing set, with a proportion of 80 percent of the data for training and 20 percent of the data for testing.
This test is done randomly using the skylern library with the train_test_split Figure 4.
Dataset splitting for Training and Testing Sets Fine-turning Model BERT In this stage, a BERT model that has been pre-trained will be used as the basis for further training .
ine turnin.
with more specific datasets.
Fine-turning is done by using the BertForSequenceClassification model in the Transformers library which is optimized for text classification.
This model is adjusted to the number of categories/classes that exist, which is 4 classes.
Yuyun yusnida lase, et al.
, bert sentiment analysis for detecting fraudulent messages Figure 5.
Pre-trained BERT Model for Sequence Classification Training Process In this stage, the model will be trained using the AdamW optimization algorithm and the CrossEntropyLoss loss function which is generally used for multi-class classification.
This training will last 3 epochs, where the model will learn from the training set to minimize the loss value.
Figure 6.
Training Process Model Evaluation After the previous training process has been completed, the model will be evaluated using a testing set that serves to measure its performance.
The evaluation carried out at this stage is to test accuracy and also test sample messages to be detected according to existing categories.
Yuyun yusnida lase, et al.
, bert sentiment analysis for detecting fraudulent messages Figure 7.
Accuracy Test Model Evaluation on Test Set Figure 8.
Message Test Model Evaluation on Test Set
RESULTS AND DISCUSSION
1 System Implementation The method used has been tested, and the results demonstrate that SMS classification can be conducted effectively using the BERT Sentiment model.
In this study, the model was trained to classify SMS messages into four categories: Fraud.
Gambling.
KTA (Kredit Tanpa Aguna.
, and Others.
The evaluation of the model's performance yielded high scores, with precision values of 0.
92 for Fraud, 0.
89 for Gambling, 0.
91 for KTA, and 94 for Others.
The average precision, recall, and F1-score across all categories were 0.
92, 0.
90, and 0.
These metrics confirm that the model is capable of accurately learning and generalizing the patterns present in various types of SMS content.
Compared to traditional methods such as keyword-based filtering or classical machine learning models like SVM or Random Forest.
BERT provides improved contextual understanding and higher classification accuracy, making it a reliable approach for real-world fraud message detection.
2 Preparing Libraries and Data The First step in the system implementation process is to prepare the library and research data.
The dataset used has been done data cleaning before so that the dataset used is much better for processing.
Yuyun yusnida lase, et al.
, bert sentiment analysis for detecting fraudulent messages Figure 9.
Library and Data Settings Data Selection Define the variables to be analyzed Figure 10.
Selection Data 4 Data Visualization To understand the data distribution, a visualization using a bar chart showing the number of messages in each category should be done.
Figure 11.
Chart Visualization Data Yuyun yusnida lase, et al.
, bert sentiment analysis for detecting fraudulent messages 5 Fine turning Model BERT Fine-tuning was performed on the BERT model using a pre-processed dataset.
This model was optimized to perform classification tasks across four categories.
The dataset was split into 80% training data and 20% test data to ensure generalizability of the model.
The training process employed the AdamW optimization algorithm with a learning rate that was tuned based on validation performance.
After training for 3 epochsAiselected based on early signs of overfitting and plateauing accuracyAithe model demonstrated strong learning capabilities in recognizing text patterns and generating accurate category predictions for SMS messages.
The evaluation metrics on the test set showed an average precision of 0.
92, recall of 0.
90, and F1-score of 0.
90 across all classes, confirming the modelAos effectiveness in classifying various types of SMS content with high accuracy.
Figure 12.
Accuracy and loss progression during BERT model training 6 Evaluation Results The trained model will be evaluated using performance metrics such as accuracy, precision, recall, and F1score.
The evaluation results can be seen in the following table:
Table 1.
Evaluation Results Category Precision Recall F1-Score Fraud Gambling Unsecured Loan Others Average 7 Scatter Plot Visualization To visualize the prediction results, a scatter plot is used to show the relationship between a predicted value and the actual label.
Yuyun yusnida lase, et al.
, bert sentiment analysis for detecting fraudulent messages Figure 13.
Scatter plot of token length distribution across categories Figure 14.
Scatter plot based on model probabilities.
CONCLUSION
From this research it can be concluded that successfully implementing the BERT Sentiment model to detect fraudulent SMS messages very well, this model can be trained using a dataset divided into 4 categories: fraud.
KTA gambling, and others, the process in this study includes several stages that are quite complex starting from data cleaning, label encoding, tokenization, model training with the Adam W algorithm, and also performance evaluation using precision, recall and F1-Score metrics.
The evaluation results have shown that the performance is very good with precision, recall and F1-Score values 92, 0.
90, and 0.
Then also the use of scatter plots to visualize the prediction results can also show a good relationship between token length and model probability, so it can be concluded that the model can understand text patterns effectively, so the conclusion is that BERT Sentiment analysis to detect fraudulent messages can be done very well and can also be used as a good tool to detect fraudulent messages in today's digital age.
REFERENCES