Jurnal Ilmu Komputer dan Informatika (JIKI) P-ISSN: 2807-6664
E-ISSN: 2807-6591
Vol.
No.
December 2025.
Page.
https://jiki.
jurnal-id.
DOI: https://doi.
org/10.
54082/jiki.
Limitations of Support Vector Machine and Random Forest in Multi-Class Sentiment Analysis: Evidence from Neutral Sentiment Misclassification on Imbalanced Data Ardy Wicaksono*1.
Suyahman2.
Muhammad Anwar Fauzi3.
Deny Prasetyo4.
Dwi Utari Iswavigra5.
Yulaikha Mar'atullatifah6.
Agatha Pricillia Sekar Tamtomo7.
Muhammad Adi
Pratama8
1, 2, 4, 5, 6
Computer Science.
Universitas Sugeng Hartono.
Indonesia Digital Business.
Universitas Sugeng Hartono.
Indonesia English Language and Culture.
Universitas Sugeng Hartono.
Indonesia
3, 7
Email: 1ardywicaksono166@gmail.
Received : Dec 09, 2025.
Revised : Jan 20, 2026.
Accepted : Jan 23, 2026.
Published : Mar xx, 2026 Abstract The rapid growth of mobile applications has generated large volumes of user reviews, making automated sentiment analysis essential for understanding user perceptions.
Previous studies have shown that while machine learning models perform well in binary sentiment classification, they often struggle in multi-class settings, particularly in identifying neutral sentiment due to linguistic ambiguity and class imbalance.
This study aims to comparatively evaluate the performance of Support Vector Machine (SVM) and Random Forest in multi-class sentiment analysis, with a specific focus on their ability to handle the neutral sentiment category.
A supervised learning approach was employed using 2,112 Indonesian-language user reviews collected from the Google Play Store.
The data were preprocessed using standard Natural Language Processing techniques and represented using TF-IDF features.
Both models were trained and evaluated using accuracy, precision, recall.
F1-score, and confusion matrices.
The results indicate that SVM achieved an accuracy of 86.
52%, outperforming Random Forest, which obtained 83.
However, both models completely failed to classify the neutral sentiment class, yielding zero precision and recall for this category.
This failure highlights the dominant influence of severe class imbalance and insufficient feature discrimination for neutral sentiment.
The findings underscore a critical limitation of traditional machine learning approaches in multi-class sentiment analysis and emphasize the need for improved strategies, such as data resampling, advanced feature representation, or hybrid models, to enhance neutral sentiment detection in real-world applications.
Keywords: Machine-Learning.
Sentiment-Analysis.
Support-Vector-Machine.
Random-Forest.
Text-Classification.
Imbalanced-Data.
This work is an open access article licensed under a Creative Commons Attribution 4.
0 International License.
INTRODUCTION
The rapid proliferation of mobile applications has led to an exponential increase in user-generated content, particularly in the form of online reviews.
These reviews have become a critical source of information that influences application reputation, user adoption, and continuous system improvement .
, .
From a business and governance perspective, user feedback provides valuable insights into usability, system reliability, and service quality, making it an essential asset for data-driven decision making .
, .
However, the massive volume and unstructured nature of textual reviews render manual analysis inefficient and impractical, thereby necessitating automated sentiment analysis techniques .
, .
Sentiment analysis, a key task within Natural Language Processing (NLP), focuses on identifying and categorizing subjective opinions expressed in text .
, .
Early sentiment analysis research primarily addressed binary classification, distinguishing positive and negative opinions .
While binary sentiment classification has demonstrated high predictive performance, it fails to adequately represent real-world opinion distributions, where a substantial proportion of user feedback is neutral.
Jurnal Ilmu Komputer dan Informatika (JIKI) P-ISSN: 2807-6664
E-ISSN: 2807-6591
Vol.
No.
December 2025.
Page.
https://jiki.
jurnal-id.
DOI: https://doi.
org/10.
54082/jiki.
descriptive, or ambiguous in nature .
, .
Neutral sentiment often reflects mixed experiences or factual statements and lacks strong emotional polarity, making it considerably more difficult to identify using lexical-based features alone .
, .
To address this limitation, multi-class sentiment classification has been increasingly adopted, typically incorporating positive, neutral, and negative sentiment categories .
, .
However, multiclass sentiment analysis introduces additional challenges, particularly related to linguistic ambiguity and data distribution imbalance .
, .
In real-world application review datasets, positive sentiments often dominate, while neutral sentiments form a minority class, resulting in skewed class distributions .
, .
Such imbalance can bias machine learning models toward majority classes and significantly degrade their ability to learn meaningful patterns from minority sentiment categories .
, .
Numerous studies have demonstrated that class imbalance can lead to misleading performance evaluations when relying solely on overall accuracy .
, .
, .
High accuracy values may conceal poor predictive performance on minority classes, particularly neutral sentiment, which is often misclassified or entirely ignored .
, .
Consequently, class-level evaluation metrics such as precision, recall.
F1-score, and confusion matrix analysis are essential to reveal the true behavior of sentiment classification models in multi-class scenarios .
, .
Despite the growing popularity of deep learning and transformer-based architectures, traditional machine learning algorithms such as SVM and Random Forest remain widely used in sentiment analysis research .
, .
, .
These models are favored for their interpretability, computational efficiency, and robustness in high-dimensional sparse feature spaces such as TF-IDF representations .
, .
, .
In contrast, existing comparative studies predominantly focus on overall accuracy or binary sentiment settings, offering limited insight into how these models handle neutral sentiment under imbalanced data conditions .
, .
, .
Recent literature has begun to highlight the persistent difficulty of neutral sentiment classification, emphasizing that neutral opinions are frequently misclassified as either positive or negative due to overlapping lexical features and insufficient representation during model training .
, .
Nevertheless, many studies do not explicitly address this limitation in their analysis or discussion, resulting in an incomplete understanding of model reliability in practical sentiment analysis applications .
, .
Based on these observations, this study aims to conduct a comparative evaluation of SVM and Random Forest for multi-class sentiment analysis of mobile application reviews, with a specific focus on their ability to handle neutral sentiment in an imbalanced dataset.
Unlike prior studies that emphasize accuracy as the primary performance metric, this research adopts a class-level evaluation approach to uncover hidden model weaknesses.
The contributions of this study are: .
providing empirical evidence of neutral sentiment misclassification in traditional machine learning models.
demonstrating the limitations of accuracy-based evaluation in imbalanced multi-class sentiment analysis.
offering methodological insights to guide future research toward more robust and reliable sentiment classification frameworks.
METHOD
This study employed a structured methodological framework to analyze user sentiment toward the OSS mobile application using supervised machine learning techniques.
The methodological process consists of four main stages: data collection, text preprocessing, model training, and model evaluation.
Each stage was designed to ensure the integrity, reproducibility, and reliability of the sentiment classification pipeline.
The flowchart below illustrates the complete workflow to offer a visual overview of the sequential procedures involved in transforming raw review data into model-ready features and evaluating classification performance in Figure 1.
It presents the overall research workflow that structures the methodological procedures of this study.
The following subsections describe each methodological stage in detail.
Jurnal Ilmu Komputer dan Informatika (JIKI) P-ISSN: 2807-6664
E-ISSN: 2807-6591
Vol.
No.
December 2025.
Page.
https://jiki.
jurnal-id.
DOI: https://doi.
org/10.
54082/jiki.
Figure 1.
Research Workflow Data Collection The initial stage of this methodological framework involved data collection, which is critical for establishing a representative and robust dataset for sentiment analysis .
In the context of mobile application analysis, user feedback is primarily channeled through public review platforms .
The foundational dataset for this study was constructed from user-generated reviews extracted from the Google Play Store page of the target Open Sources Software mobile application.
The application is publicly accessible via the following URL:
Auhttps://play.
com/store/apps/details?id=id.
oss&hl=idAy To facilitate the automated and systematic retrieval of the reviews, the google-play-scraper Python library was utilized.
This tool was instrumental in extracting publicly available review data in A total of 2,112 of the most recent user reviews were collected.
The acquisition was strictly constrained to the Indonesian language and region .
sing the parameters ycoycaycuyci = "ycnycc" and ycaycuycycuycycyc = "ycnycc") to ensure linguistic homogeneity, which is crucial for subsequent language-dependent Natural Language Processing (NLP) and sentiment classification tasks.
The scraping process focused on extracting two primary attributes from each review instance, which constitute the core components for the sentiment analysis pipeline:
Textual Review Content (Conten.
: The raw text provided by the user.
Associated Star Rating (Scor.
: The numerical rating .
Ae5 star.
, which serves as the ground truth label for training the supervised machine learning model.
Data integrity was enforced during the acquisition phase through immediate filtering.
Reviews lacking textual content, duplicate entries, or those identified as non-Indonesian language content were systematically excluded.
Furthermore, only publicly visible review data were collected.
no personally identifiable information .
uch as usernames or email addresse.
was accessed or stored, aligning with established ethical research guidelines.
All retrieved data were organized and stored in a structured tabular format to facilitate seamless integration into the subsequent analytical procedures.
The data collection methodology fully complied with Google PlayAos terms of service regarding the use of publicly posted user reviews.
No attempts were made to infer user identities.
For the sake of research transparency and reproducibility, detailed scraping scripts and timestamp metadata are archived and can be provided upon request.
Jurnal Ilmu Komputer dan Informatika (JIKI) P-ISSN: 2807-6664
E-ISSN: 2807-6591
Vol.
No.
December 2025.
Page.
https://jiki.
jurnal-id.
DOI: https://doi.
org/10.
54082/jiki.
Table 1.
Sample of Filtered Google Play User Review Data score sentiment sekarang jadi mudah bikin nib sendiri Oke Terimakasih OSS yg bisa memberikan NIB secara .
harus pakai aplikasi ini biar mudah mantab, terimakasih aplikasi ga bisa di pake buat nib malah jd ribet ko ga bisa akses kelola nib ya .
selalu kelu.
uhh bikin susah ini mah super ribet banget aplikasi nyusahin rakyat ya.
Gak berfungsi Text Preprocessing The text preprocessing stage is essential in Natural Language Processing (NLP) methodologies, serving to transform raw, noisy, and unstructured textual data into a refined, consistent, and numerically representable format that is optimal for machine learning algorithms .
This process is paramount for enhancing feature quality and maximizing the classification model's performance .
First, each review was assigned a categorical sentiment label based on its associated numerical star rating, establishing the ground truth for classification .
This rule-based approach, which follows common practices in prior sentiment analysis studies, maps the ratings into a ternary classification schema: reviews rated 1Ae2 stars were labeled as Negative, ratings of 3 stars as Neutral, and ratings of 4Ae5 stars as Positive.
Following sentiment labeling, the textual content underwent rigorous cleaning and normalization to reduce noise and ensure linguistic consistency.
This included converting all text to lowercase, removing non-alphanumeric characters using regular expressions, and normalizing excessive These operations produced a standardized text corpus suitable for feature extraction.
Feature representation was conducted using the Term Frequency-Inverse Document Frequency (TF-IDF) method, implemented via the ycNyceycnyccyceycO yceycaycycuycycnycyceyc in the ycycaycnycoycnyc Oe ycoyceycaycycu library.
TF-IDF
assigns a weight to each term based on its frequency within a document and its inverse rarity across the entire dataset .
The TF-IDF score for a term yc in document ycc within corpus ya is calculated as:
ycIya Oe yayaya .
c, ycc, y.
= ycNya .
c, yc.
c, y.
Equation .
Shows the TF-DF Score A maximum vocabulary size of 5000 features was adopted to balance model expressiveness with computational efficiency.
The transformation generated a sparse matrix in which each review was represented as a weighted vector capturing its most informative terms.
Finally, the resultant feature set was partitioned into training and testing subsets using an 80:20 train-test split.
Crucially, stratification by sentiment class was applied to preserve the original class distribution across both sets.
This mitigates the risk of model bias toward overrepresented categories and ensures that the testing set provides an unbiased basis for performance evaluation.
The resulting training set was used to fit the classification models.
Model Training and Classifier Implementation The penultimate stage of the methodology involved training the selected supervised machine learning models using the feature-engineered dataset .
This study employed two distinct algorithms a linear classifier and an ensemble-based method to benchmark classification performance on the Indonesian sentiment corpus .
Both models were exclusively trained on the stratified 80% training subset of the dataset, ensuring a proportional and unbiased representation of the Positive.
Neutral, and Negative sentiment classes.
The Support Vector Machine (SVM) algorithm, specifically the Linear Support Vector Classifier .
aycnycuyceycayc ycIycOy.
, was utilized.
SVM operates by finding the optimal hyperplane that maximally separates the data points (TF-IDF feature vector.
belonging to different sentiment classes in the high-dimensional Jurnal Ilmu Komputer dan Informatika (JIKI) P-ISSN: 2807-6664
E-ISSN: 2807-6591
Vol.
No.
December 2025.
Page.
https://jiki.
jurnal-id.
DOI: https://doi.
org/10.
54082/jiki.
feature space.
The .
aycnycuyceycayc ycIycOy.
was specifically chosen due to its inherent efficiency and proven robust performance on high-dimensional and sparse text data, characteristic of TF-IDF representations .
The model was trained using the default hyperparameter settings, establishing a competitive and reproducible baseline performance without relying on resource-intensive optimization techniques.
The Random Forest (RF) algorithm was selected as the representative ensemble-based classifier.
RF operates by constructing a multitude of decision trees during the training phase and outputting the class that is the mode of the classes .
or mean prediction .
of the individual This method excels at capturing complex, non-linear relationships within the feature space .
The RF classifier was configured with 300 decision trees ( ycuyceycycycnycoycaycycuycyc = 300 ).
This specific configuration was chosen to achieve an optimal balance between enhancing model stability .
educing varianc.
and managing computational cost.
The ycycaycuyccycuyco_ycycycaycyce parameter was fixed to ensure the computational experiment is fully reproducible.
Both models learned the underlying patterns and weighted relationships between the TF-IDF features and the assigned sentiment polarity.
Following the training completion, the resulting classifiers were subjected to rigorous testing on the reserved 20% test set to assess their generalization capabilities, an evaluation process detailed in the subsequent section.
Model Evaluation and Model Metrics The final stage of the methodological workflow involved a rigorous Model Evaluation to objectively assess the predictive performance and generalization capability of the trained classifiers (SVM and RF) .
Evaluation is a crucial step that validates the utility of the constructed models using a reserved, unseen dataset.
For classification tasks, performance must be quantified using metrics that provide both a high-level summary and granular insight into class-specific behavior.
Accuracy was selected as the primary metric to summarize overall model performance.
Accuracy reflects the ratio of correctly predicted instances to the total number of samples in the test set .
Due to its direct interpretability, it is highly suitable for providing a clear, high-level reflection of the model's success in this multiclass sentiment classification context.
Accuracy is formally defined as:
ycNycycyce ycEycuycycnycycnycycnyceyc ycNycycyce ycAyceyciycaycycnycyceyc yaycaycaycycycaycayc = .
ycNycycyce ycEycuycycnycycnycyceyc ycNycycyce ycAyceyciycaycycnycyceyc yaycaycoycyce ycEycuycycnycycnycyceyc yaycaycoycyce ycAyceyciycaycycnycyceyc Equation .
Shows the Accuracy In addition to overall accuracy.
Confusion Matrices were generated for both algorithms.
Confusion Matrices provide a crucial, granular understanding of classification behavior across the three sentiment categories (Positive.
Neutral, and Negativ.
By mapping the actual classes against the predicted classes, the matrix allows for the quantification of True Positives .
orrectly classifie.
False Positives (Type I erro.
, and False Negatives (Type II erro.
This level of detail was essential, particularly considering the potential for class imbalance often inherent in user-generated review datasets .
here positive sentiments tend to dominat.
The Confusion Matrix facilitated the identification of systematic misclassification patterns, such as the tendency to confuse Neutral reviews with either Positive or Negative sentiments.
To further dissect the model's performance on individual classes, metrics derived from the Confusion Matrix namely Precision and Recall were analyzed.
These metrics provide insight into the model's predictive reliability for each category:
Precision: The proportion of positive identifications that were actually correct.
ycNycycyce ycEycuycycnycycnycyceyc ycEycyceycaycnycycnycuycu = ycNycycyce ycEycuycycnycycnycyceyc yaycaycoycyce ycEycuycycnycycnycyceyc Equation .
Shows the Precision Recall: The proportion of actual positives that were identified correctly.
ycNycycyce ycEycuycycnycycnycyceyc ycIyceycaycaycoyco = ycNycycyce ycEycuycycnycycnycyceyc yaycaycoycyce ycAyceyciycaycycnycyceyc Equation .
Shows the Recall Jurnal Ilmu Komputer dan Informatika (JIKI) P-ISSN: 2807-6664
E-ISSN: 2807-6591
Vol.
No.
December 2025.
Page.
https://jiki.
jurnal-id.
DOI: https://doi.
org/10.
54082/jiki.
Together, the overall accuracy, the visual interpretation of the Confusion Matrices, and the derived class-level metrics provided a comprehensive assessment of the modelsAo predictive performance and reliability in the ternary sentiment classification task.
RESULT
Exploratory Data Analysis: Sentiment Word Clouds Initial analysis of user sentiment was performed using Word Clouds, which visualize the frequency of words in the text data.
This step provides an intuitive understanding of the prevailing themes in positive and negative reviews Positive Sentiment The Word Cloud for positive reviews (Figure .
is dominated by words expressing satisfaction and utility.
Key terms such as "mantap" .
, "bisa" .
, "baik" .
, "bagus" .
, and "membantu" .
are prominent.
This indicates that users primarily appreciate the system's core functionality, ease of use, and overall service quality.
Figure 2.
Word Cloud for Positive Sentiment.
Negative Sentiment Conversely, the negative sentiment Word Cloud () highlights issues related to complexity and technical difficulties.
The most frequent words include "aplikasi" .
, "ribet" .
, "susah" .
, "email", "verifikasi" .
, and "login".
This strong pattern suggests that negative feedback is centered on specific pain points in the user experience, particularly concerning access, registration, and complicated procedural steps.
Figure 3.
Word Cloud for Negative Sentiment.
Model Performance Evaluation Two machine learning models.
Support Vector Machine (SVM) and Random Forest, were evaluated for their performance in classifying the three sentiment classes (Negative.
Neutral.
Positiv.
Jurnal Ilmu Komputer dan Informatika (JIKI) P-ISSN: 2807-6664
E-ISSN: 2807-6591
Vol.
No.
December 2025.
Page.
https://jiki.
jurnal-id.
DOI: https://doi.
org/10.
54082/jiki.
Support Vector Machine (SVM) Perfomance The SVM model achieved an overall accuracy of 86.
Detailed performance metrics for each class are presented in Table 2.
Table 2.
Classification Report for Random Forest.
Class Precision Recall F1-Score Support Negative Neutral Positive Accuracy Macro Avg Weighted Avg The high F1-Scores for the Negative .
and Positive .
classes demonstrate the model's high capability in identifying clear sentiment orientations.
However, the Confusion Matrix (Figure .
clearly illustrates the model's major deficiency: the Neutral class.
The model failed to correctly predict any neutral samples (True Positive = .
, resulting in zero values for Precision.
Recall, and F1-Score for this class.
The 19 neutral samples were incorrectly distributed across the Negative .
and Positive .
Figure 4.
Confusion Matrix for SVM Model.
Random Forest Performance The Random Forest model yielded an overall accuracy of 83.
The classification report is summarized in Table 3.
Table 3.
Classification Report for Random Forest.
Class Precision Recall F1-Score Support Negative Neutral Positive Accuracy Macro Avg Weighted Avg While exhibiting strong performance on the majority classes, the Random Forest model shared the same fundamental weakness as the SVM.
As shown in the Confusion Matrix (Figure .
, the model
Jurnal Ilmu Komputer dan Informatika (JIKI) P-ISSN: 2807-6664
E-ISSN: 2807-6591
Vol.
No.
December 2025.
Page.
https://jiki.
jurnal-id.
DOI: https://doi.
org/10.
54082/jiki.
was completely unable to correctly classify the Neutral class, with a True Positive count of 0, confirming 00 F1-Score.
Figure 5.
Confusion Matrix for Random Forest Model.
DISCUSSIONS
In accordance with the finished analysis result, the implication is obviously validated.
Comparative Analysis: The SVM model (Accuracy: 86.
52%) outperformed the Random Forest model (Accuracy: 83.
45%) by approximately 3%, establishing SVM as the superior classifier for this specific In regards to the discussion on Class Imbalance, the most critical finding is the pervasive failure of both advanced models to classify the Neutral class.
This is directly attributable to severe class imbalance, as the Neutral class accounts for only 4.
5% of the total data .
The high overall accuracy figures are thus misleading, as they primarily reflect the excellent classification of the majority classes (Negative and Positiv.
In a real-world scenario, this model limitation would render the classification of truly neutral feedback unreliable.
CONCLUSION
To sum up, as this study investigated the performance of SVM and Random Forest algorithms for multi-class sentiment analysis, the results indicate that the SVM model is the better classifier in terms of the effectiveness area, achieving an overall accuracy of 86.
Both models demonstrated high proficiency in distinguishing between Negative and Positive sentiments, as evidenced by their high F1scores in these categories.
However, the key conclusion is the complete failure of both models to predict the Neutral sentiment class (F1-score 0.
This critical limitation stems from the severe class imbalance in the dataset, where the Neutral class was overwhelmingly underrepresented.
For future research, it is strongly recommended that methods to mitigate class imbalance, such as oversampling or undersampling, are implemented to ensure that the classifier could learn meaningful patterns from the minority class.
Alternatively, transforming the problem into a binary classification task (Negative vs.
Positiv.
may yield more robust and reliable results given the current data distribution.
REFERENCES