JOISIE (Journal Of Information Systems And Informatics Engineerin.
Vol.
No.
Desember 2024.
Hlm 209-220
Received:14 September 2024,Revised: 27 September 2024,Accepted: 1 Oktober 2024
p- ISSN: 2503-5304
e- ISSN: 2527-3116
ANALYSING SERVICE QUALITY USING SENTIMENT ANALYSIS AND TOPIC
MODELING: A CASE STUDY OF THE LIVIN MANDIRI APPLICATION
Khairin Andini Safarah.
Dedi I.
Inan.
Ratna Juita.
Victor Arie L.
Sirait.
Department of Informatics Engineering.
Faculty of Engineering.
University of Papua.
Email: andinisafarah@gmail.
com1, d.
inan@unipa.
id2, r.
juita@unipa.
id3, lambadya421@gmail.
Abstract With the increasing number of m-banking users, understanding customer satisfaction has become crucial for banks in improving service quality and maintaining loyalty.
This study aims to evaluate the satisfaction level regarding the service quality of the Livin Mandiri m-banking application using sentiment analysis and topic modeling.
The data were gathered from 13,692 user reviews on the Google Play Store through web scraping techniques.
After data cleansing and processing, sentiment analysis was conducted to identify trends in positive, negative, and neutral sentiments.
Topic modeling using the Latent Semantic Indexing (LSI) method was employed to gain deeper insights into user discussions about service quality.
The findings reveal that, although the Livin Mandiri application offers various useful features, the majority of user reviews are negative.
Topic modeling further highlights that the primary complaints focus on technical issues such as transaction failures and verification challenges.
Additionally, the study indicates a need to enhance application stability and customer service to improve user satisfaction.
This study makes a significant contribution to understanding the service quality of m-banking applications by combining sentiment analysis and topic modeling, offering valuable insights for the future development and improvement of applications in the banking sector.
Keywords: Livin Mandiri.
Latent Semantic Indexing (LSI).
Sentiment Analysis.
Topic Modeling.
User Satisfaction.
INTRODUCTION
The rapid advancement of Information Technology (IT) in this era of globalization has been met with responses from companies and the banking sector, leading to demands for banking services to be faster, more convenient, and flexible (Risdiyanto et al.
, 2.
Various factors influence how individuals accept technology, encompassing both positive and negative aspects that may originate from internal or external sources (Inan et al.
, 2.
Mobile banking applications have become one of the most widely used banking services by consumers (Alalwan et al.
, 2.
One popular mobile banking application in Indonesia is Livin by Mandiri.
The Livin Mandiri application holds a rating of 4.
0 on the Google Play Store, with over 571.
In providing internet-based banking information technology services, the bank strives to deliver services that are designed to be clear, practical, and easy to understand, ensuring that customers find them user-friendly (Dianta & Zusrony, 2.
With the rise in mobile banking users in 2023.
Livin Mandiri saw a 55% growth in users, reaching 21 million by September, with a target of 25 million by the end of the year.
Bank Mandiri continues to innovate digitally to support user growth and transaction volume, underscoring the importance of user satisfaction in enhancing service quality and maintaining customer loyalty (Baabdullah et al.
, 2.
The research data was collected through a scraping process, followed by manual cleaning using Excel and further refinement during the text preprocessing stage.
This study aims to evaluate user satisfaction with the platform through sentiment analysis and topic modeling using the Latent Semantic Indexing (LSI) method.
On the Google Play Store, each application contains user reviews, which provide opinions on their satisfaction with the application and are often considered by prospective users when selecting an application (Aaputra et al.
, 2.
Sentiment analysis, often https://doi.
org/10.
35145/joisie.
JOISIE licensed under a Creative Commons Attribution-ShareAlike 4.
0 International License (CC BY-SA 4.
210 | Jurnal JOISIE.
Volume 8.
Nomor 2.
Desember 2024 referred to as opinion mining, is the process of identifying and revealing opinions, emotions, evaluations, attitudes, subjectivity, judgments, or sentiments contained within a text (Sepri et al.
However, sentiment analysis is crucial for understanding the level of user satisfaction and identifying complaints made by users of the Livin' by Mandiri application.
Therefore, it is necessary to integrate sentiment analysis with topic modeling to identify the topics discussed by users in their One effective method for performing sentiment analysis and topic modeling is through Latent Semantic Indexing (LSI).
This method can identify the main topics in user reviews and determine the tendency of positive or negative sentiment toward the application (Campbell et al.
, 2.
In this study, the author will use the Latent Semantic Indexing (LSI) approach to analyze sentiment and the topics discussed in reviews of the Livin Mandiri mobile banking application on the Google Play Store.
Previous research has applied sentiment analysis to evaluate user satisfaction with mobile applications.
Additionally.
Amornpashara et al.
used topic modeling with Latent Dirichlet Allocation (LDA) to analyze user reviews of mobile applications on the Google Play Store to understand the topics discussed by However, these studies have not comprehensively integrated sentiment analysis with topic Therefore, this research will employ the Latent Semantic Indexing (LSI) approach to analyze sentiment and the topics discussed in user reviews of the Livin' by Mandiri mobile application on the Google Play Store.
This study is expected to make a significant contribution to the fields of sentiment analysis and topic modeling, particularly in relation to mobile banking applications.
The findings are anticipated to provide valuable insights for the development of applications, aiming to enhance the quality and services of the Livin Mandiri application (Liu et al.
, 2.
Additionally, this research will enrich the literature on the use of Latent Semantic Indexing (LSI) in sentiment analysis and topic clustering.
RESEARCH METHODOLOGY
The research methodology begins with data collection using Python for scraping, aimed at analyzing user sentiment for the Livin Mandiri application.
The collected data is then manually cleaned using Microsoft Excel to ensure accuracy and cleanliness.
Following this, the data undergoes text preprocessing using the Orange tool to remove irrelevant words.
Further analysis is conducted through topic modeling using Latent Semantic Indexing (LSI) to understand and identify the sentiments expressed by users.
Additionally.
LSI helps reduce noise and improve the accuracy of the analysis (Tscherne et al.
, 2.
Figure 1.
Research Flow
1 RESEARCH MODEL DESIGN
The research design employed in this study utilizes the Orange tool to enhance the effectiveness of analysis and prediction by leveraging various widgets provided for classification using the LSI topic modeling method, as illustrated in the image below.
This entire process is designed to conduct an in-depth text analysis by utilizing topic modeling to identify and visualize the main topics within the corpus, as well as to understand the relationships and distribution among these topics.
https://doi.
org/10.
35145/joisie.
JOISIE licensed under a Creative Commons Attribution-ShareAlike 4.
0 International License (CC BY-SA 4.
Safarah.
Analysing Service Quality Using Sentiment Analysis And Topic Modeling: A Case Study Of The Livin Mandiri Application, 209-.
211 Figure 2.
Research Model Design in Orange 2 DATA COLLECTION To support this research, data was collected from user reviews of the Livin Mandiri application, obtained from the Play Store, which has a rating of 4.
1 based on 562,000 reviews.
Data collection was performed using web scraping with the Python programming language, utilizing the token 'id.
livin' as the key for data retrieval.
The collected comments were then saved in a spreadsheet or CSV file format to facilitate the subsequent steps.
The next step is to analyze the sentiment of the reviews by categorizing the comments into positive, negative, and neutral sentiments to understand users' perceptions of the Livin Mandiri application.
Figure 3.
Rating of the Livin Mandiri Application 3 DATA PROCESSING In the data processing to be conducted, the text data obtained from user reviews of the Livin' by Mandiri application will be processed using Orange, a data mining software capable of data analysis and visualization (Singh et al.
, 2.
Data Preprocessing the collected reviews will undergo cleaning, filtering, or augmentation before analysis.
The primary objective of this preprocessing is to ensure that the data used is of high quality and relevance (Habibi et al.
, 2.
This step is essential for preparing the data for accurate and meaningful analysis:
Cleansing The data cleansing process involves removing duplicate reviews from the Livin Mandiri application and eliminating specific characteristics such as usernames, hashtags.
URLs, and typographical errors.
This step is crucial to ensure that the data is clean, consistent, and ready for further analysis.
Tokenizing https://doi.
org/10.
35145/joisie.
JOISIE licensed under a Creative Commons Attribution-ShareAlike 4.
0 International License (CC BY-SA 4.
212 | Jurnal JOISIE.
Volume 8.
Nomor 2.
Desember 2024 This process involves separating text into individual words (Sari & Hayuningtyas, 2.
For example, the sentence " As long as there is Livin, all transactions are made easier", "can withdraw cash", "cheap monthly fee" would be tokenized into "As long as," "there is," "livin," "all," "transactions," "so," "made easy," "can," "withdraw," "cash," "fee," "monthly," "cheap".
This step is fundamental in preparing the text for further analysis by breaking it down into manageable units.
Stopword To remove certain verbs, adjectives, and other less meaningful words, the text index can be augmented with a stopword list (Giovani et al.
, 2.
This process involves deleting common words that do not carry significant meaning, such as "dan" ("and") or "atau" ("or").
This step is essential for refining the text, allowing the analysis to focus on the more meaningful and relevant content.
Transformation Convert all the letters in the text to lowercase to ensure consistency in letter casing ( et al.
, 2.
This step is important for standardizing the text, making it easier to process and analyze without discrepancies due to differences in capitalization.
Filtering Filtering serves to clean the text data by removing irrelevant or non-essential elements before analysis.
This includes eliminating words that are not relevant, such as "and" "and", "which" "which", and "untuk" "for", as well as filtering out words based on their frequency to reduce the presence of words that appear too rarely or too frequently.
This process helps refine the dataset, making it more suitable for meaningful analysis.
4 SENTIMENT ANALYSIS
Sentiment refers to attitudes, thoughts, or judgments driven by feelings.
Sentiment analysis, also known as opinion mining, studies peopleAos sentiments toward specific entities.
The internet is a rich source of sentiment information.
Sentiment analysis is widely applied across various fields and has become one of the most active research topics, with a growing number of research findings published (Rahmad & Pribadi, 2.
This technique enables the identification and classification of sentiments contained within various textual data sources, such as product reviews, social media comments, or customer feedback.
By applying machine learning algorithms and computational linguistic techniques, sentiment analysis can reveal whether a text expresses positive, negative, or neutral attitudes toward a particular topic or entity.
5 TOPIC MODELING LATENT SEMANTIC INDEXING (LSI)
In this step, the results of the sentiment analysis are further examined to identify the topics discussed by users of the application, with the analysis guided by the number of samples used in the The objective is to enable application developers to understand user concerns regarding satisfaction while using the application, which is particularly crucial given the numerous negative reviews circulating.
During the topic modeling phase, the algorithm searches for and clusters user reviews based on frequently co-occurring phrases, employing an unsupervised learning approach to analyze the words in the original text and identify patterns within large datasets (Ponweiser, 2.
While methods such as Latent Semantic Indexing (LSI) and Singular Value Decomposition (SVD) provide insights into the frequently occurring themes or concepts (Fernando & Toba, 2.
, challenges remain that must be addressed.
To understand the issues occurring in sentiment analysis and topic modeling, it is essential to conduct a thorough analysis of the data used, including an examination of the sample size and the context of the reviews, which may not have been detected in the initial analysis.
Furthermore, an evaluation of the analytical methodology, including the algorithms applied, should also be considered, with alternative approaches such as supervised learning https://doi.
org/10.
35145/joisie.
JOISIE licensed under a Creative Commons Attribution-ShareAlike 4.
0 International License (CC BY-SA 4.
Safarah.
Analysing Service Quality Using Sentiment Analysis And Topic Modeling: A Case Study Of The Livin Mandiri Application, 209-.
213 potentially yielding more accurate results (Tscherne et al.
, 2.
Additionally, gathering direct feedback from users through interviews or surveys can provide further insights into their experiences.
Benchmarking against similar applications that have successfully implemented sentiment analysis can offer ideas on best practices to adopt.
By taking these steps, developers can better understand and address the challenges encountered during the analysis, thereby enhancing overall user satisfaction.
3 RESULTS AND DISCUSSION
1 SCRAPING DATA
From the data collection process through web scraping, a total of 16,686 comments were successfully gathered from the Play Store.
This scraping process involved the use of tools or scripts capable of accessing the Play Store pages, navigating to the review section, and collecting review texts along with metadata such as ratings, dates, user identities, and user comments.
After preprocessing to remove irrelevant data, the number of comments remaining and ready for analysis amounted to 13,692 reviews related to the Livin Mandiri application on the Play Store.
Figure 4.
Data Scraping Results 2 PREPROCESS TEXT The text preprocessing transformation process involves several critical steps to prepare the text data for further analysis.
First, the text is converted to lowercase using the "lowercase" option to eliminate case differences.
Next, accents are removed by clicking "remove accents" to standardize the Tokenization is performed using the "regexp" option, which splits the text into individual words or tokens.
Filtering continues with the "stopword" option, which removes common, non-essential words for the analysis, and the "relative" option is used to adjust word frequency.
To include languagespecific stopwords in Indonesian, the "stopword id text" option is utilized to ensure that common Indonesian words are also filtered out.
These steps are crucial for producing clean, well-prepared text data suitable for further analysis.
https://doi.
org/10.
35145/joisie.
JOISIE licensed under a Creative Commons Attribution-ShareAlike 4.
0 International License (CC BY-SA 4.
214 | Jurnal JOISIE.
Volume 8.
Nomor 2.
Desember 2024 Figure 5.
Preprocess results text tools orange Cleansing Expanding abbreviations into full terms is a crucial step in text data analysis to ensure consistency and improve accuracy.
This process, known as "abbreviation expansion," aims to reduce ambiguity in the text.
In Microsoft Excel, the first step in cleaning the data is performed using the "Find and Replace" feature.
By accessing this option, users can input abbreviations like "app" and replace them with "aplikasi.
" Excel will then automatically search for and replace each occurrence of the abbreviation within the data cells, ensuring consistent terminology usage and enhancing the accuracy of the data analysis.
Figure 6.
Excel Cleansing Results 3 TOPIC MODELLING Based on the results of topic modeling using Latent Semantic Indexing, the analysis of user feedback on the Livin Mandiri application reveals that, despite positive comments regarding the application's features and performance, many users have encountered serious technical issues.
The primary complaints include difficulties with application updates, failed transfers, verification problems, and password issues.
Furthermore, inadequate customer support has also emerged as a https://doi.
org/10.
35145/joisie.
JOISIE licensed under a Creative Commons Attribution-ShareAlike 4.
0 International License (CC BY-SA 4.
Safarah.
Analysing Service Quality Using Sentiment Analysis And Topic Modeling: A Case Study Of The Livin Mandiri Application, 209-.
215 significant concern.
Users frequently seek assistance with these issues, underscoring the need for improvements in both technical aspects and customer service to enhance overall satisfaction.
Figure 7.
Results of Topic Modeling Using Latent Semantic Indexing Table 1.
Interpretation of Topics No.
Topik application, livin, mandiri, update, card, help, and, open, very application, and, app, this, mobile, good, online, number, they, different app, card, livin, mandiri, atm, withdraw, password, bank, feature, debit card, good, livin, update, application, atm, withdraw, number, online, can't good, livin, application, card, atm, withdraw, online, customer, can't, update, livin, good, app, password, new, help, always, application, menu password, good, new, input, application, update, step, back, menu, after Pembahas Applicatio n updates Applicatio n features User Feature User Applicatio n security Applicatio n issues password, good, new, input, application, update, step, back, menu, after Applicatio mandiri, can't, livin, menu, very, card, new, account, help, maintenance https://doi.
org/10.
35145/joisie.
JOISIE licensed under a Creative Commons Attribution-ShareAlike 4.
0 International License (CC BY-SA 4.
216 | Jurnal JOISIE.
Volume 8.
Nomor 2.
Desember 2024 mandiri, cant, livin, menu, banget, kartu, new, rekening, tolong, verification, transfer, failed, very.
ID card, login, photo, account, credit, credit, transfer, phone, center, help, bca, issues, internet, connection, feature, account, help, open, bank, withdraw, transfer, mandiri, failed, application, good, feature, this, mobile, banking, have, maintenance, very, error, account, bank, automatic, settings, mandiri, failed verification, maintenance Applicatio User Connectivi ty issues Transactio n issues Satisfactio n level Applicatio n errors The table above presents the results of 2 positive topics, 13 negative topics, and 3 neutral topics.
Overall, the Livin by Mandiri application is highly useful for users in accessing banking services.
Although updates and new features are often appreciated, several technical issues and usability challenges need to be addressed to enhance user satisfaction.
Common problems include application errors, transaction failures, and difficulties in the verification process.
Users value the convenience provided by online services, but continuous improvements in security, stability, and customer support are essential to ensure a better and more reliable experience.
4 SENTIMENT ANALYSIS
Tabel 2.
Rating Sentiment Analysis Results Rating Sentime Negativ Negativ Neutral Positive Positive Count Total Percenta Based on the review analysis presented in the table, the Livin Mandiri application shows that the majority of reviews are negative, particularly for ratings 1 and 2.
Rating 1 accounts for 4,074 reviews with a negative sentiment of 29.
8%, while rating 2 has 2,493 negative reviews with a percentage of Rating 3 is generally neutral, with 2,195 reviews or 16.
Conversely, positive reviews are https://doi.
org/10.
35145/joisie.
JOISIE licensed under a Creative Commons Attribution-ShareAlike 4.
0 International License (CC BY-SA 4.
Safarah.
Analysing Service Quality Using Sentiment Analysis And Topic Modeling: A Case Study Of The Livin Mandiri Application, 209-.
217 more common in ratings 4 and 5, with rating 4 having 2,119 positive reviews at 15.
5% and rating 5 having 2,811 positive reviews with a percentage of 20.
This indicates that while there are some satisfied users, the majority of reviews suggest dissatisfaction with the Livin Mandiri application.
The highest percentage of negative sentiment, at 29.
8%, highlights several key issues, including "transaction failures," "difficulties in the verification process," and "missing OTP codes".
Issues such as transaction failures, difficulties in the verification process, and missing OTP codes on the Livin Mandiri application can severely impact user trust and the bank reputation.
Transaction failures cause frustration, especially in urgent situations, and diminish the perceived reliability of the bank digital Recurring verification difficulties further degrade the user experience, increasing the risk of losing customer loyalty.
Disappointed users tend to leave negative reviews and may switch to more reliable platforms, hindering the growth of new user acquisition.
Moreover, these technical issues threaten to undermine Bank Mandiri reputation as a trusted provider of digital services, ultimately damaging the bank expansion efforts and digital performance in this competitive era.
5 HEAT MAP
The heatmap consists of two main components: the dendrogram and the heatmap itself.
The dendrogram illustrates the cluster relationships between topics grouped by content similarity, while the heatmap displays the frequency or intensity of each topic's occurrence.
Bright colors indicate high intensity, whereas dark colors denote low intensity.
Each topic cluster has its own interpretation.
Cluster 1 (Topics 12, 9, 8, .
is likely associated with general technical issues, such as verification or login problems.
Cluster 2 (Topics 13, 10, .
appears to be related to the use of specific features within the application, such as QRIS or mobile banking.
Cluster 3 (Topics 11, 4, .
is probably connected to overall user experience issues, including complaints about application updates or problems with specific features.
Figure 8.
Heat Map Results 6 DISTANCE MAP Distance map, also known as a distance matrix, displays the distance matrix among the fifteen subjects found in a dataset.
The colors on the map indicate how close the topics are to each other.
Dark blue signifies very small or zero distance, meaning the topics are very similar or identical.
represents a greater distance, while orange to red indicates a medium to large distance.
Some topics exhibit related content or themes, as evidenced by the clusters formed.
On the other hand, the heatmap provides a visualization that facilitates understanding the differences between topics in the dataset.
Overall, the distance map offers an overview of the complex relationships between the analyzed https://doi.
org/10.
35145/joisie.
JOISIE licensed under a Creative Commons Attribution-ShareAlike 4.
0 International License (CC BY-SA 4.
218 | Jurnal JOISIE.
Volume 8.
Nomor 2.
Desember 2024 Figure 9.
Distance Map Results 7 WORD CLOUD A word cloud is a text visualization that highlights the most frequently occurring words.
In the reviews of the Livin Mandiri application on Google Play Store, the word cloud reveals the frequency of specific words, providing insights into the most commonly discussed topics.
For example, the word "aplikasi" indicates that many users are discussing the application in general, "good" reflects positive feedback, "Livin" focuses on this particular application, and "update" highlights users' attention to The word "Mandiri" relates to the development bank, "kartu" may refer to issues or features related to cards, "transfer" denotes discussions about the money transfer feature, and "error" signifies technical problems experienced by users.
The word cloud allows developers to quickly identify areas that need improvement and features that are most valued, facilitating decision-making.
Figure 10.
Word Cloud Results CONCLUSION This study concludes that although the Livin Mandiri application has a rating of 4.
0 and offers significant benefits in accessing banking services, the majority of user reviews on the Google Play Store indicate a negative sentiment.
The primary complaints involve technical issues such as transaction failures, verification difficulties, and application errors, which have led to a negative sentiment of 48%.
Despite some appreciation for certain features of the application, positive sentiment only reaches 36%, reflecting a significant level of dissatisfaction among users.
Topic modeling using Latent Semantic Indexing and data visualizations such as heatmaps, distance maps, and word clouds identify that the main complaints focus on application stability and customer support.
To improve user satisfaction, significant improvements in application stability, security, and customer service are necessary to address various complaints and maximize the benefits of this application.
This discussion highlights three main negative topics related to the application: stability, security, and customer Issues related to application instability, such as frequent crashes and delays, significantly https://doi.
org/10.
35145/joisie.
JOISIE licensed under a Creative Commons Attribution-ShareAlike 4.
0 International License (CC BY-SA 4.
Safarah.
Analysing Service Quality Using Sentiment Analysis And Topic Modeling: A Case Study Of The Livin Mandiri Application, 209-.
219 impact user experience by causing frustration.
Furthermore, concerns regarding security, particularly the protection of personal data, are considered suboptimal and can diminish user trust in the This dissatisfaction may lead users to hesitate in utilizing digital banking services.
Additionally, slow customer service responses to complaints exacerbate the situation, as users may feel neglected and may leave negative assessments of both the application and the bank.
The combination of these three issues not only affects user satisfaction but also has the potential to hinder the growth of the user base and damage Bank Mandiri reputation within the digital banking industry.
Implementing these recommendations will likely improve user trust and satisfaction, potentially leading to better overall ratings and user retention for the application.
5 REFERENCES