Advance Sustainable Science.
Engineering and Technology (ASSET) Vol.
No.
July 2023, pp.
0230201-01 0230201-010
ISSN: 2715-4211 DOI: https://doi.
org/10.
26877/asset.
Fast and Accurate Indonesian QnA Chatbot Using Bag-of-Words and Deep-Learning for Car Repair Shop Customer Service Muchamad Taufiq Anwar1*.
Azzahra Nurwanda1.
Fajar Rahmat1.
Muhammad Aufal1.
Hindriyanto Dwi Purnomo2.
Aji Supriyanto3 Automotive Industry Information System.
Politeknik STMI Jakarta.
Jl.
Letjen Suprapto No.
Central Jakarta 10510.
Jakarta.
Indonesia Faculty of Information Technology.
Universitas Kristen Satya Wacana.
Jl.
Dr.
Notohamidjodjo.
Salatiga 50715.
Central Java.
Indonesia Faculty of Information Technology.
Universitas Stikubank.
Jl.
Tri Lomba Juang No 1 Semarang 50241.
Central Java.
Indonesia *taufiq@stmi.
Abstract.
A chatbot is a software that simulates human conversation through a text chat.
Chatbot is a complex task and recent approaches to Indonesian chatbot have low accuracy and are slow because it needs high resources.
Chatbots are expected to be fast and accurate especially in business settings so that they can increase customer satisfaction.
However, the currently available approach for Indonesian chatbots only has low to medium accuracy and high response This research aims to build a fast and accurate chatbot by using Bag-of-Words and DeepLearning approach applied to a car repair shop customer service.
Sixteen different intents with a set of their possible queries were used as the training dataset.
The approach for this chatbot is by using a text classification task where intents will be the target classes and the queries are the text to classify.
The chatbot response then is based on the recognized intent.
The deep learning model for the text classification was built by using Keras and the chatbot application was built using the Flask framework in Python.
Results showed that the model is capable of giving 100% accuracy in predicting usersAo intents so that the chatbot can give the appropriate responses and the response time is near zero milliseconds.
This result implies that developers who aim to build fast and accurate chatbot software can use the combination of bag-of-words and deep-learning Several suggestions are presented to increase the probability of the chatbotAos success when released to the general public.
Keywords: Indonesian chatbot, car repair shop customer service, fast, accurate, deep learning, bag of words (Received 2023-03-06.
Accepted 2023-04-30.
Available Online by 2023-07-.
Introduction A chatbot is a software that simulates human conversation through a text chat.
It is one of the tasks in Natural Language Processing (NLP) in the field of Artificial Intelligence.
Chatbots are expected to be fast and accurate .
and chatbot performance is important to increase customer experience and therefore improve customer satisfaction .
However, the currently available approach for Indonesian chatbots only has low to medium accuracy and high response time.
The accuracy of the chatbot is a measure of how likely the chatbot is to give the correct .
answer to the question asked by the user .
Whereas response time is defined as the time required by the chatbot to give a response based on the userAos query .
Research .
has an accuracy of 26-78% using pre-built platforms namely Artificial Intelligence Markup Language (AIML) and Pandorabots.
Another research has an accuracy of up to 75% .
and another research using RNN for intent classification has an accuracy of 81% .
Another research using the GloVe model and CNN produces the best accuracy of 95.
84% .
Only one research reports the chatbotAos response time and it has a low response time with an average response time of 3.
Chatbot in Bahasa Indonesia had been used for FAQ in a university library during a pandemic situation .
, virtual assistant for banking service .
, .
, hotel .
, .
, software requirements elicitation .
, legal information service .
, digital historian .
, maternal and child health .
, and e-commerce .
The approach to building a chatbot varies from traditional methods such as pattern matching .
and cosine similarity .
, to the more modern, neural-network-based approaches such as sequence-to-sequence model .
and Transformers.
However, it has been proved that the development of a pattern-matching or rule-based approach is very difficult .
The later approaches will also allow the users to use free text as their queries, rather than a fixed pattern .
Some authors may use the pre-build platform such as RASA .
Dialogflow .
, and AIML .
, .
Ae.
This research aims to build a deep-learning-based chatbot from scratch using TensorFlow/Keras.
A deep neural network is a type of neural network that has more than one hidden layer.
The higher complexity of the deep neural network will allow the model to do a more complex task including language modeling and Question-and-Answering task such as chatbot.
For a computer to be able to process text, the text needs to be pre-processed and undergo feature extraction steps before being fed into the model.
The text pre-processing tasks include tokenization, lemmatization, stemming, stop-word removal, etc.
There are some text feature extraction techniques such as (Term FrequencyAeInverse Document Frequenc.
TF-IDF .
Term Document Matrix (TDM) .
Bag of Words (BoW) .
Word Embeddings .
, etc.
Research showed that when using BoW, the text conversion of uppercase letters into lowercase letters gave a significant improvement in text classification accuracy .
Whereas stop word removal may improve the accuracy but the result is dependent on the dataset used .
The result from text feature extraction is a matrix that later is used for text classification.
The text classification part may use some AutraditionalAy Data Mining techniques such as Random Forest and Gradient Boosting .
or a more modern, neural-network-based approach, such as Transformers.
The text classification task may use TransformersAo zero-shot learning approach but it is slow / resource-intensive .
Customer Service (CS) is one of the important parts of every business, including a car repair shop.
Traditionally.
CS tasks are held by human CS.
However, when so many customers only ask about basic information, this can burden the utilization of human CS where the precious resource of human CS are wasted on answering very basic questions from the customers.
The use of a chatbot can help CS .
to alleviate this burden and reduce human resource costs for CS .
Furthermore, the use of chatbots can also avoid human error .
and provide the latest information about the business .
Regardless of being a non-human CS, chatbots are proven to increase business agility .
, form positive customer relationships .
, and increase customer satisfaction .
, .
, .
This research will apply the chatbot to help a car repair shop CS.
If the customers need further questions, the chatbot will direct the customer to a human CS.
Methods The research methods are shown in Figure 1 which shows that the methods are separated into two phases: the training phase and the implementation phase.
In the training phase, a set of preset queries and their corresponding intent is prepared as the training dataset in the JSON file format.
The queries consist of questions such as AuWhat is the opening hour?Ay and AuHow long will the repair take time?Ay.
There are a total of 16 intents with each intent having several possible queries.
There is one special intent to handle the case when the user inputs an invalid or blank query or queries that the chatbot can not understand.
This special intent will be responded to by the chatbot asking the user to type a valid / better query.
The approach for this chatbot is by using a text classification task where intents will be the target classes and the queries are the text to classify.
The chatbot response then is based on the recognized intent.
An example of the training data is shown in Figure 2.
The preset queries will undergo text preprocessing and feature extraction steps which includes tokenization, lemmatization / stemming, and bag-of-words creation.
Tokenization will list all words in the Lemmatization and stemming will change the words into their root form.
Bag of Words will create a corpus that lists all unique words and count the word frequency for each word.
These steps will result in a matrix that is ready to be used for model training.
The model consists of one input layer, two hidden layers, and one output layer.
The details of the model architecture are shown in Table 2.
The model is trained using SGD optimizer with Learning Rate = 0.
The trained model will be saved as an H5 file to be further used in the chatbot application development.
In the deployment phase, the user will input queries in the app which will then be replied to by the chatbot.
The query will undergo the same text preprocessing and feature extraction as in the training phase before being fed into the model.
The model will then output the predicted intent and the app will give the appropriate response based on the predicted intent.
The confidence threshold for the intent prediction is set to 0.
25 and the model will give a response based on the class with the highest confidence score.
If there is more than one preset response, the app will give a random preset response for that intent.
In case of a low confidence prediction score, the app will respond with a statement that the chatbot does not understand the user query.
And for special or further queries, the chatbot will direct the customer to human customer service.
Both the training and deployment phases are done in the Python environment by using several libraries such as NLTK .
for tokenization and lemmatization.
Sastrawi for Indonesian stopword removal and stemming.
Tensorflow .
and Keras .
for the deep learning model.
Pickle for saving the model, and Flask .
as the framework to develop the web-based chatbot app.
Figure 1.
Research Methods Figure 2.
A sample of intent, its preset queries, and its preset response in the JSON format Table 1.
Model architecture Layer Input layer Hidden layer 1 Hidden layer 2 Output layer Details Input shape equals the number of BoW elements 128 neurons, dropout rate = 0.
ReLu AF 64 neurons, dropout rate = 0.
ReLu AF Neuron units = number of intents.
Softmax AF Results and Discussion By using 200 epochs, the training phase resulted in a model with 100% accuracy as shown in Figure 3.
The training time was two minutes.
The trained model is saved and then deployed in a web-based Python The result of the app is shown in Figure 4 and Figure 5.
User will input their queries in the text field and hit the AuSendAy button and the chatbot will respond to it.
For example, when the user asked about facilities, the chatbot will respond with information regarding the available facilities in the car repair shop.
The timestamp of each message is displayed in each of the bubble chats.
Figure 5 shows that when users' queries contain abbreviated words and/or typos .
riting error.
, the chatbot can still understand them and give the appropriate response .
s long as another/other important keywords are When the words in the query are all abbreviated and/or typos, the chatbot will return a response saying that it can not understand the query, as shown in Figure 6.
Figure 7 shows the Python console showing the difference between the userAos query time and chatbot response time.
The difference is near zero milliseconds.
This shows that our results is better in both accuracy and speed than previous research mentioned in the Introduction section.
The use of this Chatbot will certainly help the car repair shop CS to serve their customers who want to ask questions about their services.
CS will only focus on serving customers who need further information about its services.
Figure 3.
The trained model has 100% accuracy Figure 4.
The result of the chatbot app Figure 5.
A screenshot of the chatbot app showing its capability to respond to incomplete/short Figure 6.
A screenshot of the chatbot app showing a respond to a sentence containing all abbreviated words and typos.
The chatbot will respond by asking the user to input a more detailed / better-typed query Despite its great accuracy and speed, when released to the public, research showed that a successful chatbot needs to be useful and easy to use .
, .
Research also showed that user satisfaction in using a chatbot is influenced by System Quality.
Information Quality, and Service Quality .
Whereas the intention to use a chatbot is influenced by Information Quality and Personal factors .
uch as age and occupatio.
Further research may address and implement those insights to guarantee a successful chatbot implementation that is well-received by the public.
Further research may also use a more sophisticated out-of-scope intent detection .
rather than by providing one extra intent class as used in this research.
Latent Dirichlet Allocation (LDA), a text clustering technique .
, might be explored to be used as an approach to / to assist the text classification task .
, .
Future research may also add Typographical Error correction to make the chatbot more robust in handling Typographical Errors using a method such as Schema Matching Technique .
Future research may add more datasets for both preset queries and their corresponding intents to further expand the functionality of the chatbot.
The chatbot can be further improved by integrating the chatbot into the car repair shop information system so that it can handle service orders, provide real-time information about the current service queue, provide real-time information on the estimated time to complete service, etc.
But such system may require a referee policy so that it will not be abused by irresponsible users.
Figure 7.
The Python console showing the difference between the userAos query time and chatbot response time.
The difference is near zero milliseconds Conclusion This research aims to build a fast and accurate Indonesian QnA chatbot using bag-of-words and deeplearning approach applied to a car repair shop.
The training phase resulted in a 100% accurate model.
Results showed that the chatbot app can provide a fast .
ear-zero milliseconds response tim.
and highly accurate response based on usersAo queries.
This chatbot can help to reduce the cost use of hiring human CS and reduce human errors.
Despite from its already great performance, there are several suggestions to increase its chance of success when released to the general public.
References