Aksaqila International Humanities and Social Sciences Journal [AIHSS] Vol 3 Nomor 2 Agustus 2024, hal: 76-82
ISSN: 2964-8831
TFOOD (TensorFlow Object Detectio.
Application Design Smart Solution to Create Digital Communication Tools for People with Deaf and Speech Disabilities Nur AoAfifah1.
Sella Gustrinita2.
Sylvi Agustin3.
Yudi FirmansyahA.
Ukhairi AlpatihAA 1,2,3,4,5Universitas Muhammadiyah Sumatera Utara.
Medan.
Indonesia 1nurafifah@umsu.
id, 2sellagustrinita13@gmail.
com, 3shilvyagustin08@gmail.
4yudifirmansyahksp008@gmail.
com, 5ukhairialfath27@gmail.
Abstract Advances in technology have paved the way for innovative solutions aimed at improving the well-being of the lives of deaf and speech-impaired individuals.
This paper presents the design of TFOOD (TensorFlow Object Detectio.
, a smart application designed to facilitate digital communication for deaf and speech impaired people.
Utilizing TensorFlow object detection technology.
TFOOD translates visual cues, such as SIBI and BISINDO sign language cues, into text or audio output that is easily understood by others.
Prototype of AuTFOODAy application which is designed to help reduce the difficulties faced by people with hearing and speech These applications involve training sophisticated models on diverse data sets and optimizing algorithms to ensure high accuracy and responsiveness in real-world Results show that TFOOD significantly increases communication accessibility for deaf and speech-impaired individuals, providing an effective tool for interacting in a variety of social and professional contexts.
This paper also discusses the challenges faced during implementation, including model accuracy and integration, and discusses potential future developments to further improve the system's capabilities and accessibility.
Through this exploration.
TFOOD demonstrates its contribution to digital inclusion and offers insight into the development of supporting communications technologies.
Keywords: TensorFlow.
Object detection.
Digital communication.
Hearing impaired.
Speech disabled This work is licensed under a Creative Commons Attribution-ShareAlike 4.
0 International License.
INTRODUCTION
Humans are social creatures, which means that all humans can communicate, interact and socialize with other humans.
Language is a way for humans to communicate with one another.
However, normal people often communicate using language verbally.
However, not all humans can use language verbally.
For example, a person with disabilities, namely deaf and speech impaired.
People with disabilities can only communicate using sign language.
Language as a communication tool is extremely important to human life.
Understanding another language is very important for humans, especially when interacting with other people.
In the news that is spread throughout Indonesia, deaf and speech impaired people have limitations, namely their hearing.
Successful communication requires effort from everyone involved in the conversation.
In this case, human senses play an important role in the Jurnal Homepage: https://aksaqilajurnal.
com/index.
php/aksaqila Aksaqila International Humanities and Social Sciences Journal [AIHSS] Vol 3 Nomor 2 Agustus 2024, hal: 76-82
ISSN: 2964-8831
information exchange process.
In the Technology 4.
0 era, machine learning has become one of the most popular methods.
Because technology has made AI a significant communication tool for society.
According to the National Statistics Agency, in 2019, there were 1,820,000 deaf people among Indonesia's 268,100,000 population.
Compared to other children with special needs, physically, deaf children are no different from normal children in general.
Table 1.
Distribution of deaf people in Indonesia .
Category Deaf Population Non-Deaf Population Number of People 1,820,000 266,280,000 From the table it is obtained that:
The percentage of deaf people is 0.
The percentage of the non-deaf population is 99.
Deaf people have obstacles in hearing, because they have these obstacles, deaf people also have obstacles in speaking so they are usually called speech impaired.
Body gestures like hand and face gestures are used in sign language.
This motion can convey a word or Every nation has a unique sign language.
Two sign languages are used in Indonesia:
Indonesian Sign Language System (SIBI) and Indonesian Sign Language (BISINDO).
A sign language called BISINDO emerges and changes in a disability-related setting.
Additionally.
SIBI, or Indonesian sign language, is a standard and consistent sign language utilized nationally and typically in formal gatherings like official activities and school functions.
There is a problem with this dialectical approach, namely the different levels of people's understanding of the signals used by each group.
This situation causes deaf people, who are generally quite unpleasant in conversations with the public.
In study of Luh Putu Ary Sri Tjahyanti.
Gede Danu Setiawan ad previously designed an application system for learning sign language letters and numbers for deaf people based on a However, currently there are still many people who do not understand the sign language of those they speak to, which causes limited communication between normal people and deaf and speech impaired people.
So, from these problems in this research a prototype AuTFOOD (TensorFlow Object Detectio.
Ay will be designed in the form of an application using computer vision and the TensorFlow library and using the Python programming language which can be used on smartphones.
With the design of this application model, it is hoped that it can facilitate sign language communication between normal people and people with hearing and speech impairments.
TFOOD is a system model design in the form of an application that can translate sign language into letters of the alphabet where the system can be used via smartphone, as well as a communication tool to be able to communicate with deaf and speech impaired people using the CNN method.
CNN is a combination of artificial neural networks and deep learning A CNN consists of one or more convolutional layers, often with a subsampling layer followed by one or more fully connected layers as standard neural networks.
Jurnal Homepage: https://aksaqilajurnal.
com/index.
php/aksaqila Aksaqila International Humanities and Social Sciences Journal [AIHSS] Vol 3 Nomor 2 Agustus 2024, hal: 76-82
ISSN: 2964-8831
METHODOLOGY
This research method has several stages, starting with problem formulation, literature study, taking sample data in the form of photos or sign language videos directly from deaf people, data processing, and testing.
In creating this prototype system, so that the system can run optimally, the programming language used is Python with several main libraries, namely.
Tensorflow.
Numpy.
CV2.
MediaPipe and using the Flutter framework.
In developing this prototype system, the sign language datasets used were SIBI and BISINDO.
and for one letter requires up to 60 shots to get a high level of accuracy.
The development of this system prototype also uses Android Studio, which is an Integrated Development Environment (IDE) used to develop applications on Android.
A description of the hardware and software used to create the sign language detection application is as follows:
Table 2.
Specifications hardware Hardware specifications Laptop/PC LENOVO IDEAPAD slim 1
processor laptop
AMD Ryzen 5 7520U with Radeon Graphics 80 GHz
Laptop RAM
8 GB
Table 3.
Specifications software Software specifications Laptop operating system
Windows 11
IDE Tool
Androin Studio
programming language Phyton
RESULT
TFOOD Prototype Design Development The development of this prototype design has several features that can help users to communicate between the deaf and speech impaired and normal people.
For example, there is a dictionary feature where several words will be translated into sign language automatically using AI, but this feature requires a further stage of development.
As well as the main feature, namely the camera which will be used to carry out the process of transferring gesture movements into Indonesian text.
Results Accuracy Level Jurnal Homepage: https://aksaqilajurnal.
com/index.
php/aksaqila Aksaqila International Humanities and Social Sciences Journal [AIHSS] Vol 3 Nomor 2 Agustus 2024, hal: 76-82
ISSN: 2964-8831
In the process of making this prototype, two tests were carried out, namely the SIBI and BISINDO languages, where each test consisted of 5 tests.
This trial was divided into two parts, namely trials to detect words and trials to detect alphabet letters.
Table 5.
BISINDO Word Test Results Table 4.
Sibi Word Test Results No Test Data Terima kasih Tolong Maaf Permisi Makan Minum Bermain Halo Dengar 10 Bagus 11 Sedih 12 Senang Total accuracy Results Correct Correct False Correct Correct Correct Correct Correct Correct Correct Correct Correct 11 Correct No Test Data Terima kasih Tolong Maaf Permisi Makan Minum Bermain Halo Dengar Bagus Sedih Senang Total accuracy Results Correct Correct False Correct Correct Correct False Correct Correct Correct Correct Correct 10 Correct Based on the results of the SIBI word test, there are 12 test data in table 3, it was found that 11 words could be translated correctly and 1 word could not be translated.
Thus, the system accuracy in this test is 12 y 100% = 91.
Furthermore, in table 4, it shows that there are 2 words that cannot be translated, with an accuracy level of 12 y 100% = 83.
Tabel 6.
Letter Testing Results SIBI Alphabet Data Uji Hasil Correct Correct Correct Correct Correct Correct Correct Correct Correct False Correct Correct Correct False Correct Correct Correct Correct Correct Table 7.
Letter Testing Results BISINDO Alphabet Jurnal Homepage: https://aksaqilajurnal.
com/index.
php/aksaqila Data Uji
Hasil
Correct
Correct
Correct
Correct
Correct
Correct
Correct
Correct
Correct
False Correct
Correct
Correct
False Correct
Correct
False Correct
Correct
Aksaqila International Humanities and Social Sciences Journal [AIHSS] Vol 3 Nomor 2 Agustus 2024, hal: 76-82
ISSN: 2964-8831
Total accuracy Correct Correct Correct Correct Correct Correct Correct 24 Correct Total accuracy Correct Correct Correct Correct Correct False Correct 22 Correct Based on the results of the SIBI letter test, there are 26 test data in table 5, it was found that 24 words could be translated correctly and 2 words could not be translated.
Thus, the system accuracy in this test is 26 y 100% = 92.
Furthermore, in table 6, it shows that there are 4 BISINDO letters that cannot be translated, with an accuracy level of 26 y 100% = 84.
From the results of this trial it can be concluded that the system has a good level of accuracy in recognizing the word SIBI .
6%), recognizing the word BISINDO .
3%), recognizing SIBI alphabet letters .
3%), and recognizing alphabet letters .
6%).
DISCUSSION
Looking at current developments, technological advances are growing rapidly.
technology brings many conveniences to people's daily lives.
The use of technology can make understanding sign language easier for the deaf and speech impaired and normal people.
The technology used is an Android-based application that can convert sign language into objects to be scanned.
The method used in Object Detection is the Haar Cascade Classifier.
In this case, the aim is to design a system that can detect hand shapes through images captured by an IP camera.
Hand shape recognition uses feature extraction methods, namely Tensorflow Object Detection.
Vector Analysis and Pixel to Pixel Distance Analysis.
The name of the hand shape will be classified using the k-NN algorithm.
Apart from that, communication conditions for deaf-speech disabled people in the general or normal community still experience difficulties because people do not know sign language so it is difficult to start communicating with deafspeech disabled people.
So this can cause miscommunication in normal society with people with hearing and speech impairments.
Design And Manufacture of Tools Literature Study In the initial process of designing and making this tool, the team conducted literature studies in several reading materials such as: journals, books, articles and news regarding communication limitations between normal people and people with hearing and speech From the literature study that we obtained, there are limitations in communication between normal people and people with hearing impairments, therefore the Jurnal Homepage: https://aksaqilajurnal.
com/index.
php/aksaqila Aksaqila International Humanities and Social Sciences Journal [AIHSS] Vol 3 Nomor 2 Agustus 2024, hal: 76-82
ISSN: 2964-8831
team conducted further research regarding the development of technology that can facilitate communication between normal people and people with disabilities.
Team discussion After conducting a literature study, the team held discussions with accompanying lecturers regarding the stages of work and designing the prototype system, prototype design, features contained in the prototype, making progress reports, filling in daily logbooks and financial logbooks.
Designing a prototype system design At this stage, the team used the Adobe Photoshop application to design the prototype and used Android Studio to input some text using Text View, to input images using Image View, and to input information into the prototype system using List View by adding some code to the file.
activity_main.
xml, stirng.
xml, and mainactivity.
Where.
Android Studio is used as a bridge for programming languages using Flutter.
The programming language we use is Python using Flutter as a framework to be connected to API integration.
Taking and collecting sign language samples In the stage of taking and collecting SIBI sign language samples, the team took videos or portraits directly from people who had a background of people with hearing and speech Where, for one SIBI and BISINDO language alphabet letter it requires 60 shots to get high accuracy results.
So the prototype system can translate gestures into Indonesian that can be understood by normal people.
The process of inputting the data into the system data collection tools using python.
There are 3 main libraries, namely:
Mediapipe C Tensorflow Opencv Opencv, to take pictures/videos.
Mediapipe, to recognize hand gestures.
Tensorflow, this is the main part for storing data and introducing machine learning methods.
For the cellphone, it's just a display, when doing the introduction, the Android application only makes a call/sends a video to the server (Pytho.
, then Python sends it back and displays the translation results in Flutter/Android application.
Following is the work flow:
User selects the video source, from gallery or camera.
For the gallery he used a flutter package called file_picker.
For cameras, the package name is camera.
After the video has been selected/recorded.
Flutter will make an api call, namely sending a video file to the server for processing.
Jurnal Homepage: https://aksaqilajurnal.
com/index.
php/aksaqila Aksaqila International Humanities and Social Sciences Journal [AIHSS] Vol 3 Nomor 2 Agustus 2024, hal: 76-82
ISSN: 2964-8831
Once finished, the server will send back the translation results in the form of text and video files.
The server reply is displayed with a flutter widget called Text and a package called video_player to display the video.
Testing And Analysis Trials were carried out on several students at one of the Medan special schools and trials were also carried out on several people with hearing and speech impairments.
Evaluation How well the system can recognize sign language movements inputted by the user, measuring the response time of the prototype system from movement detection to displaying translation results, checking how accurate the translation is from sign language movements to Indonesian text, trials were carried out on several students at one of the Medan SLBs and trials were also carried out on several communities with hearing and speech impairments.
CONCLUSION
The benefits that can be obtained from this Student Creativity Program activity are:
Enabling deaf and speech impaired people who use sign language to communicate with normal people more easily and effectively in everyday life, whether at home, at school, at work, or in general society.
Help expand the social circle and increase interaction between the deaf-speech impaired and the normal community around them.
Potential for use in the health, public services, banking, tourism and other industries to improve services for the deaf-speech impaired.
The next stage of the plan is to carry out an in-depth evaluation of the prototype to identify deficiencies and add necessary features.
Level gesture recognition algorithm to improve the accuracy and response of the prototype to different sign languages, carry out extensive testing to ensure the stability, security and performance of the prototype in various user situations so as not to cause debugging.
And the next development is the process of refining the prototype into a communication tool in the form of an application that is suitable for dissemination to normal society, and then developing this prototype so that it can be used on smartwatches.
REFERENCES