Indonesian Sign Language (SIBI) Learning Media Application Based on Deep Learning Technology for Deaf Children

: One of the main methods in teaching deaf children effectively is through visualization of learning materials. Previous research has developed augmented reality-based interactive learning media for deaf children. However, these learning media have limitations related to smartphone memory, dependence on certain flashcards, and limited amount of material due to flashcard limitations. Here, researchers innovate to develop SIBI learning media applications based on deep learning technology that material does not depend on a specific flashcard, but the application is able to detect real objects and various images. TensorFlow Lite is considered lighter but still allows the system to run deep learning on mobile devices with low latency. This application is designed to capture a real object or image through the camera, then the object or image will be processed and recognized using TensorFlow Lite. The object recognition process is carried out at the classification layer using switch-case syntax and global class synchronization with the YouTube API. After the image or object is recognized, the system will issue an output in the form of an Indonesian sign language video link from the object.


INTRODUCTION
Indonesian survey data [1] shows that 7,87% of people suffer from hearing disorder.While 2,74% of people have speech and hearing disorders.As many as 1,307,094 hearing disorder people live in the province of East Java.Based on-field study at TKLB Karya Mulia Surabaya, learning media used for the teaching and learning process to hearing-impaired children using original objects, imitation objects, and images through flashcards.Even though [2], shows one of the key methods of teaching hearing-impaired children is through visualization of learning materials.Then the presentation of learning materials using technology and ICT is vital to support the teaching and learning process both during class activity and in the self-study process.Especially during the pandemic covid-19, direct meeting class activities are limited, so students are required to study independently.
The limited sense of hearing encourages hearing-impaired children to adapt by using other ways to communicate, such as using sign language.Which one of the National Sign Language standards in Indonesia is using the Indonesian Sign Language or SIBI [2].The process of understanding sign language and mastering vocabulary for children with hearing impairment is the earlier the better.In this case, the role of parents is very influential for hearing-impaired children.Therefore, appropriate learning media are needed to help teachers, parents, and children with hearing-impaired to improve their communication learning skills from early stage.
Hearing-impaired children would learn more effectively through a learning system that is supported by as many visual displays as possible.The use of technology makes a major contribution to improving the quality of education in deaf schools [6]- [8].The importance of technology in learning media for individuals with hearing loss was also investigated [9].The results show that it is highly recommended that a mobile application based on the Android and iOS operating systems be developed as a learning medium for hearing-impaired children.
Quite a number of learning media technologies for deaf children have been developed, such as [10] who developed AR as a learning medium for deaf children in Arabic sign language.Meanwhile, researchers here have also developed AR as a learning medium for deaf children in Indonesian sign language (SIBI) [4] and [5].The digital learning media is used by teachers and students in TKLB Karya Mulia Surabaya until now.However, these learning media have limitations tied to smartphone memory, dependence on certain flashcards, and the limited amount of material due to flashcard limitations.
Here researchers are innovating to develop SIBI learning media application based on deep learning technology for hearing-impaired children.The materials don't depend on certain flashcards, but the application would detect various real objects and images.That way, learning materials would be sustainably added and developed.We hope to give benefit and contribute for hearing-impaired children through the development of this application.

METHOD
In this study, learning media was developed based on deep learning so that learning materials were not limited and depended on the availability of flashcards.Deep learning systems teach computers to do the work humans are supposed to do, just as computers can learn from the training process.Deep learning uses a problem-solving approach in a computer learning system that uses the concept of a hierarchy.The concept of hierarchy allows computers to learn complex concepts by combining simpler concepts.The architecture used consists of a visible layer and a hidden layer where the load of each perceptron unit is optimized using the backpropagation algorithm.In this research, we use TensorFlow Lite as a solution for best implementing deep learning to mobile devices.TensorFlow Lite is considered lighter but still allows the system to run machine learned models on mobile devices with low latency.Figure 1 is an overview of the design of the learning media application system that was built in this study.The application system requires a user interface design so that users are interested and easy to use.This application is designed to capture an object or image through the camera, then the object or image will be processed and recognized using TensorFlow Lite.This object recognition has been assigned to the classification layer.After the image or object is recognized, the system will issue an output in the form of a sign language video link from the object.All sign language videos are stored on YouTube cloud storage.

Figure 1. Application System Design
In the development of this application, we have set the appropriate color and text type for children aged 4-6 years which is the age range of deaf kindergarten children in Indonesia.The mockup design for the application consists of five parts, as follows: splash screen, home screen, instructions screen, information screen, and camera screen as shown in Table 1.

Splash Screen
The initial appearance when the android application is opened/run for the first time, its function is to make the application more attractive and professional.

Home Screen
The ADIP (Deep Learning Application) logo will appear on the home screen.There are five buttons that appear as vocabulary, wording, instructions, information and exit buttons, where each button is connected to another screen.

Instruction Screen
This screen displays instructions for using the application.This screen is equipped with an exit button to return to home screen.

Information Screen
This screen displays information regarding the application.This screen is equipped with an exit button to return to home screen.

Camera Screen
On this screen, users can scan images or objects to be studied sign language.The scanned image or object will be processed by the system.The system will provide output in the form of a link that leads to a sign language demonstration video of the image or object.
The working system of this application is presented in the flowchart in Figure 2. First, the user is asked to scan an image or object that he wants to recognize, using a smartphone camera; then the system will process the image or object via TensorFlow Lite; the system will provide output in the form of a sign language demonstration video link from the image or object so that users could watch it.

Figure 2. Flowchart System
In the classifier section, there is a recognition section which object classification using switch-case syntax, and global class synchronization with the YouTube API.Here we don't carry out the manual recognition of the asset input stage but uses existing assets from TensorFlow lite.The type of model used is mobilenet_quant_v1_224.tflite, with the number of label names originating from the TensorFlow Lite is 1001 words.However, with that many labels, it is not necessarily used entirely, since the output of this application is a video display of SIBI taken from previous research.The application would distinguish which one should be used or not.The application requires a class to classify objects then given the Switch and Case statements.The switch-case statement will execute the program if the value in the switch statement is the same as the value in the case statement.It makes easier for the system to display a video link to display the SIBI display results from recognizing objects and captured images.There are 14 Indonesian sign language demonstration videos that have been synchronized with Tensorflow Lite in this study.The addition of recognizable objects is very possible, by adding the number of Indonesian sign language demonstration videos in the next development as shown in Table 2.
There are several daily objects that are selected as SIBI objects.The selection of these objects refers to the TKLB Karya Mulia curriculum and is based on the results of consultations with teachers.These objects are included in the application program.In practice, if the application scans one of the objects, the application would display the SIBI video for learning Indonesia Sign vocabulary for deaf children.

RESULT AND DISCUSSION
There are four types of application testing that have been carried out in this study.They are functionality testing, objects testing, device testing, and user testing.The application testing process could be seen in Figure 3.The test results are discussed in the following subsections:

Functionality Testing
Functionality testing aims to determine the functionality of each button on each application screen.There are seven types of buttons tested from six screens as follows: • Scan button The main device that supports this test is an Android smartphone that has specifications as shown in Table 3.The functionality testing result can be seen in Table 4.The test results show that each button on every screen is running according to its function properly.

Objects Testing
We tested the performance of the application with 14 different types of objects in-order-to determine the success rate of the application in recognizing objects and displaying Indonesian sign language demonstration videos correctly.The test results show the application could recognize all objects and display videos correctly as shown in Table 5.

Devices Testing
Device testing aims to find out the recommended device specifications for this application, including the minimum device specifications required for the application to run properly.There are five types of devices tested in this study; they have specifications as listed in Table 6.
The device test results are listed in table VII, showing that device specifications also affect application performance.Device numbers 1,2,3, and 4 require an average processing time of less than 10 seconds.We recommend this app run on android device version 8 or above.Referring to the device testing that has been carried out on Android version 5, the application failed to recognize objects so that it could not display videos on the YouTube media player.The application also has an error when trying to exit the application as seen in Table VII

User Testing
User testing in this study used in-depth interviews method with users.We tested two users, each of whom had a different scientific background.The first is a teacher of deaf children at the Karya Mulia TKLB school in Surabaya, the second is a software developer.Users are asked to try using the application and all its features.Then we interviewed them to explore their experience of using the application and asked for their opinion according to their scientific field.User test results are summarized in table VIII, in which each aspect is rated using the Likert's scale (5 = excellent, 4 = good, 3 = fair, 2 = poor, and 1 = very poor, need improvement).From the results of the expert assessment, it would be concluded that the application can run well, all buttons and features function properly.Users also feel that the selection of color and text themes is appropriate for children aged 4-6 years who are the targeted of the application.The 1 st user is welcoming the presence of this learning media application for deaf children, she requests this application be developed by adding the number of Indonesia sign language demonstration videos provided in the database to enrich learning materials.The 2 nd user provide suggestion for application development, should add additional pop-up contained of instruction usage method on the camera screen.

CONCLUSION AND SUGGESTION
Based on the test results, it is concluded Indonesian sign language learning media applications based on deep learning could running properly.It's proven from button functional test and objects test result are 100% succeed.The results of user testing also show satisfaction and a good response to the presence of this application.The addition of the Indonesian sign language demonstration video in the database is expected to enrich the learning material.In practice, this learning media is recommended to run on Android version 8 or more.In the future, the application can be equipped with hundreds or thousands of objects connected to the SIBI video display so that a digital SIBI dictionary can be realized for Indonesian deaf children.

Figure 3 .
Figure 3. Image Recognition Experiments with Wall Clock as SIBI Object

Table 2 .
The Recognizable Objects for SIBI

Table 4 .
Result of Functionality Testing

Table 5 .
Result of SIBI Objects Testing