Langit Biru: Jurnal Ilmiah Aviasi Vol.
17 No.
2 June 2024 ISSN .
1979-1534 ISSN .
2745-8695 Page.
62 : 71
DESIGN OF IMAGE PROCESSING APPLICATION FOR FOREIGN OBJECT
INSPECTION USING DRONE ON AIRCRAFT RUNWAYS
Ferry Budi Cahyono .
Suparlan .
Arif Hidayat.
1,2,3 Curug Indonesian Aviation Polytechnic e-mail: 1ferry.
budi@ppicurug.
id, 2suparlan@ppicurug.
id, 3arif.
h@ppicurug.
coresponding: 3arif.
h@ppicurug.
Received :
25 March 2024 Revised :
25 May 2024 Accepted :
27 June 2024 Abstract: Flight safety is a crucial aspect of the aviation industry, significantly dependent on optimal runway conditions for successful takeoffs and landings.
Foreign Object Debris (FOD) such as stones and potholes on the runway poses significant threats, requiring preventive measures to avoid operational failures and aircraft damage.
Therefore, this research aimed to design an image processing application for drone usage to enhance runway inspections.
Research and Development (R&D) method was used from application design to field testing.
Image processing used You Only Look Once (YOLO) method for swift and accurate FOD detection, maximizing drone effectiveness in identifying risks.
The results showed that object size and image capture distance significantly affected detection performance.
This analysis provided a basis for model optimization through improved data augmentation methods and parameter adjustments to address challenges in detecting objects of various sizes.
Keywords: Aviation Safety.
Runway Inspection.
Drone Imaging.
YOLO.
FOD Introduction Aviation industry is essential in global transportation, with flight operation safety being a major concern.
The integrity of aircraft runway is crucial to ensuring safe takeoffs and landings.
However, the recent challenge in runway safety is the presence of Foreign Object Debris (FOD), including items such as stones and potholes.
FOD poses a significant threat, potentially causing operational disruptions, aircraft damage, and harmful incidents in extreme cases.
Traditional runway inspection methods based on direct observation by ground personnel using vehicles have limitations in terms of safety and comprehensive condition assessment.
To address the limitations, several research have explored various related methods to develop mitigation strategies.
These include video-based FOD detection (Qungyu.
Huansheng, & Weishi, 2.
using image processing algorithms to identify foreign objects through camera sensor data.
A prototype Runway Line Detection System based on image processing (Ardi.
Harjoko, & Sumiharto, 2.
has used image processing to identify runway lines through OpenCV.
This application is capable of detecting lines at heights less than three meters from the ground surface.
Other investigations such as the Assessment of Foreign Object Debris Management Using Group 1 Unmanned Aerial Systems (Lee, 2.
have applied MATLAB's image processing algorithms to recognize foreign objects based on several parameters, including dimension and color.
During this process.
Unmanned Aerial Vehicle (UAV) height and window filter size are adjusted for optimal customization.
A review on FOD Detection Technologies and Advancement for Airport Safety and Surveillance (Fizza, et al.
, 2.
also shows that the millimeter-wave radar method demonstrates promising results in trials at KLIA airport.
Therefore, the method is considered the most effective solution for replacing manual inspections and maximizing airport operational efficiency.
DOI: https://doi.
org/10.
54147/langitbiru.
Design Of Image Processing Application For Foreign Object Inspection Using Drone On Aircraft Runways Previous research on runway FOD detection (Papadopoulos & Gonzales, 2.
have used UAV and artificial intelligence (AI) to identify foreign objects such as paper, plastic, bottles, metal, and bolts, through YOLOv3 and Microsoft Azure Custom Vision.
Pothole detection and dimension estimation system using deep learning (You Only Look Once/YOLO) (Chitale.
Kekre.
Shenai.
Karani, & Gala, 2.
applies YOLOv4 and YOLOv3-based image processing algorithms to identify road potholes using datasets from Google Image.
Due to these challenges, there is a growing need for innovative and technological solutions in runway inspection practices.
In this context.
UAV or drone has been proven effective as promising tools.
UAV offer an alternative and more efficient perspective for visual and datadriven decision-making in inspection programs, as well as enabling substantial cost and time savings (Gillins.
Parrish.
Gillins, & University, 2.
, (Irizarry & Costa.
Exploratory Research of Potential Applications of Unmanned Aerial Systems for Construction Management Tasks, 2.
(Costa.
Irizarry, & Kim, 2016.
The use of UAV for topographical surveys can decrease the manhours needed for airport inspections and enhance measurement accuracy.
Therefore, the integration into airport inspection procedures allows for the assessment of runway pavement conditions and the identification of obstructions (Irizarry.
Kim.
Johnson, & Lee, 2.
UAV
photogrammetry also enables the creation of 3D models from 2D images collected at points of interest (Kim.
Irizarry.
Costa, & Mendes, 2016.
, (Rodriguez-Gonzalvez.
Gonzalez-Aguilera.
Lopez-Jimenez, & Picon-Cabrera, 2.
Based on the description, this research aimed to design an image processing application for drone usage to enhance runway inspections.
The application uses advanced methods, namely YOLO, to detect and classify FOD, including stones and potholes.
The integration of YOLO into the image processing workflow enables fast and accurate analysis, maximizing drone effectiveness in identifying potential hazards on runway.
Method This research applied Research and Development (R&D) method using Analyze.
Design.
Develop.
Implement, and Evaluate (ADDIE) method.
The analyze stage aims to deeply understand the research needs and objectives, while the design is the detailed pattern of the program to be developed based on the previous analysis.
The develop stage is where the program is developed according to the established design.
Subsequently, the implement stage aims to conduct the program in the research environment.
In the evaluate stage, the program is examined comprehensively to ensure the proper functioning and achievement of predetermined objectives.
Continuous monitoring and evaluation are conducted to ensure the effective and efficient operation over time (Sugihartini & Yudiana, 2.
ADDIE method provides a structured and systematic framework for conducting this research.
Figure 1.
ADDIE Method The hardware required to implement this research is shown in Table 1.
Table 1.
Hardware Device name Remote Control Radio Telemetry .
MHz/915 MH.
Function as remote control as receiver Langit Biru: Jurnal Ilmiah Aviasi Vol.
17 No.
2 June 2024 ISSN .
1979-1534 ISSN .
2745-8695 ArduPilot GPS Module Frame Hexacopter Motor Brushless 2212 920K Electronic Speed Controller ESC 30A Simonk Lion 4s Battery NVIDIA Jetson Nano Developer Kit Runcam Thumb Pro Camera Dongle Wifi Access Point Laptop/PC as controller as navigation tool and location determiner as UAV framework as main drive motor as motor speed controller as energy storage device Mini PC for image processing Captures images for processing and detection by Jetson Nano mini PC to catch the WiFi signal To increase the internet range that can be received by the wifi dongle To display the results of object detection and create a detection program The software needed to implement this research can be seen in Table 2.
Table 2.
Software Software name Mission Planner Ubuntu Operating System Python OpenCV Roboflow Function To design drone paths and drone autopilot controllers as operating system for mini PC Programming Language Python Modul for image processing Software used to carry out training objects The system developed in this research serves to detect the presence of foreign objects on the runway recorded by drone camera.
The design of drone path for detection is carried out using a mission planner application.
Figure 2 shows drone equipped with GPS module, mini PC, and radio telemetry.
Subsequently, the configuration is based on a mission planner to determine the flight path.
Figure 2.
Drone Used Figure 3 shows the image processing flow diagram for detecting stones and potholes.
starts with initializing the camera to the mini PC and adjusting the camera height for image capture.
Subsequently.
YOLOv4 is set up and searches for object references that are taken from Roboflow.
The next step includes preparing a dataset containing labeled images of stones and potholes.
YOLO model is trained using the prepared dataset on Roboflow and implemented on the camera.
When stones or potholes are detected, labels are added to the image along with additional information such as object coordinates and classification.
Design Of Image Processing Application For Foreign Object Inspection Using Drone On Aircraft Runways Figure 3.
Image Processing Flowchart You Only Look Once (YOLO) Model This section describes the main process of detecting and classifying stones and potholes.
The steps for detection and classification include image pre-processing.
YOLO model implementation, and post-processing to generate accurate detection results.
Moreover, the use of YOLO algorithm with Darknet enables precise real-time object detection for applications in surveillance, face recognition, fault detection, and character recognition.
Other benefits include future improvements such as faster GPU model, 360-degree camera, and GPS integration for instant detection (Gothane, 2.
The entire data processing process is carried out using Python software and OpenCV configured to support YOLO implementation on drone.
The preprocessing phase of the data includes several steps for both training and testing.
Training phase consist of resizing and cropping the images to 416 x 416, adhering to YOLO requirements to standardize the input data format (Redmon.
Divvala.
Girshick, & Farhadi, 2.
Additionally, the preprocessing includes adjusting dimensions and placing bounding boxes on the images, alongside data annotation.
This is carried out to provide information on the bounding box locations and object classes for each image.
During testing, the preprocessing also includes introducing disturbances to image, such as blurring and adjusting brightness levels.
Data annotation remains integral to this phase, providing details for each image, such as bounding box information and object classes.
The design phase of model includes implementing YOLOv4 for detection and classification in this research.
The steps of YOLO model are outlined as follows.
RGB image data resulting from preprocessing is inputted, with image dimensions of 416 y 416 y 3.
This is followed by the determination of initial weights and YOLO network model from pre-trained weights.
Subsequently, training process is carried out by fine-tuning the initial weights and YOLO model using the dataset from this research, which comprises 9 convolutional and 6 maxpool layers.
The weights produced from the training process are used to test the images.
Object classification accuracy can be computed using the provided equations.
These include true positive (TP), true negative (TN), false positive (FP), and false negative (FN) classifications (Ryu.
Kim, & Kim.
Langit Biru: Jurnal Ilmiah Aviasi Vol.
17 No.
2 June 2024 ISSN .
1979-1534 ISSN .
2745-8695 Drone Installation and Autopilot Configuration using Mission Planner In this research, drone is simulated based on the design of automatic UAV flight.
Ardupilot Mega requires software such as Mission Planner to plan missions or download new This is because Mission Planner software can monitor all aircraft conditions including altitude, trajectory, battery status, and more.
Telemetry kits also enable real-time tracking of UAV or changing mission while UAV is airborne (Saroinsong.
Poekoel, & Manembu, 2.
In using Mission Planner, it is recommended to include a design phase to obtain results consistent with flight simulations using Mission Planner and utilizing Ardupilot Mega.
Flight Path Design In this research, the design for runway inspection flight route is carried out using UAV with APM.
After completing the design of the flight path, it is saved by clicking the save button for a new menu to appear, and name the flight planning file with the .
txt format.
Figure 4 shows the flight plan that has been created on the workspace.
Figure 4.
Flight Path Design The simulation was successfully conducted for the flight mission using the flight path at UPBU Budiarto.
Therefore.
UAV was able to execute the mission successfully, starting from takeoff, stabilization, the mission, and landing normally.
Dataset Collection and Training Data Dataset is a collection of data derived from previous information and can be processed into new information through supervised learning method.
In predictive system, dataset serves as the foundation or reference for decision-making (Arnesia.
Pratama, & Sjafrina, 2.
Moreover, dataset of stones and potholes used for object training in this research is obtained from Roboflow, an online platform that allows users to share and simultaneously process datasets.
For stone objects, dataset is located at https://universe.
com/w/wedfas/dataset/2, consisting of 1051 labeled images.
Meanwhile, dataset for pothole objects is located at https://universe.
com/universal-college-of-engineering/road-pothole-detectionmw5zg/dataset/2, which comprise 101 labeled images.
After training using YOLOv4, dataset testing was performed.
Design Of Image Processing Application For Foreign Object Inspection Using Drone On Aircraft Runways Figure 5.
Stone Dataset Figure 6.
Pothole Dataset Implementation of Video Delivery This discussion focuses on the development of a video transmission design for runway inspection using UAV connected to WiFi and Jetson Nano.
Initially, the conceptual design to be implemented is explained before detailed description of the design steps.
Figure 7.
Video Delivery Design System The Jetson Nano acts as a data processing device for GPS and video to integrate location Subsequently, the processed video is delivered through WiFi connection, enabling wireless transmission to monitoring devices or ground stations.
The devised design is tested to ensure compliance with the established criteria.
In this context, the design is implemented on UAV to transmit GPS and video data, as described in the following steps.
Langit Biru: Jurnal Ilmiah Aviasi Vol.
17 No.
2 June 2024 ISSN .
1979-1534 ISSN .
2745-8695 Discussion In this section, the evaluation of YOLO model is explained, consisting of testing dataset, quality, tables, and testing analysis.
Before proceeding to the evaluation stage, testing dataset is prepared, which comprises images obtained using Roboflow.
This dataset is designed to include variations of conditions that may be encountered in the field, such as weather changes, lighting, and different types of foreign objects.
Additionally, it plays a crucial role in training and testing the detection model using YOLO.
The quality of dataset significantly affects the performance of the detection model.
This is attributed to diversity, which allows model to learn effectively and improve the ability to recognize as well as process foreign objects in different environmental conditions.
Therefore, it is essential to ensure that the dataset reflects situations representing various conditions in the field.
Redmon.
Divvala.
Girshick, & Farhadi .
showed that diverse datasets improved YOLO's generalization capabilities in various scenarios.
Research by (Bochkovskiy.
Wang, & Liao, 2.
on YOLOv4 showed significant performance improvements with high-quality, diverse datasets.
The testing was conducted at several drone altitude levels, where detection accuracy, classification accuracy, and detection suitability calculations were performed.
This method was consistent with the research by (Wang.
Liao.
Wu, & Chen, 2.
, which evaluated detection accuracy at different altitudes and lighting conditions.
Detection Accuracy = ycNycycyce ycEycuycycnycycnycyce yayceycyceycaycycnycuycu ycNycuycycayco ycCycaycyceycayc Classification Accuracy = ycNycycyce ycEycuycycnycycnycyce yaycoycaycycycnyceycnycaycaycycnycuycu x100 Conformity Detection = ycNycuycycayco ycCycaycyceycayc ycNycycyce ycEycuycycnycycnycyce yaycuycuyceycuycycoycnycyc yayceycyceycaycycnycuycu ycNycycyce ycEycuycycnycycnycyce yayceycyceycaycycnycuycu The testing is conducted with object sizes of 5x5, 10x10, 15x15, and 20x20.
The table below presents the test results for each parameter, where the X-axis represents the image capture distance and the Y-axis is the percentage results for object size.
Image capture distance Figure 8.
Detection Accuracy Figure 8 shows that as the distance of image capture increases, the detection accuracy Additionally, as the image size increases, the percentage of detection accuracy also rises for each distance of image capture.
Design Of Image Processing Application For Foreign Object Inspection Using Drone On Aircraft Runways Image capture distance Figure 9.
Classification Accuracy Figure 9 shows that as the distance of image capture increases, the classification accuracy Additionally, as the image size increases, the percentage of classification accuracy also increases for each distance of image capture.
Image capture distance Figure 10.
Detection Suitability Figure 10 shows that as the distance of image capture increases, the classification accuracy decreases, with varying values for each image size.
However, for the 20x20 image, it remains with the highest percentage of classification accuracy.
The Influence of Image Capture Distance and Object Size: Several parameters such as detection accuracy, classification, and suitability metrics, can be used to analyze changes in model performance with increasing image capture distance.
This provides a deep understanding of model effectiveness to address variations in distance to detect foreign objects.
FOD Accuracy: Specific analysis for each class of foreign objects, including stones and potholes can be conducted by calculating detection and classification accuracy metrics.
This provides insights into model ability to recognize and classify each type of foreign object.
Conclusion In conclusion, this research showed that object size significantly influenced the detection model performance, with smaller objects proving harder to detect at longer distances.
It was also Langit Biru: Jurnal Ilmiah Aviasi Vol.
17 No.
2 June 2024 ISSN .
1979-1534 ISSN .
2745-8695 observed that the selection of image capture distance required careful considerations to ensure optimal model accuracy.
These results provided valuable information as a basis for model optimization, including enhancing data augmentation methods or adjusting parameters to overcome challenges in detecting objects of varying sizes.
Despite the valuable insights, there were limitations in detecting objects at longer distances, particularly for smaller objects.
Therefore, careful consideration should be applied during the design of detection systems in the field.
Bibliography