International Journal of Electrical and Computer Engineering (IJECE) Vol.
No.
February 2019, pp.
ISSN: 2088-8708.
DOI: 10.
11591/ijece.
Automated PCB identification and defect-detection system
(APIDS)
Shams Ur Rehman.
Ka Fei Thang.
Nai Shyan Lai School of Engineering.
Faculty of Computing.
Engineering & Technology.
Asia Pacific University of Technology & Innovation.
Malaysia
Article Info
ABSTRACT
Article history:
Ever growing PCB industry requires automation during manufacturing process to produce defect free products.
Machine Vision is widely used as popular means of inspection to find defects in PCBs.
However, it is still largely dependent on user input to select algorithm set for the PCB under inspection prior to the beginning of the process.
Continuous increase in computation power of computers and image quality of image acquisition devices demands new methods for further automation.
This paper proposes a new method to achieve further automation by identifying the type of PCB under inspection prior to begin defect inspection process.
Identification of PCB is achieved by using local feature detectors SURF and ORB and using the orientation data acquired to transform the PCB image to the reference image for inspection of defects.
A close-loop system is produced as a prototype to reflect the practicality of the idea.
A Graphical User Interface was developed using MATLAB to present the proposed system.
Test data contained 29 PCBs.
Each PCB was tested 5 times for camera acquired images and 3 times for database images.
The identification accuracy is 66% for database images and 100% for images acquired from the camera.
The time taken to detect the model of PCB is recorded and is significantly lower for ORB based identification than SURF based.
The system is also a close loop system which detects defects in PCB units.
The detection of defects has highest accuracy of 92.
3% for best controlled environment With controlled environment, the system could detect defects in PCB pertaining to smallest of components such as SMDs.
Received Apr 12, 2018 Revised Jun 25, 2018 Accepted Sep 10, 2018 Keywords:
Machine vision
ORB
PCB defects PCB identification SURF Copyright A 2019 Institute of Advanced Engineering and Science.
All rights reserved.
Corresponding Author:
Ka Fei Thang.
School of Engineering.
Faculty of Computing.
Engineering & Technology.
Asia Pacific University of Technology & Innovation.
Malaysia.
Email: dr.
fei@apu.
INTRODUCTION
The PCB industry is ever-growing and according to .
, the need is for reliable and accurate vision system because PCBs manufactured today has much more component density than earlier and it continues to Study by .
shows that PCB industry is expected to reach US$ 93.
9 billion in 2017.
Manufacturers want their products defect free to satisfy their customer needs and remain up par with the competitors.
Once a product is manufactured in a plant, it has to go through a quality assurance process to ensure that the product that is being sold to the customer does not have any defects.
In this process the product under inspection is tested for defects and flaws.
Most manufacturers prefer means of non-destructive inspection system such as Computer Vision for their products quality assurance.
Computer Vision also known as Machine Vision in industrial applications is the process of extracting digital information from an image of a scene and then utilizing that information to achieve Journal homepage: http://iaescore.
com/journals/index.
php/IJECE A ISSN: 2088-8708 different tasks .
Computer vision has vast applications in the quality assurance field.
A survey conducted in .
describes the importance of computer vision in quality assurance in different industries such as component manufacturing, integrated circuits, printing, glass manufacturing etc.
To utilize computer vision effectively in quality assurance system, it should have the following steps .
Image Acquisition Image Processing Feature Extraction Decision Making Existing inspection systems to detect defects in the PCBs are not fully automated and require human input in differentiating the type and model of the PCB under inspection to scan for defects.
When manufactured products are undergone through a computer vision assisted quality assurance process, the algorithm set is manually selected and controlled by the operator of the machine.
While the accuracy of the existing systems is acceptable, it still relies on human input for algorithm selection.
Incorporating PCB identification prior to inspection in quality assurance is required to reduce human input and fully automate the inspection process.
This paper suggests a method to train an Automated Optical Inspection (AOI) bot to recognize the PCBs from the existing database of the manufacturer.
This would allow the bot to inspect different products under a single inspection cycle without manually setting up pre-programmed algorithm for each product (Manual identificatio.
The proposed method could allow a manufacturer successfully implement or improve their Just in Time (JIT) manufacturing .
and allow for fully automated inspection process with minimal requirement of human input in the process.
Related Work There is plenty of research work focused on pattern recognition in PCB industry.
The idea of using local features for pattern recognition and describing the algorithm as Scale Invarient Feature Transform (SIFT) due to the robustness of the algorithm to changes in scale of the images or different affine transformations was proposed by Lowe .
Szymanski & Stemmer .
proposed classification of the defects of PCB boards in small series production using SIFT features.
Recent advancement in pattern recognition is the algorithm called FAsT-Match which works on minimizing the Sum of Absolute Differences (SAD) error .
Operations such as scaling, translation, rotation etc.
can interfere with the recognition result of other algorithms due to explosion.
This developed algorithm deals with the issue by discretizing the space of affine This work achieved accurate detection result on different images of varying sizes and experimental result on RealWorld scenarios.
SIFT pattern matching involves calculation of log of Gaussian which is time consuming.
Speeded-Up Robust Features (SURF) is an upgraded version of SIFT .
SURF
differs from SIFT in regard as it takes approximation of log of Gaussian using Box Filter which speeds up the recognition process.
Related pattern recognition study in differentiate and matching different types of leaves from a database to find the type of the plant was proposed and comparative study was performed on Histogram of Oriented Gradients (HOG).
Local Binary Pattern (LBP) and SURF .
SURF is termed as optimal interest point detector when detecting objects based on interest points .
Current advent of AI provides human-like ability of decision making to machines.
Combined with vision capability, machine learning approaches for pattern recognition bears great results.
Studies utilizing machine learning techniques using Neural Networks (NN) .
, .
are in favor of classifying objects using patterns in images on a large Recent research based on similar area on classification of PCB for recycling and retirement purposes .
also undertake similar methodologies as proposed in this paper.
However, that work does not utilize further data obtained from classification to proceed with defect detection in PCB units.
Defect detection techniques in machine vision culminate to be the crux of PCB inspection in any inspection equipment.
Throughout the years, various research works has been undertaken to advance the algorithms in this field.
Information obtained from RGB color spectrum can be utilized to find the flux defect in IC pins .
Most techniques involves comparing image under inspection to a reference image to find the variation and describe the irregularities as faults in the PCB .
Background subtraction techniques used by Sundaraj .
in the color spectrum to find missing components or misaligned components using Mixture of Gaussian method has procured high accuracy.
Machine learning techniques have helped to to detect and classify different types of solder joints based on reduction method with the help of neural networks and genetic algorithm trained to classify the defects based on the irradiated color from the solder joints .
PROPOSED SYSTEM
To minimize the human input that is provided to inspection machines prior to inspection process, the machine must be aware of the type of the circuit board that is under inspection so that it can load relevant Int J Elec & Comp Eng.
Vol.
No.
February 2019 : 297 - 306
Int J Elec & Comp Eng
ISSN: 2088-8708
algorithms for defect detection.
The proposed system in this paper provides a solution for inspection system to quickly detect and classify the type of board under inspection with robustness to different physical properties of the boards so that defect detection can proceed for different PCBs in the same inspection cycle without human input.
As seen in Figure 1, the system starts with acquiring image data, either from database files that are available or from Image Acquisition Device (IAD).
Once image is obtained, it is pre-processed and SURF and ORB features are extracted.
These features are then compared with the trained feature set of the database images and the identified PCB is returned.
Once the PCB is identified, the input image is transformed to register it with the reference image and defect-detection process is performed.
There are four major processes in the proposed system as shown in Figure 2.
The first process of the system consists of image acquisition where the image to be used for inspection purposes is acquired either from the database or the camera.
The second involves detection of the PCB that is found in the image.
In the third process, the PCB image obtained is transformed relevant to the reference image of the PCB that has been identified.
The last process involves testing the PCB for any defects that may be introduced either physically or digital morphed into it.
Figure 1.
Flowchart of the proposed system Figure 2.
Block diagram of the proposed system Automated PCB Identification and Defect-Detection System (APIDS) (Shams Ur Rehma.
A ISSN: 2088-8708 Image acquisition The images were obtained with a smartphone camera having specification of 12MP Dual Pixel sensor, f/1.
7 aperture.
LED flash.
OIS (Optical Image Stabilisatio.
with a resolution of 4032x3024.
Image dataset was obtained from the work submitted for free research use in previous study .
During the operation of the camera, a focused image was sent wirelessly from the camera to MATLAB for processing.
Higher resolution was selected to provide meaningful data for image processing as the electronic density on the PCB is high.
PCB identification Once the image is obtained.
SURF and ORB keypoints are detected.
Keypoint is an image location in spatial coordinate where physical features in the image may lie such as corner points.
For SURF keypoints, each of keypoint detected is assigned an orientation obtained from wavelet responses in the x and y direction for neighborhood of 6s, where s is the scale.
A neighborhood region is divided into 4x4 sub regions and the wavelet response for both x and y direction gives the vector used to transform the images .
For ORB keypoints, centroid based on weighted intensity is found inside a small patch of pixels where the feature .
orner poin.
is maintained at the center of the patch.
The direction vector between the corner point and the centroid provides the orientation data .
Once the keypoints are detected, a descriptor is used to describe the keypoints detected.
A descriptor is an algorithm that summarizes the neighborhood around the keypoint to describe a feature by differentiating it from other features thus providing pattern recognition ability.
These SURF or ORB points are then compared with a database of other similar points stored in a database.
The database basically includes long list of these SURF points or ORB points for all the images to be compared with.
To cluster the SURF points of the newly acquired image to the database.
Fast Approximate Nearest Neighbor Search Library (FLANN) index is used.
It provides nearest search algorithm and works well for large databases.
This is helpful to keep the detection time to its minimum even if the database contains large number of feature points for SURF.
ORB points are matched using a BruteForce technique by finding the Hamming distance of the points which compares a point of interest in one image to all the points in other images stored in the database.
Once the points are clustered, the matches in each PCB are tabulated and highest performer is picked as the detected PCB.
For SURF-based identification, the FLANN index is trained before every PCB identification.
Image transformation Once the keypoints are matched, outliers are excluded using the M-estimator SAmple Consensus (MSAC) algorithm.
The process of registration of the input image to the reference image is similar to .
where R-RANSAC algorithm is used instead to find the outliers.
Out of the inliers, 2 points are selected to transform the image acquired to the reference image in the database.
This gives a transformation matrix that can be used to warp the image of the PCB under inspection to the reference image of the matched PCB in the This registers the input image to the reference image where defect-detection can be performed.
Defect detection The transformed image is then compared with reference image to find the defects in the PCB.
The types of defects to detect in this paper are kept missing components and scratches on the PCB as defect detection is the secondary objective.
The algorithm for fault detections takes a dont-care part of the image which is not in the ROI defined by the user for selected reference image.
This is done to minimize the effect of poorly achieved controlled environment for the prototype built due to budget limitation.
The ROIs declared by the user is used to obtain two masks for filtering the image with to obtain dont-care pixels in the The ROIs are stored in a structure with information about the potential defect that the ROI is for.
These are specified before the inspection process.
In industrial application, defect detection algorithms are tailored to customer needs.
The transformed image obtained in the previous step is in RGB color model.
Each of the channels is extracted from the image.
The absolute difference of each of the channel in the transformed image and reference image is found for each pixel of input image denoted by I and reference image by R where x and y are the spatial coordinates and c is the color channel R.
G, and B given in .
This highlights changes in the input image based on the reference image.
Color information is used to obtain better detection of missing components where PCB board itself is identical in color to the component to a certain level where subtraction of Gray level image might result in loss of information.
Absolute difference equation shown in .
AbsoluteDifference = |Ic.
, .
Oe Rc.
, .
| Int J Elec & Comp Eng.
Vol.
No.
February 2019 : 297 - 306
Int J Elec & Comp Eng
ISSN: 2088-8708
The resultant images are thresholded to an 8bit value ranging between .
of the image intensity to obtain logical binary images.
This value is amendable by the user.
This is done to remove the noise produced by the image registration, as there will always be a certain noise after the absolute difference of the The masks obtained prior are then filtered with the logical image to consider the area outside the ROIs as dont-care region.
Additional noise is removed using median filter of 5x5 neighborhoods and followed by morphological operations to identify changes in the components as blobs.
If the area size of the blob is greater than a threshold value for both missing component and scratches, the blob is classified as a The defect detection process is depicted in Figure 3.
Figure 3.
Defect detection process illustration EXPERIMENTAL SETUP MATLAB was used for programming the system.
Image Processing toolbox libraries were utilized along with OpenCV libraries .
available as a wrapper for MATLAB in form of mexOpenCV.
This wrapper was used to reduce identification time significantly as compared to matching SURF and ORB points using MATLAB libraries because of the large size of the images in this case.
A prototype was constructed to act as an image acquisition station.
The bed for placing the PCB was 44x40 cm in dimension.
LEDs strips were used to provide a control environment to a certain degree.
The hardware implementation of the prototype is presented in Figure 4.
The hardware was not able to achieve completely uniform controlled lightening environment to mimic industrial equipment.
A GUI was designed carefully considering the user friendly approaches to challenges.
Configuration button is especially included to perform soft changes such as training for new boards to the code using setting file which is stored locally in a file for future use.
The main window of the GUI is presented in Figure 5.
There are total of 6 tests, grouped into 3, performed on the system to evaluate the accuracy of the result, overall efficiency and speed of the system.
These tests are listed as follows:
Classification .
ata from IAD) Classification .
atabase file.
Defect-detection Classification tests are performed on the images acquired using IAD or the database files.
Both set of images have different resolutions than one another.
The test images obtained from a database set are in different orientation too.
These differences in the quality of images result in different behavior of the applied These tests are focused on the timing of the tasks and matching of features between these images and the accuracy of classification.
The third test is fault detection test based on different faults introduced to the PCB images acquired from IAD.
These faults, as mentioned in section 3 may be introduced digitally or physically to the PCB.
Automated PCB Identification and Defect-Detection System (APIDS) (Shams Ur Rehma.
A ISSN: 2088-8708 Figure 4.
Hardware setup Figure 5.
Graphical user interface of the system
RESULTS AND ANALYSIS
In this section result data is shared along with analysis on the data collected.
Result data is classified into three test groups as explained in section 3.
Identification results .
ata from IAD) In this test, identification of the PCB is done and the result data is tabulated.
This test has 3 sub-tests.
Identification based on SURF features on 20000 Hessian (High Sensitivit.
Identification based on SURF features on 40000 Hessian (Low Sensitivit.
Identification based on ORB features For this test, images are acquired from the IAD.
The PCB in these images are oriented in the same direction in the input image and the reference image as per the industry standard.
Each PCB is tested Ave These tests are performed on four different PCBs.
The images obtained from the IAD are high resolution images.
The test data reveals that for these images, if SURF descriptor is to be used, higher hessian threshold such as 40000 could be used.
It does not aAect the accuracy of the results but reduces the number of keypoints found which helps to decrease Int J Elec & Comp Eng.
Vol.
No.
February 2019 : 297 - 306
Int J Elec & Comp Eng
ISSN: 2088-8708
computational time.
This is the sensitivity of the SURF features.
From Table 1 it can be deduced that detection time is based on the number of keypoints that are found and matches from those keypoints which is in turn dependent on the sensitivity of the SURF.
So SURF based identiAcation could be speed up if the sensitivity is lowered.
The ORB extractor extracts constant 500 keypoints from the image.
Match ratio represents that ratio of matches to the keypoints.
It can be observed that SURF with lower sensitivity, the average matching ratio is the highest which proves it to be more reliable for images where the image quality corresponds to the quality of images available for this test i.
high resolution images.
Majority of the time consumed during the process is taken by the setup time in which matching of features process initializes.
From the Table 1, it can be observed that while ORB based detection is fastest, it results in weaker match ratio.
The identiAcation accuracy of using the SURF and ORB features for this set of images is 100%.
Table 1.
Averages of the Data Obtained from IAD Conducted on 20 Images Feature extractor Setup time .
Detection time .
Transformation time .
Total time .
Matches Keypoints Match ratio SURF .
SURF .
ORB Identication results .
atabase image.
The database is trained to compute SURF features of 54 images where SURF hessian threshold to obtain SURF features is set to 20,000.
Lower hessian threshold has to be used than the one used for the IAD Ales as they are higher resolution than this.
Four of these images, obtained from the IAD are also included in this database.
25 PCBs are selected for testing where each PCB has three images.
Each image is of resolution 2 out of 3 images for each PCB have diAerence in the orientation to the one used as reference in the trained database.
Each PCB image is presented for identiAcation 3 times and changes in orientation has also occurred.
Table 2 shows the averages of the result data obtained.
From the table, it can be observed that identiAcation based on ORB is a lot faster than SURF complying with the test done on IAD images in Table 1.
During the testing, drop in matching ratio was experience over the course of the experiment.
This drop in the matching ratio is explained by the change in orientation.
This does not hinder the identiAcation Total images test are 75 .
for each PCB, 25 total PCB).
74 were identiAed correctly in both cases.
The identiAcation accuracy for the system based on image data also containing oriented images for both SURF and ORB based feature is 98.
Table 2.
Averages of the Data Obtained from Database from 75 Images Feature extractor Setup time .
Detection time .
Transformation time .
Total time .
Matches Keypoints Match ratio
SURF
ORB
The detection rate for both database images and IAD images is comparatively superior to other methods of classification using SURF features .
based on leaves and Feedforward Neural Network (FNN) classification based on fruits .
In terms of similar classification performed on PCBs, the detection accuracy is similar to .
, the time consumption is far more compared to the same study.
Defect detection result This test is a measurement of accuracy of the fault detection algorithm employed to detect either missing components or scratches on PCB.
The dont-care region is neglected based on the ROI which the user registers it for each PCB in the conAguration menu.
This ROI selection ensures that minor noises which are caused by changes in the illumination or view point are avoided as in industrial grade equipment, the noise could be minimized by conducting the process under controlled environment.
Two types of defects are tried to be identiAed by the algorithm.
missing component (MC) and Scratched Components (SC).
The MC and SC are identiAed and drawn over the input image to be displayed to the user.
The images that are selected for testing the accuracy of fault detection algorithm and tabulating the results are obtained from IAD.
The images are then saved locally and digitally modiAed to have defects in the PCB.
All defects are introduced to single image of the PCB, however, this image is not the same as the reference image.
The image resolution is 4032x3024.
The test samples include 4 PCBs with numerous Automated PCB Identification and Defect-Detection System (APIDS) (Shams Ur Rehma.
A ISSN: 2088-8708 Table 3 shows the accuracy of the identiAed faults.
The highest accuracy of 100% is achieved while testing the Arduino UNO R3.
Fault in the Arduino board was made by removing a chip from its location then the algorithm was tested on it.
However, the chip removed was fairly large in size.
The program sends cropped image of the missing component to the output folder which is shown in Figure 6.
Asrock 960GC GS board has almost perfect alignment with reference image.
Hence, in this case, the surrounding environment is fairly controlled.
Further data is collected on performing same test with diAerent threshold values and it is tabulated in Table 4.
The increasing accuracy of the result when threshold decreases signiAes that with control environment provided, the algorithm can work to achieve higher efAciency and accurate results.
The accuracy score for lowest threshold in the experiment shows that the algorithm performs similar to other methods of defect detection .
Low threshold values cannot be used for images where there is error in image registration process and this error can be depicted as defect.
For the tested PCB, when the threshold values were decreased, it also detected faults in smaller components such as SMDs.
Figure 6.
Missing components output data Table 3.
Defect Detection Data (Threshold MC 70.
SC .
PCB Acer Board Asrock 960GC GS Yellow UNO R3 Arduino UNO R3 ve Table 4.
Defect-detection Result of 'Asrock 960GC GS' at Various Threshold
Threshold
(MC/SC) 0-255
70/50
60/40
50/30
35/25
ve Possible sources of errors There are number of factors that can negatively inCuence the result of the applied algorithm.
While the identiAcation is seamlessly executed, the defect-detection process is subjected to certain errors.
During observation, the transformation matrix obtained to warp the input image gives better result when SURF features are used.
ORB features produces bit of noise when the images are compared.
Due to this error, lower threshold values for MC and SC detection underperforms and produces false positives.
Due to uncontrolled lightening environment, detection of darker components on a darker shaded PCB is also limited and prone to Int J Elec & Comp Eng.
Vol.
No.
February 2019 : 297 - 306
Int J Elec & Comp Eng
ISSN: 2088-8708
CONCLUSION AND FUTURE WORK
The aim of the project is develop a system that would allow diAerent products for processing with pattern recognition and image processing that would help provide the manufacturer with diAerent product testing under single inspection cycle.
The ideology behind the project was allow PCB inspection equipment with machine vision to identify the PCB model and make decision based on the result of the identiAcation, in this case.
PCB defect-detection.
PCB identiAcation in this system prior to inspection process has been achieved using feature detectors and descriptors which is a pattern recognition method.
The identiAcation rate gives of 98.
66% for 75 database images and 100% for 20 IAD images.
The defect detection algorithm developed accuracy of 92.
3% for best case scenario performs very well in comparison with other methods for similar problem.
The accuracy of defect-detection is subjected to many factors but with controlled environment provided, it can be signiAcantly improved for all types of PCBs.
The complete system is implemented with having a user friendly GUI to operate the system with.
Multiple tests are performed on the test to evaluate diAerent parameters of the system.
The test results obtained focuses on the speed and accuracy of the algorithms.
Proposed system was developed to simplify the Machine Vision based QC process.
However, the system that has been implemented is not perfect and have limitation.
While the speed of the system is a fraction of a second and considered fast.
Machine Vision inspection system is relatively faster.
Therefore, the speed of the system needs to be further improved.
Some methods could be made simpler with next version of the program.
Defect-detection process needs improvement as it is only based on detection of missing components in the PCB or scratches on any components and is not consider to be very reliable during our However, this process could be largely improved by accessing controller environment of the obtained If the histogram of the image is in higher region for all channels, it also induces the incapability of the program to distinguish between a bad component and a false positive.
The system developed could be improved upon further research or recommendation based modiAcation.
The program is written in MATLAB which is a high level language, the speed of the system could easily be improved by translating it into a low level language such as C where OpenCV library is more easily available to use.
The image registration process method could also be replaced with a better performing method to avoid errors in next process due noise after image registration.
However, this could also be achieved by providing controlled environment.
To conclude, machine vision inspection equipment can perform better if equipped with identiAcation techniques to base the decisions on.
Moreover, a machine vision friendly color standards could set for PCB manufacturers to color the components and PCB in certain standards for easier recognition of the components or the defects in the PCBs.
REFERENCES